Add t5 summarization example

[patrickvonplaten]

Adds TF 2.0 Example for T5 summarization.

Adds dataset download file via tensorflow_datasets and a rouge scorer.

Example is currently being tested on T5-large on GPU to see how rouge scorer performs in comparsion to examples/summarization/bart rouge scorer.

[patrickvonplaten]

@craffel -
I wanted to add an example for T5 summarization for the CNN/DM dataset.

These are the default params, I use for t5 summarization. I saw in the paper that you use num_beams=4 as well and from the code:
https://github.com/google-research/text-to-text-transfer-transformer/blob/master/t5/models/gin/beam_search.gin
and
https://github.com/google-research/text-to-text-transfer-transformer/blob/master/t5/models/gin/sequence_lengths/cnn_dailymail_v002.gin, I assumed that you pad/cut to 512 tokens. transformers uses the "simple" length_penalty instead of the "google" length_penalty so instead of alpha=0.6, I set the length penalty here to 2.0 - do you think that is good? It's the default parameter for Bart. And regarding min_length and max_length, I am not sure whether these params are good - they are copied form the Bart defaults.

[craffel]

Hey Patrick, thanks for making this. Some notes

• The released checkpoints are after multitask pre-training. You will get decent summarization results with them, but in order to get our SoTA-at-the-time summarization results you'll need to finetune the model further before generating from it. You can probably ignore that for the sake of an example, but it may be worth mentioning in the README.
• We didn't play with the num beams or length penalty at all, just set them to values that had been used in the past on machine translation and never tweaked them at all. So, your values are probably fine (especially since you already are going to be losing some performance without fine-tuning).
• We don't have min length/max length functionality in our code; we just let the model choose whatever length. If BART uses this trick, it probably helps, and it seems somewhat model-agnostic so we can just use their values.
• The model should work equivalently whether you pad to a max length or not. You should also be able to use a sequence length longer than 512 and it should work fine too.

[patrickvonplaten]

Thanks a lot!

[craffel]

Any reason not to use a Python ROUGE scorer, e.g.
https://github.com/google-research/google-research/tree/master/rouge
That way you can just score within the evaluate_cnn.py file instead of having people run this separate code.

[patrickvonplaten]

Added the google-research rouge scorer :-). Works well, but it's not the nicest python API to score a list of strings :D.

[thomwolf]

Ok for me, I let you take care of @craffel comments

[sshleifer]

pending my comments

[patrickvonplaten]

pending my comments

Very much down to share the summarization code in another PR!

[patrickvonplaten]

Code quality test fails because of unpinned isort library (see #3449)

[thomwolf]

Ok for me!

[craffel]

1. Suggest changing "the T5 Model t5-large" to "the multitask pre-trained checkpoint for t5-large
2. If you want to be more specific than "slightly", it looks like they are about 0.5 ROUGE points (on all ROUGE variants) lower than they would be with fine-tuning.