Group beam search with sampling?

[adrian-valente]

Feature request

In the current generation code, group beam search is necessarily greedy. From a theoretical point of view, it is not very clear why that should be the case, since the diversity penalty is applied on the logits anyway, yielding a full distribution from which sampling can still be performed.

Motivation

I think there is a reasonable use case for such a feature: diversity beam search is very useful in particular for modalities like biological sequences which increasingly use the transformers library, but I could see it be useful as well for natural language or code, to generate diverse paths without falling to the drawbacks of greedy generation. From a more abstract point of view it is also seemingly unjustified to allow sampling for standard beam search and not for diversity beam search.

Your contribution

I am aware of the work in #30810 so don't want to disrupt but would be happy to look into it.

[zucchini-nlp]

cc @gante

[gante]

Hi @adrian-valente 👋 Thank you for opening this issue!

We've just merged support for custom generation code on the Hub (#36405), and we will move a few less popular generation techniques to code on the Hub. The idea is to improve the maintenance quality on a few select techniques, and normalize the use of advanced generation methods from the Hub.

Group beam search will be one of those techniques, and we will redirect calls to a Hub repo. If you're open to it, you can create a clone of the existing group beam search, but with sampling enabled, and we will redirect calls to your repo 🤗 Otherwise, when moving the code to the Hub, I will add this functionality.

Docs: https://huggingface.co/docs/transformers/generation_strategies#custom-decoding-methods

[adrian-valente]

Amazing, thank you, will give it a try! That answers my issue, I will link my hub function here then if I publish it.

[gspeter-max]

Fixes #38268

--- Generating with Greedy Group Beam Search ---

Setting pad_token_id to eos_token_id:50256 for open-end generation.

Greedy Outputs:
0: The best way to learn about large language models is to learn about the language.
1: The best way to learn about large language models is to learn about the language.
2: The best way to learn about large language models is to look at a few examples of how to use them.
The first step is to look at a few examples of how to use them. The second step is to look at a few examples of how to use them. The third step
3: The best way to learn about large language models is to look at a few examples of how to use them.
The first step is to look at a few examples of how to use them. The first step is to look at a few examples of how to use them. The second step

--- Generating with Sampling Group Beam Search ---

Sampling Outputs:
0: The best way to learn about large language models is to look at a few simple examples of how a language can be used to understand languages. For example, a simple example can look at a simple example of how a language can be used to understand languages. For example, a simple example can look at
1: The best way to learn about large language models is to find new ways to work around them.
When you start making your own languages, you should be careful not to think about how you are doing this.
You should not be just focusing on the tools that you
2: The best way to learn about large language models is to look at a few simple examples of how a language can be used to understand languages. For example, a simple example can look at a simple example of how an interpreter can be used to understand languages.
3: The best way to learn about large language models is to find new ways to work around them.
When you start making your own languages, you should be careful not to think about how you are doing this.
You should not be just focusing on the tools that are

--- Generating with MORE RANDOM Sampling Group Beam Search ---

More Random Sampling Outputs:
0: The best way to learn about large language models is through the research paper published in Psychological Science .
I was so lucky (because everyone else I was lucky to be with has been with) that it was very helpful to do some small tasks so I couldn't stop feeling inspired by
1: The best way to learn about large language models is through the research paper published in Psychological Science .
I was so lucky (because everyone else I was lucky to be with has been with) that it was very helpful to do some small tasks so I couldn't stop feeling very good
2: The best way to learn about large language models is to take a look at a few simple language modeling tutorials you can be sure you’ll learn a lot about the languages and frameworks that you’ll be used to writing them. These tutorials can be found in The Cucumber, Python
3: The best way to learn about large language models is to take a look at a few simple language modeling tutorials you can be sure you’ll learn a lot about the languages and frameworks that you’ll be used to writing them. These tutorials can be found in The A Language Modeler blog