[generate] Run custom generation code from the Hub

[gante]

What does this PR do?

This PR expands our modeling code on the hub support: it enables the definition of custom decoding methods on the hub, sharing the same interface as generate.

How does it work?

Assumptions:
📖 custom decoding methods are defined in model repos
✍️ the custom generation code is defined in custom_generate/generate.py
🔎 these repos have a custom_generate tag so we can quickly find them

The custom generation code defined in the repo "foo/bar" can be accessed in two ways:

1. If we load model = AutoXXX.from_pretrained("foo/bar"): model.generate will always call the custom generate defined in "foo/bar";
2. Any other model: we can specify the repo to load the custom generate from, i.e. model.generate(..., custom_generate="foo/bar")

(for more info, read the added documentation)

Expected benefits

🧠 Creators of new decoding methods:

• No need to go through a lengthy PR review cycle;
• Day 0 support of their techniques, no need to wait for a transformers release;
• Same model.generate interface, but much higher implementation freedom
• Users don't need to install a new Python repo in their environment
• Hub features (custom readme, discussion pages, track downloads, ...)

🔨 transformers team:

• No more gatekeeping (technique A gets added, technique B doesn't get added)
• More generation methods, lower expected maintenance 🤞
• Empower our community 💛

Working Example

(generate.py in transformers-community/custom_generate_example)

from transformers import AutoModelForCausalLM, AutoTokenizer

# Qwen/Qwen2.5-0.5B-Instruct copy, but with custom generation code -> `generate` uses the custom code
# note: calling the custom method prints "✨ using a custom generation method ✨"
tokenizer = AutoTokenizer.from_pretrained("transformers-community/custom_generate_example")
model = AutoModelForCausalLM.from_pretrained("transformers-community/custom_generate_example", device_map="auto", trust_remote_code=True)

inputs = tokenizer(["The quick brown"], return_tensors="pt").to(model.device)
gen_out = model.generate(**inputs)
print(tokenizer.batch_decode(gen_out, skip_special_tokens=True))

# `generate` with `custom_generate` -> `generate` uses custom code
# note: calling the custom method prints "✨ using a custom generation method ✨"
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct", device_map="auto")

inputs = tokenizer(["The quick brown"], return_tensors="pt").to(model.device)
gen_out = model.generate(**inputs, custom_generate="transformers-community/custom_generate_example", trust_remote_code=True)
print(tokenizer.batch_decode(gen_out, skip_special_tokens=True))

Sanity checks, copy paste after the script above:

# Sanity check: `transformers-community/custom_generate_example` contains a simplified greedy decoding method
custom_generated_text = tokenizer.batch_decode(gen_out, skip_special_tokens=True)[0]
gen_out = model.generate(**inputs, do_sample=False, repetition_penalty=1.0, max_length=20)
greedy_generated_text = tokenizer.batch_decode(gen_out, skip_special_tokens=True)[0]
assert custom_generated_text == greedy_generated_text

# Sanity check 2: `trust_remote_code` is required
try:
    model.generate(**inputs, custom_generate="transformers-community/custom_generate_example")
except Exception as e:
    print("got exception 👍")
else:
    raise Exception("This should have failed -> no `trust_remote_code`")

try:
    model = AutoModelForCausalLM.from_pretrained("transformers-community/custom_generate_example", device_map="auto")
except Exception as e:
    print("got exception 👍")
else:
    raise Exception("This should have failed -> no `trust_remote_code`")

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[gante]

Reorganized the generation strategies docs to make it easier to find the docs regarding custom generation methods.

The outer level is now basic decoding methods (moved greedy decoding, sampling, and beam search here), advanced decoding methods (moved the other ones here), and custom decoding methods (new section for the code on the Hub)

[stevhliu]

Oh cool! I think it could be even more helpful if we add under "Basic decoding methods" a few sentences about when you should use these methods, and the same for "Advanced decoding methods" to help users further differentiate between basic and advanced.

[stevhliu]

Wow super cool! Feels like we can load more and more custom things from the Hub into Transformers these days 🚀

[stevhliu]

Oh cool! I think it could be even more helpful if we add under "Basic decoding methods" a few sentences about when you should use these methods, and the same for "Advanced decoding methods" to help users further differentiate between basic and advanced.

[gante]

The new docs start here

[gante]

We were throwing an exception and a warning 🤔

Lowered the severity, otherwise we were throwing warnings in most model from_pretrained (see changes in modeling_utils.py)

[gante]

Previously: we could only load root level custom modules
With this change: we can load custom modules in any folder

[LysandreJik]

nice!

[gante]

Logic for the requirements exception as below

ValueError: Missing requirements for joaogante/test_generate_from_hub_bad_requirements:
foo (installed: None)
bar==0.0.0 (installed: None)
torch>=99.0 (installed: 2.6.0+cu126)

[LysandreJik]

This looks good to me!

As mentioned offline, let's make sure that it's only runnable with trust_remote_code=True.

Also, this was very recently merged and might make sense for your versions comparisons:

transformers/src/transformers/utils/import_utils.py

Lines 2091 to 2146 in 23d79ce

	class VersionComparison(Enum):
	EQUAL = operator.eq
	NOT_EQUAL = operator.ne
	GREATER_THAN = operator.gt
	LESS_THAN = operator.lt
	GREATER_THAN_OR_EQUAL = operator.ge
	LESS_THAN_OR_EQUAL = operator.le
	@staticmethod
	def from_string(version_string: str) -> "VersionComparison":
	string_to_operator = {
	"=": VersionComparison.EQUAL.value,
	"==": VersionComparison.EQUAL.value,
	"!=": VersionComparison.NOT_EQUAL.value,
	">": VersionComparison.GREATER_THAN.value,
	"<": VersionComparison.LESS_THAN.value,
	">=": VersionComparison.GREATER_THAN_OR_EQUAL.value,
	"<=": VersionComparison.LESS_THAN_OR_EQUAL.value,
	}
	return string_to_operator[version_string]
	@lru_cache()
	def split_package_version(package_version_str) -> Tuple[str, str, str]:
	pattern = r"([a-zA-Z0-9_-]+)([!<>=~]+)([0-9.]+)"
	match = re.match(pattern, package_version_str)
	if match:
	return (match.group(1), match.group(2), match.group(3))
	else:
	raise ValueError(f"Invalid package version string: {package_version_str}")
	class Backend:
	def __init__(self, backend_requirement: str):
	self.package_name, self.version_comparison, self.version = split_package_version(backend_requirement)
	if self.package_name not in BACKENDS_MAPPING:
	raise ValueError(
	f"Backends should be defined in the BACKENDS_MAPPING. Offending backend: {self.package_name}"
	)
	def is_satisfied(self) -> bool:
	return VersionComparison.from_string(self.version_comparison)(
	version.parse(importlib.metadata.version(self.package_name)), version.parse(self.version)
	)
	def __repr__(self) -> str:
	return f'Backend("{self.package_name}", {VersionComparison[self.version_comparison]}, "{self.version}")'
	@property
	def error_message(self):
	return (
	f"{{0}} requires the {self.package_name} library version {self.version_comparison}{self.version}. That"
	f" library was not found with this version in your environment."
	)

cc @Rocketknight1 in case you want to give a review as the owner of remote code :)

[LysandreJik]

nice!

[ArthurZucker]

Thanks! hopping to see this enable otherdevs!

[ArthurZucker]

all of this can be done in loda_custom_generate that would return the generate arguments!

[gante]

It needs locals() from generate to redirect non-keyword arguments :')