[Bug] Models generate whitespace-only output when temperature is in range [1e-4, 1e-5], regardless of model type

[saumya-saran]

Setting the temperature in a particular range causes vllm to generate whitespace-only outputs. Values above/below this range work correctly. I have seen this with facebook/opt-125m, fine-tuned mistral-7B models, codellama-13B, and several other models. It seems like this is an issue with vllm rather than the particular model:

To reproduce:
python -m vllm.entrypoints.openai.api_server --model facebook/opt-125m

Send request:

curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "facebook/opt-125m",
"prompt": "San Francisco is a",
"max_tokens": 7,
"temperature": <temperature>
}'

With temperature:
1e-3: Generates " great place to live. I"
1e-4: Generates "<s><s><s><s><s><s><s>"
1e-5: Generates "<s><s><s><s><s><s><s>"
1e-6: Generates " great place to live. I"

[hmellor]

The reason it starts working again at 1e-6 is because it is less than _SAMPLING_EPS, so it reverts to greedy sampling:

vllm/vllm/sampling_params.py

Line 15 in b98cc28

_SAMPLING_EPS = 1e-5

vllm/vllm/sampling_params.py

Lines 254 to 259 in b98cc28

	if self.temperature < _SAMPLING_EPS:
	# Zero temperature means greedy sampling.
	self.top_p = 1.0
	self.top_k = -1
	self.min_p = 0.0
	self._verify_greedy_sampling()

[hmellor]

Looks like this was fixed in #2937 with:

vllm/vllm/sampling_params.py

Line 16 in b98cc28

_MAX_TEMP = 1e-2

vllm/vllm/sampling_params.py

Lines 220 to 225 in b98cc28

	if 0 < self.temperature < _MAX_TEMP:
	logger.warning(
	"temperature %s is less than %s, which may cause numerical "
	"errors nan or inf in tensors. We have maxed it out to %s.",
	self.temperature, _MAX_TEMP, _MAX_TEMP)
	self.temperature = max(self.temperature, _MAX_TEMP)