[High Risk]Support for immediate saving

[Kaihui-intel]

Accuracy

scheme /(opt-125m,)	format	RTN	iter>0
W4A16	auto_round	0.2882	0.3526
W2A16	auto_round		0.1657
W3A16	auto_round		0.3247
W8A16	auto_round		0.3784
bit s group_size 32	auto_round	0.3749	0.3679
bit s group_size 32	auto_gptq	0.3747	0.3658
bit s group_size 32	auto_awq	0.3749	0.3646

#788

Memory

memory check
Qwen2.5-7B-Instruct-w4g32 RTN auto_round
mprof peak
16659.441MiB->9200.250MiB ~55%

Time

quantization and saving time
opt-125m

branch	RTN	iter>0
cur branch	67s	78s
main branch	50s	88s

Qwen2.5-7B-Instruct

branch	RTN	iter>0
cur branch	4min4s	18min10s
main branch	3min54s	17min53s

immediate pacing and saving now only support formats[0]

[wenhuach21]

Thanks for the great work! Could you check the maximum RAM usage to see whether it has been reduced significantly, as expected?

[wenhuach21]

Packing immediate is set automatically before. Have you handled this when exporting >1 formats? So it’s better not to set it in the API. Besides, as discussed, set save_immediate to True.
Another thing to verify is the time cost of save_immediate, have you measured the total quantization time comparing to main branch ?

[wenhuach21]

no need to expose this arg, automatically setting this is a better way

[wenhuach21]

wrapper it as a function

[wenhuach21]

where is the code to set it to False for fake format or multiple formats

[Kaihui-intel]

auto-round/auto_round/compressors/base.py

Lines 1546 to 1562 in 5375be6

	if not hasattr(self, "formats"):
	logger.warning("this API is deprecated, please use `quantize_and_save` instead")
	else:
	# Determine if immediate packing is required
	formats = self.formats
	if (
	len(formats) == 1
	and (
	"awq" in formats[0]
	or "gptq" in formats[0]
	or "auto_round" in formats[0]
	or "gguf" in formats[0]
	or "llm_compressor" in formats[0]
	)
	and self.inplace
	):
	self.immediate_packing = True