NotImplementedError is raised in static INT8 Quantization with PT2E Backend default recipe

[haitamhawa]

Following example from neural-compressor/docs/source/3x/PT_StaticQuant.md:

I get a crash in MinMaxObserver when calling prepare(model):
"NotImplementedError: MinMaxObserver's qscheme only support torch.per_tensor_symmetric and torch.per_tensor_affine."
The scheme requested prior to error is per_channel_symmetric.

i am using default StaticQuantConfig.

i think that per_channel_symmetric quantization is done on weights only.
And by using minmax algorithm, its simply absolute max for each channel, i'm not sure why observer was used in this case.

Environment:

• Linux Ubuntu 22.04.1 LTS
• Python 3.10
• torch 2.4.0+cpu
• neural_compressor 3.0

To reproduce:

import torch
from torch import nn
from neural_compressor.torch.export import export
from neural_compressor.torch.quantization import StaticQuantConfig, prepare, convert

def main():
    # Prepare the float model and example inputs for export model
    model = nn.Linear(5,5)
    example_inputs = (torch.rand(size=(1,5)),)


    # Export eager model into FX graph model
    exported_model = export(model=model, example_inputs=example_inputs)
    # Quantize the model
    quant_config = StaticQuantConfig()
    prepared_model = prepare(exported_model, quant_config=quant_config)
    # Calibrate
    for _ in range(100):
        prepared_model(torch.rand_like(example_inputs[0]))

    q_model = convert(prepared_model)
    # Compile the quantized model and replace the Q/DQ pattern with Q-operator
    from torch._inductor import config

    config.freezing = True
    opt_model = torch.compile(q_model)

if __name__=="__main__":
    main()

[yiliu30]

Hi @haitamhawa, thanks for raising this issue. Could you try to set w_granularity to per_tensor?

from neural_compressor.torch.quantization import get_default_static_config, StaticQuantConfig
quant_config = get_default_static_config()
# or 
quant_config  = StaticQuantConfig(w_granularity="per_tensor")

[haitamhawa]

It did work
But the weights now are quantized per tensor, which might cause high accuracy degradation.
Are you planning to support per_channel quantization for weights only?

[yiliu30]

It did work But the weights now are quantized per tensor, which might cause high accuracy degradation. Are you planning to support per_channel quantization for weights only?

For weights only, per-channel quantization is already supported. For example, you can quantize the model using RTN with the per-channel setting.

quant_config = RTNConfig(group_size=-1)

Model details can be found here. https://github.com/intel/neural-compressor/blob/master/docs/source/3x/PT_WeightOnlyQuant.md#common-arguments

[haitamhawa]

Sorry for the miss-understanding
i mean:

1. Weights are quantized symmetrically per_channel
2. Activation are quantized asymmetrically per_tensor

as in the default recipe of StaticQuantConfig

w_dtype: str = "int8",
w_sym: bool = True,
w_granularity: str = "per_channel",
w_algo: str = "minmax",
act_dtype: str = "uint8",
act_sym: bool = False,
act_granularity: str = "per_tensor",
act_algo: str = "minmax"

[yiliu30]

I see, I added PerChannelMinMaxObserver support on the pt2e_per_channel branch. Please try it again.

neural-compressor/test/3x/torch/quantization/test_pt2e_quant.py

Lines 100 to 126 in 6ec6178

	@pytest.mark.skipif(get_torch_version() <= TORCH_VERSION_2_2_2, reason="Requires torch>=2.3.0")
	@pytest.mark.parametrize("granularity", ["per_tensor", "per_channel"])
	def test_quantize_simple_model(self, granularity, force_not_import_ipex):
	from neural_compressor.torch.quantization import StaticQuantConfig
	model, example_inputs = self.build_simple_torch_model_and_example_inputs()
	float_model_output = model(*example_inputs)
	quant_config = None
	def calib_fn(model):
	for i in range(4):
	model(*example_inputs)
	quant_config = StaticQuantConfig(w_granularity=granularity)
	q_model = quantize(model=model, quant_config=quant_config, run_fn=calib_fn)
	from torch._inductor import config
	config.freezing = True
	q_model_out = q_model(*example_inputs)
	assert torch.allclose(float_model_output, q_model_out, atol=1e-2), "Quantization failed!"
	# test save and load
	q_model.save(
	example_inputs=example_inputs,
	output_dir="./saved_results",
	)
	from neural_compressor.torch.quantization import load

[haitamhawa]

It is working
Will this change be merged into master?

[yiliu30]

Sure, I created a PR for it, I will merge it after some tests.

#1990