ValueError: Torch var mask.3 not found in context when convert the Transforms ClipTextModel to coreml

[vecpeng]

🐞Describing the bug

ValueError: Torch var mask.3 not found in context when I convert the Transforms ClipTextEncoder model to coreml.

Stack Trace

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-52-dd4c56533b83>](https://localhost:8080/#) in <module>
     11 traced_model = torch.jit.trace(model, [input_ids, attention_mask], strict=False)
     12 
---> 13 model = ct.convert(
     14     traced_model,
     15     convert_to="mlprogram",

12 frames
[/usr/local/lib/python3.8/dist-packages/coremltools/converters/mil/frontend/torch/converter.py](https://localhost:8080/#) in __getitem__(self, torch_name)
     86             if torch_name in current_graph:
     87                 return self._current_graph[idx][torch_name]
---> 88         raise ValueError(
     89             "Torch var {} not found in context {}".format(torch_name, self.name)
     90         )

ValueError: Torch var mask.3 not found in context

To Reproduce

colab: https://colab.research.google.com/drive/1f_mvFftpCdiGDJBZnI18RzsjU0tz-YWT?usp=sharing

! pip install coremltools --pre -U  
! pip install transformers

from transformers import AutoTokenizer, CLIPTextModel
import torch
import coremltools as ct
model = CLIPTextModel.from_pretrained("openai/clip-vit-base-patch32")
tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32")

inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")

input_ids = inputs.input_ids
attention_mask = inputs.attention_mask

traced_model = torch.jit.trace(model, [input_ids, attention_mask], strict=False)

model = ct.convert(
    traced_model, 
    convert_to="mlprogram",
    inputs=[
        ct.TensorType(shape=input_ids.shape), 
        ct.TensorType(shape=attention_mask.shape)])

model.save("CLIPTextModel.mlpackage")

System environment (please complete the following information):

• coremltools version: lastest
• OS (e.g. MacOS version or Linux type): ubuntu
• Any other relevant version information (e.g. PyTorch or TensorFlow version): PyTorch 1.13.1

[TobyRoseman]

This is related to (probably a duplicate of) #1580.

@vecpeng - Have you tried this workaround?

[vecpeng]

This is related to (probably a duplicate of) #1580.

@vecpeng - Have you tried this workaround?

No，but I try another clip model and succeed.

[fukatani]

@TobyRoseman

I found this issue can be fixed in latest master.
I can finish convert CLIP model with no error.

We need return_dict=False.

from transformers import AutoTokenizer, CLIPTextModel
import torch
import coremltools as ct
model = CLIPTextModel.from_pretrained("openai/clip-vit-base-patch32", return_dict=False)
tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32")

inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")

input_ids = inputs.input_ids
attention_mask = inputs.attention_mask

traced_model = torch.jit.trace(model, [input_ids, attention_mask], strict=False)

model = ct.convert(
    traced_model,
    convert_to="mlprogram",
    inputs=[
        ct.TensorType(shape=input_ids.shape),
        ct.TensorType(shape=attention_mask.shape)])

model.save("CLIPTextModel.mlpackage")

[TobyRoseman]

@fukatani - When running your code I get the following error in the ct.convert call:

ValueError: In op, of type select, named mask.3, the named input `b` must have the same data type as the named input `a`. However, b has dtype fp32 whereas a has dtype int32.

In [2]: model = ct.convert( ...: traced_model, ...: convert_to="mlprogram", ...: inputs=[ ...: ct.TensorType(shape=input_ids.shape), ...: ct.TensorType(shape=attention_mask.shape)]) Tuple detected at graph output. This will be flattened in the converted model. huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... To disable this warning, you can either: - Avoid using `tokenizers` before the fork if possible - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) Converting PyTorch Frontend ==> MIL Ops: 0%| | 0/938 [00:00 MIL Ops: 4%|███▍ | 40/938 [00:00<00:00, 8674.88 ops/s] --------------------------------------------------------------------------- ValueError Traceback (most recent call last) Cell In[2], line 1 ----> 1 model = ct.convert( 2 traced_model, 3 convert_to="mlprogram", 4 inputs=[ 5 ct.TensorType(shape=input_ids.shape), 6 ct.TensorType(shape=attention_mask.shape)])

File /Volumes/DevData/workspace/coremltools/coremltools/converters/_converters_entry.py:530, in convert(model, source, inputs, outputs, classifier_config, minimum_deployment_target, convert_to, compute_precision, skip_model_load, compute_units, package_dir, debug, pass_pipeline)
527 if specification_version is None:
528 specification_version = _set_default_specification_version(exact_target)
--> 530 mlmodel = mil_convert(
531 model,
532 convert_from=exact_source,
533 convert_to=exact_target,
534 inputs=inputs,
535 outputs=outputs_as_tensor_or_image_types, # None or list[ct.ImageType/ct.TensorType]
536 classifier_config=classifier_config,
537 skip_model_load=skip_model_load,
538 compute_units=compute_units,
539 package_dir=package_dir,
540 debug=debug,
541 specification_version=specification_version,
542 main_pipeline=pass_pipeline,
543 )
545 if exact_target == "mlprogram" and mlmodel._input_has_infinite_upper_bound():
546 raise ValueError(
547 "For mlprogram, inputs with infinite upper_bound is not allowed. Please set upper_bound"
548 ' to a positive value in "RangeDim()" for the "inputs" param in ct.convert().'
549 )

File /Volumes/DevData/workspace/coremltools/coremltools/converters/mil/converter.py:188, in mil_convert(model, convert_from, convert_to, compute_units, **kwargs)
149 @_profile
150 def mil_convert(
151 model,
(...)
155 **kwargs
156 ):
157 """
158 Convert model from a specified frontend convert_from to a specified
159 converter backend convert_to.
(...)
186 See coremltools.converters.convert
187 """
--> 188 return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, **kwargs)

File /Volumes/DevData/workspace/coremltools/coremltools/converters/mil/converter.py:212, in _mil_convert(model, convert_from, convert_to, registry, modelClass, compute_units, **kwargs)
209 weights_dir = _tempfile.TemporaryDirectory()
210 kwargs["weights_dir"] = weights_dir.name
--> 212 proto, mil_program = mil_convert_to_proto(
213 model,
214 convert_from,
215 convert_to,
216 registry,
217 **kwargs
218 )
220 _reset_conversion_state()
222 if convert_to == 'milinternal':

File /Volumes/DevData/workspace/coremltools/coremltools/converters/mil/converter.py:286, in mil_convert_to_proto(model, convert_from, convert_to, converter_registry, main_pipeline, **kwargs)
281 frontend_pipeline, backend_pipeline = _construct_other_pipelines(
282 main_pipeline, convert_from, convert_to
283 )
285 frontend_converter = frontend_converter_type()
--> 286 prog = frontend_converter(model, **kwargs)
287 PassPipelineManager.apply_pipeline(prog, frontend_pipeline)
289 PassPipelineManager.apply_pipeline(prog, main_pipeline)

File /Volumes/DevData/workspace/coremltools/coremltools/converters/mil/converter.py:108, in TorchFrontend.call(self, *args, **kwargs)
105 def call(self, *args, **kwargs):
106 from .frontend.torch.load import load
--> 108 return load(*args, **kwargs)

File /Volumes/DevData/workspace/coremltools/coremltools/converters/mil/frontend/torch/load.py:63, in load(model_spec, inputs, specification_version, debug, outputs, cut_at_symbols, **kwargs)
55 inputs = _convert_to_torch_inputtype(inputs)
56 converter = TorchConverter(
57 torchscript,
58 inputs,
(...)
61 specification_version,
62 )
---> 63 return _perform_torch_convert(converter, debug)

File /Volumes/DevData/workspace/coremltools/coremltools/converters/mil/frontend/torch/load.py:102, in _perform_torch_convert(converter, debug)
100 def _perform_torch_convert(converter, debug):
101 try:
--> 102 prog = converter.convert()
103 except RuntimeError as e:
104 if debug and "convert function" in str(e):

File /Volumes/DevData/workspace/coremltools/coremltools/converters/mil/frontend/torch/converter.py:439, in TorchConverter.convert(self)
436 self.convert_const()
438 # Add the rest of the operations
--> 439 convert_nodes(self.context, self.graph)
441 graph_outputs = [self.context[name] for name in self.graph.outputs]
443 # An output can be None when it's a None constant, which happens
444 # in Fairseq MT.

File /Volumes/DevData/workspace/coremltools/coremltools/converters/mil/frontend/torch/ops.py:92, in convert_nodes(context, graph)
87 raise RuntimeError(
88 f"PyTorch convert function for op '{node.kind}' not implemented."
89 )
91 context.prepare_for_conversion(node)
---> 92 add_op(context, node)
94 # We've generated all the outputs the graph needs, terminate conversion.
95 if _all_outputs_present(context, graph):

File /Volumes/DevData/workspace/coremltools/coremltools/converters/mil/frontend/torch/ops.py:4244, in masked_fill(context, node)
4240 if not types.is_bool(mask.dtype):
4241 # cond must be bool type
4242 mask = mb.cast(x=mask, dtype="bool")
-> 4244 res = mb.select(cond=mask, a=value, b=x, name=node.name)
4245 context.add(res)

File /Volumes/DevData/workspace/coremltools/coremltools/converters/mil/mil/ops/registry.py:183, in SSAOpRegistry.register_op..class_wrapper..add_op(cls, **kwargs)
180 else:
181 op_cls_to_add = op_reg[op_type]
--> 183 return cls._add_op(op_cls_to_add, **kwargs)

File /Volumes/DevData/workspace/coremltools/coremltools/converters/mil/mil/builder.py:166, in Builder._add_op(cls, op_cls, **kwargs)
161 kwargs = {k: v if not isinstance(v, (list, tuple)) else v[:] for k, v in kwargs.items() if v is not None}
162 kwargs.update(cls._create_vars(
163 input_spec=op_cls.input_spec,
164 op_name=kwargs["name"], before_op=before_op,
165 candidate_kv=kwargs))
--> 166 new_op = op_cls(**kwargs)
168 # Initialize optional input Vars if it wasn't in kwargs
169 default_inputs = new_op.default_inputs()

File /Volumes/DevData/workspace/coremltools/coremltools/converters/mil/mil/operation.py:187, in Operation.init(self, **kwargs)
184 # Set inputs from kwargs
185 input_kv = {k: v for k, v in kwargs.items()
186 if k in self._input_types and v is not None}
--> 187 self._validate_and_set_inputs(input_kv)
188 self._ensure_required_inputs()

File /Volumes/DevData/workspace/coremltools/coremltools/converters/mil/mil/operation.py:495, in Operation._validate_and_set_inputs(self, input_kvs, no_check_var_types)
489 raise ValueError(
490 f"New var type {types.builtin_to_string(v_new.sym_type)} not a "
491 f"subtype of existing var type {types.builtin_to_string(v_old.sym_type)}."
492 )
493 v_old.remove_child_op(op, no_check_var_types)
--> 495 self.input_spec.validate_inputs(self.name, self.op_type, input_kvs)
497 for name, var in input_kvs.items():
498 # Remove this operation itself from existing input
499 # Var's child_ops
500 existing_input_var = self._input_vars[name]

File /Volumes/DevData/workspace/coremltools/coremltools/converters/mil/mil/input_type.py:137, in InputSpec.validate_inputs(self, op_name, op_type, candidate_kvs)
131 if not var.dtype == expected_dtype:
132 msg = (
133 "In op, of type {}, named {}, the named input {} must have the same data type "
134 "as the named input {}. However, {} has dtype {} whereas {} has dtype {}."
135 ).format(op_type, op_name, name, ref_name, name,
136 var.dtype.type_info(), ref_name, expected_dtype.type_info())
--> 137 raise ValueError(msg)
139 # Ensure candidate_kvs doesn't contain None
140 for name, var in candidate_kvs.items():

ValueError: In op, of type select, named mask.3, the named input b must have the same data type as the named input a. However, b has dtype fp32 whereas a has dtype int32.

I'm using tip of main (36544e24c23ce00c9de3c5fda9b478e0dc59e660) with:

numpy==1.21.0
torch==2.0.1
transformers==4.31.0

[fukatani]

@TobyRoseman
Sorry, that is another issue and may be fixed by #1915
Could you try with main (e0f8918baf95a55d29dcd40649c50f091ffd4114) ?

[TobyRoseman]

Confirmed; the code now executes without error. Thank @fukatani.