ValueError : 'grc_Cprt' is not a valid language code.

[pmachapman]

When attempting to run a pre-translation build for the following translation engine via Serval on QA:

{
  "id": "65525eafbb0376665c83de75",
  "url": "/api/v1/translation/engines/65525eafbb0376665c83de75",
  "name": "65525d45fe7aa96b80ba27bf",
  "sourceLanguage": "grc",
  "targetLanguage": "en",
  "type": "Nmt",
  "isBuilding": false,
  "modelRevision": 0,
  "confidence": 0,
  "corpusSize": 0
}

I receive the following error (via the Serval build message):

ValueError : 'grc_Cprt' is not a valid language code.

grc (Ancient Greek) is the language for several Koine Greek translations in the DBL (grcF35, grcbrent, GRCTR, TCGNT, ...). It looks like IetfLanguageTag.TryGetSubtags is returning the script as Cypriot?

[johnml1135]

@ddaspit and @mshannon-sil - Why is this happening? Is it that we are inferencing only and the language codes are not added? Is it a bug introduced recently?

Here is a failing experiment, but not exactly the same one: https://app.sil.hosted.allegro.ai/projects/*/experiments/90a7ad732a9d40aa83e2b96ed7836d42/info-output/log?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=parent.name&order=-started&filter=&deep=true

https://github.com/sillsdev/machine.py/blob/d307a1bde16094927ee7fafca3a588241c94560f/machine/translation/huggingface/hugging_face_nmt_engine.py#L45C1-L57C80

machine.py/machine/translation/huggingface/hugging_face_nmt_model_trainer.py

Lines 236 to 240 in d307a1b

	if isinstance(tokenizer, MULTILINGUAL_TOKENIZERS):
	if self._src_lang is not None:
	add_lang_code_to_tokenizer(tokenizer, self._src_lang)
	if self._tgt_lang is not None:
	add_lang_code_to_tokenizer(tokenizer, self._tgt_lang)

[johnml1135]

Yes - it is about inferencing. Is the error code the best? Could we say instead "No matching training segments identified. For inferencing only, both language codes need to be in the NLLB-200 model."

2023-11-10 17:27:58,086 - machine.jobs.nmt_engine_build_job - INFO - Downloading data files
2023-11-10 17:27:58,707 - machine.jobs.nmt_engine_build_job - INFO - No matching entries in the source and target corpus - skipping training
2023-11-10 17:27:59,237 - machine.jobs.nmt_engine_build_job - INFO - Pretranslating segments
2023-11-10 17:28:09
Traceback (most recent call last):
  File "/root/.clearml/venvs-builds/3.8/code/untitled.py", line 12, in <module>
    run(args)
  File "/usr/local/lib/python3.8/dist-packages/machine/jobs/build_nmt_engine.py", line 76, in run
    raise e
  File "/usr/local/lib/python3.8/dist-packages/machine/jobs/build_nmt_engine.py", line 71, in run
    job.run(progress, check_canceled)
  File "/usr/local/lib/python3.8/dist-packages/machine/jobs/nmt_engine_build_job.py", line 84, in run
    engine = stack.enter_context(self._nmt_model_factory.create_engine())
  File "/usr/local/lib/python3.8/dist-packages/machine/jobs/huggingface/hugging_face_nmt_model_factory.py", line 71, in create_engine
    return HuggingFaceNmtEngine(
  File "/usr/local/lib/python3.8/dist-packages/machine/translation/huggingface/hugging_face_nmt_engine.py", line 57, in __init__
    raise ValueError(f"'{tgt_lang}' is not a valid language code.")
ValueError: 'lat_Latn' is not a valid language code.
2023-11-10 17:28:10
User aborted: stopping task (3)

[ddaspit]

This is intentional. If we do not fine tune NLLB and try to inference using a language code that NLLB doesn't support, then we throw this exception.

[johnml1135]

Because this error would only really happen if inferencing on the wrong language codes, let's update the error message to better reflect what is going on.