Difficult to troubleshoot reason of 4XX reponses that block BZ event queue

[leplatrem]

From November 14th to November 20th, the JBI-Prod-Core queue was disabled.

• The queue was blocked because a message led to a 422 response from JBI. In the Bugzilla UI, the message was truncated: 422 (unknown). The event was event:modify:dupe_by. I thought I took a screenshot, but can't find it now (PEBKAC)
• I deleted the message and re-enabled the queue
• There was no Sentry error (4XX error) reported
• In the GCP Log Explorer, there are no application logs around what seems to be the first occurence of the 422 response (2023-11-13T13:21:50.358619468Z). Are our debug messages being sent? Shall we log schema validation errors?

[grahamalama]

Are our debug messages being sent?

Based on other observations, seems like the answer to this is no, we aren't sending debug messages.

Shall we log schema validation errors?

Indeed we should 👍🏻.

Additionally, should we actually return 422 for these validation errors? I know we've talked about this before, but maybe it's time to revisit the conversation.