out_loki: prevent race conditions when multiple workers use remove_keys

[edsiper]

Fixes #10560

Added per-thread removal contexts in Loki’s configuration to prevent race conditions when multiple workers use the Remove_Keys option. The change introduces a thread-local storage for these contexts and created helper functions to manage them safely

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[railsharipov]

@edsiper Thank you for the work you've done.
I am not sure if this change is the case but it is the only thing that I see that can be related: some keys not getting removed from Loki streams after I have updated to version 4.0.4. It is strange that some keys are removed while others aren't. This is just a guess so let me know if you think this change cannot be the cause.

[akuntsch]

@edsiper Thank you for the work you've done. I am not sure if this change is the case but it is the only thing that I see that can be related: some keys not getting removed from Loki streams after I have updated to version 4.0.4. It is strange that some keys are removed while others aren't. This is just a guess so let me know if you think this change cannot be the cause.

We are having the same issue where some keys are removed but many are not. Currently on version 4.1.0.

EDIT: Reverting back to 4.0.3 fixes the issue and all keys are removed again.

[LukoJy3D]

4.1 introduced some nice encoding improvements, so downgrading is not an option for us. But yeah, key removal is flaky now 🤔

[leonardo-albertovich]

Do you think you could create a repro case for this? I'd like to take a look at it but I'm a bit lost as to how to get started with Loki.

[LukoJy3D]

Did some testing, and I see two problems:

• Documentation is not explicit enough regarding remove_keys - for example, such fields as labels, label_keys, structure_metadata, etc, have to use record accessor ($), and remove_keys dont need it. It's not mentioned in the documentation, and honestly, it's counterintuitive.
• With current multi worker fix, not removing keys only a problem if workers are not used, when I enabled workers its working as intended.

[leonardo-albertovich]

It'd be great if you opened a PR in the documentation repository to make those improvements!
As for the repro, do I understand correctly that when workers is omitted then the key removal issue persists?

[LukoJy3D]

As for the repro, do I understand correctly that when workers is omitted then the key removal issue persists?

Thats the case, will try to look into it more, but with workers it's consistent and seems like more likely to appear when line_format is set to json instead of key_value