Unecessary library modification notifications when saving entries with duplicate keywords

[thiagocferr]

JabRef version

Latest development branch build (please note build date below)

Operating system

GNU / Linux

Details on version and operating system

JabRef 5.8, 27-09-2022 14:57,on Ubuntu 22.04.1 LTS with GNOME 42.4

Checked with the latest development build

• I made a backup of my libraries before testing the latest development version.
• I have tested the latest development version and the problem persists

Steps to reproduce the behaviour

This bug is somewhat related to issue #9159, but for a specific case: when an entry contains a keywords field with duplicate text.

How to reproduce

1. Fetch a new entry containing keywords field with duplicate (i.e. DOI 10.48550/arXiv.1103.1201)
2. Save library without any modifications
3. A "The library has been modified by another program." prompt will appear, and "review changes" will show the new added entry.

Possible fix (?)

Now, I don't really understand how this could possibly fix this issue, but this prompt does disappear whenever I simply remove the duplicate field from the equation.

For example, when importing from DOI 10.48550/arXiv.1103.1201 (by simply copy-pasting into the Main Table), we get an entry with keywords equal to "Differential Geometry (math.DG), FOS: Mathematics, FOS: Mathematics, 53C10, 53C2, 53C38". Note how the FOS: Mathematics part appears twice. Saving on this state leads to this issue's bug, but simply deleting one of them (i.e. keywords="Differential Geometry (math.DG), FOS: Mathematics, 53C10, 53C2, 53C38") AND THEN saving seems to bypass this issue altogether.

NOTE: this will not work if you first imported and saved with this duplication, proceed to ignore the "review stage" (i.e. not accepting the "changes") and then re-import and save without this duplication (or even any change to the field, for that matter). You could just accept the "changes" prior to testing the "fix" (or do so programmatically).

I've looked a bit into of the code that surrounds the invocation of this prompt, and I did find massive differences between the internal and file databases, but beyond that I could not make sense of the reason for this issue.

I found this "fix" while developing feature #9170, since I ended up generating BibEntries that included keywords with duplication, imported from ArXiv-issued DOIs entries (like the previous one). I ended up simply removing all duplication from this field before returning from the new fetcher, and the problem seemed to be gone.

Appendix

...

Log File

Paste an excerpt of your log file here

[Siedlerchr]

I can also reproduce this when manually adding a duplicate keyword and then hitting save.
Edit: I think I found the cause.
JabRef writes all keywords, duplicate ones as well but when opening/parsing it has no dupes.

[Siedlerchr]

@thiagocferr You might be interested, I was finally able to pinpoint the root cause. Saving and parsing itself are acutally fine, it's a special migration from keyword to special fields that does not take duplicates into account (it internally ues the entry's keyword set)