The identical paper is not recognized as duplicate

[bernhard-kleine]

JabRef 4.0.0-dev--snapshot--2017-04-03--master--b45575649
Windows 7 6.1 amd64
Java 1.8.0_121

Steps to reproduce:
0. open PMID 21497511 on Pubmed

1. use Jabfox/Zotero for importing
2. While in Jabref, click on the DOI symbol to open the paper on the journals home page.
3. Import the article via Jabfox/Zotero
4. While Authors, Journal, Volume, pages, DOI are all equal between the two entrys, the article, however, is not labelled as duplicate in Jabrefs import dialogue and furthermore not found with Quality->Find Duplicates.

This is indeed very strange. I wonder what Jabref now considers as duplicate of an entry.

Put the excerpt of the log file here

[Siedlerchr]

Maybe related to #2687 @lynyus Maybe you have an idea

[bernhard-kleine]

Maybe, but the difference to that issue is that the papers there are not identical while here it is the very same paper Jabref acted upon as not identical. The question here what makes them not duplicates, there why are the PhysRev not distinguished. Maybe in both cases what makes paper similar or dissimilar.

[stefan-kolb]

@bernhard-kleine Can you post the BibTeX code for both entries please.

[matthiasgeiger]

Crossref:

@Article{Schwartz_2011,
  author    = {Daniel R. Schwartz and Mitchell A. Lazar},
  title     = {Human resistin: found in translation from mouse to man},
  journal   = {Trends in Endocrinology {\&} Metabolism},
  year      = {2011},
  month     = {apr},
  doi       = {10.1016/j.tem.2011.03.005},
  publisher = {Elsevier {BV}},
}

vs. Pubmed:

@Article{,
  author          = {Schwartz, Daniel R and Lazar, Mitchell A},
  title           = {Human resistin: found in translation from mouse to man.},
  journal         = {Trends in endocrinology and metabolism: TEM},
  year            = {2011},
  volume          = {22},
  pages           = {259--265},
  month           = jul,
  issn            = {1879-3061},
  abstract        = {The discovery of resistin 10 years ago as a fat cell-secreted factor that modulates insulin resistance suggested a link to the current obesity-associated epidemics of diabetes and cardiovascular disease, which are major human health concerns. Although adipocyte-derived resistin is indisputably linked to insulin resistance in rodent models, the relevance of human resistin is complicated because human resistin is secreted by macrophages rather than adipocytes, and because of the descriptive nature of human epidemiology. In this review, we examine the recent and growing evidence that human resistin is an inflammatory biomarker and a potential mediator of diabetes and cardiovascular disease.},
  chemicals       = {Inflammation Mediators, RETN protein, human, Resistin, Retn protein, mouse},
  citation-subset = {IM},
  completed       = {2011-11-08},
  country         = {United States},
  created         = {2011-07-05},
  doi             = {10.1016/j.tem.2011.03.005},
  issn-linking    = {1043-2760},
  issue           = {7},
  keywords        = {Adipocytes, White, immunology, metabolism, secretion; Animals; Cardiovascular Diseases, metabolism; Diabetes Mellitus, Type 2, metabolism; Humans; Inflammation Mediators, blood, metabolism; Insulin Resistance; Macrophages, immunology, metabolism, secretion; Mice; Obesity, metabolism; Resistin, blood, metabolism, secretion; Species Specificity},
  mid             = {NIHMS289392},
  nlm             = {PMC3130099},
  nlm-id          = {9001516},
  owner           = {NLM},
  pii             = {S1043-2760(11)00049-X},
  pmc             = {PMC3130099},
  pmid            = {21497511},
  pubmodel        = {Print-Electronic},
  pubstatus       = {ppublish},
  revised         = {2017-02-20},
}

Problem seems to be that title, author and various other fields are sligthly different... However, as they have the same DOI this should generally be sufficient to indicate a duplicate?

[bernhard-kleine]

Sorry for being late I thought the answer from matthias was sufficient.

testj.bib.txt

By the way you should add .bib to the list of uploadable file types.

[stefan-kolb]

Ok, so here is an evaluation of the scenario.

The two entries were not considered equal because:

• first all required fields for article are compared and contribute to the equality score if they match to 0.8 percent correlated by words
• journal field is not considered equal as its equality is just below 0.8 threshold.
• the overall score is 0.76
• within a 0.05 threshold to the equality threshold of 0.75 also the optional fields are evaluated
• naturally, the optionally fields make both entries look different as one has a lot of optional fields and the other does not or they are different

To fix this we need to implement a better duplicate algorithm which is not that easy and a lot of work.

For this case i have added a step that checks if an identifier like the DOI is equal and considers them equal immediately.

[stefan-kolb]

Thank you for your report 👍
This should be fixed in current master. Please try the latest build from http://builds.jabref.org/master.

[bernhard-kleine]

Unfortunate, now there is an issue with unrelated entries identified as duplicates.
I installed the snapshot of today. And imported the two entries again. They were seen as duplicates. That is ok. However, then I started to do the Quality->Find duplicates on the 760+ entry bibfile where so far no duplicates were found. Totally unrelated entries were identified as duplicates. I show an example in the two attachments. As you will see, DOI, Authors, volume. pages, etc are different betwenn the two entries, only Journaltitle and Issn are similar. That is definitive wrong. Please try again. This is obviously not the correct solution
PMID 27294923 and PMID 25569080 are not the same.