Migrate fetchers to Search.g4 ANTLR parser. #13691

turhantolgaunal · 2025-08-13T18:34:28Z

Closes #13607

Changed inherited fetcher classes to use ANTLR generated classes instead of lucene libraries.
Changed ACMPortalFetcher.java logic for transforming the parsed syntax to URL

TO DO:

Replace the logic in other fetcher classes to transform the parsed nodes into URL calls.
Modify other fetcher classes to correctly override changed inherited functions.

Steps to test

Using the Search.g4 syntax for searching on the web with different options.

Documentation issue JabRef/user-documentation#584

Mandatory checks

I own the copyright of the code submitted and I license it under the MIT license
Change in CHANGELOG.md described in a way that is understandable for the average user (if change is visible to the user)
Tests created for changes (if applicable)
Manually tested changed features in running JabRef (always required)
[/] Screenshots added in PR description (if change is visible to the user)
Checked developer's documentation: Is the information available and up to date? If not, I outlined it in this pull request.
Checked documentation: Is the information available and up to date? If not, I created an issue at https://github.com/JabRef/user-documentation/issues or, even better, I submitted a pull request to the documentation repository.

-Changed inherited fetcher classes to use ANTLR generated classes instead of lucene libraries. - Changed ACMPortalFetcher.java logic for transforming the parsed syntax to URL

jablib/src/main/java/org/jabref/logic/importer/fetcher/ACMPortalFetcher.java

calixtus · 2025-08-14T09:59:44Z

Great start, to me it looks like your on the right track.

- Changed ACMPortalFetcherTest unit test code to use Search.g4 generated classes instead of Lucene - Removed trivial comment from ACMPortalFetcher

- Changed AbstractQueryTransformer methods to obey Search.g4 parser rules - Modified ACMPortalFetcher to use the changed transformer logic

…rrectly

…ew interface

…d the search based fetcher classes to use it

…de while still being compatible with Search.g4 parser

jablib/src/main/java/org/jabref/logic/importer/SearchBasedFetcher.java

jablib/src/test/java/org/jabref/logic/importer/fetcher/ArXivFetcherTest.java

…logic

jablib/src/test/java/org/jabref/logic/importer/fetcher/BvbFetcherTest.java

…r and node logic

…e logic

jablib/src/test/java/org/jabref/logic/importer/fetcher/DOABFetcherTest.java

jablib/src/main/java/org/jabref/logic/importer/fetcher/DOAJFetcher.java

… new parser and node logic

…the new Search.g4 logic. Fixed how AbstractQueryTransformer handles fields.

…parser

…osearchg4

CHANGELOG.md

jablib/src/main/java/org/jabref/logic/importer/fetcher/INSPIREFetcher.java

jablib/src/main/java/org/jabref/logic/importer/fetcher/JstorFetcher.java

Siedlerchr

cool thanks for the work on this!

Siedlerchr · 2025-08-25T17:43:59Z

What about the todos in the PR, are they addressed? Do you plan to address them now or in a follow up?

trag-bot · 2025-08-25T20:24:18Z

@trag-bot didn't find any issues in the code! ✅✨

calixtus · 2025-08-25T20:35:24Z

Documentation issue is JabRef/user-documentation#584

calixtus · 2025-08-25T20:35:47Z

Tested it, works fine. ❤️

calixtus

Thank you very much for this PR. Yes it was a lot of work and a hell of a ride through the search subsystem, but I hope it was worth it for you. It is a great addition to our codebase and we appreciate it very much!

turhantolgaunal · 2025-08-26T15:07:16Z

What about the todos in the PR, are they addressed? Do you plan to address them now or in a follow up?

I had added them when I opened the pr request as a draft and when the work was incomplete, they are done.

turhantolgaunal · 2025-08-26T15:41:52Z

Thank you very much for this PR. Yes it was a lot of work and a hell of a ride through the search subsystem, but I hope it was worth it for you. It is a great addition to our codebase and we appreciate it very much!

Thank you! I really appreciate you saying that. It was definitely a challenge but I think it gave me valuable experience.

koppor · 2025-08-31T15:40:25Z

The task was not completely described.

The query validator on the UI seems to use a different validation.

@turhantolgaunal Do you think, you can take care of this?

turhantolgaunal · 2025-08-31T17:39:18Z

The task was not completely described.

The query validator on the UI seems to use a different validation.

@turhantolgaunal Do you think, you can take care of this?

Ok, I will look into it

* upstream/main: (32 commits) Fix path (#13769) Mode aware consistency check (#13584) Refine JBang check (#13765) Add Language Server to the UI and add the integrity/consistency check (#13697) Fix/remove comment code (#13763) New Crowdin updates (#13760) Bump org.openrewrite.rewrite from 7.14.0 to 7.14.1 (#13757) Bump com.autonomousapps:dependency-analysis-gradle-plugin (#13756) Bump dev.langchain4j:langchain4j-bom from 1.2.0 to 1.3.0 in /versions (#13755) Bump jablib/src/main/resources/csl-locales from `fa56de1` to `e29c453` (#13754) Bump com.autonomousapps:dependency-analysis-gradle-plugin (#13753) Bump org.mockito:mockito-core from 5.18.0 to 5.19.0 in /versions (#13752) Bump actions/upload-pages-artifact from 3 to 4 (#13751) Migrate fetchers to Search.g4 ANTLR parser. (#13691) [Junie]: fix: resolve IllegalArgumentException for non-absolute URIs (#13669) Add auto-renaming of linked files on entry data change (#13295) Walkthrough additions (#13745) Switch from zulu to corretto (#13749) New Crowdin updates (#13747) Fix copy to (#13741) ...

pr: migrate fetchers to Search.g4 ANTLR parser.

b02fbde

-Changed inherited fetcher classes to use ANTLR generated classes instead of lucene libraries. - Changed ACMPortalFetcher.java logic for transforming the parsed syntax to URL

trag-bot bot reviewed Aug 13, 2025

View reviewed changes

jablib/src/main/java/org/jabref/logic/importer/fetcher/ACMPortalFetcher.java Outdated Show resolved Hide resolved

calixtus changed the title ~~pr: migrate fetchers to Search.g4 ANTLR parser.~~ Migrate fetchers to Search.g4 ANTLR parser. Aug 14, 2025

calixtus added dev: code-quality Issues related to code or architecture decisions component: fetcher labels Aug 14, 2025

turhantolgaunal added 8 commits August 14, 2025 20:32

pr: migrate fetchers to Search.g4 ANTLR parser.

9892bec

- Changed ACMPortalFetcherTest unit test code to use Search.g4 generated classes instead of Lucene - Removed trivial comment from ACMPortalFetcher

pr: migrate fetchers to Search.g4 ANTLR parser.

d802521

- Changed AbstractQueryTransformer methods to obey Search.g4 parser rules - Modified ACMPortalFetcher to use the changed transformer logic

Added a new interface for search nodes in order to parse operators co…

a645b38

…rrectly

Added 2 new node types and changed SearchQueryNode to implement the n…

44853c3

…ew interface

Created a new visitor class for parsing the search syntax and modifie…

a643d22

…d the search based fetcher classes to use it

Updated AbstractQueryTransformer to be more in line with the older co…

c890e92

…de while still being compatible with Search.g4 parser

Updated ACMPortalFetcher to use the new parser and node logic

954e1a4

Updated ArxivPortalFetcher to use the new parser and node logic

3977901

trag-bot bot reviewed Aug 17, 2025

View reviewed changes

jablib/src/main/java/org/jabref/logic/importer/SearchBasedFetcher.java Outdated Show resolved Hide resolved

trag-bot bot reviewed Aug 17, 2025

View reviewed changes

jablib/src/test/java/org/jabref/logic/importer/fetcher/ArXivFetcherTest.java Show resolved Hide resolved

Updated BvbFetcher and BvbFetcherTest to use the new parser and node …

60d2034

…logic

trag-bot bot reviewed Aug 17, 2025

View reviewed changes

jablib/src/test/java/org/jabref/logic/importer/fetcher/BvbFetcherTest.java Show resolved Hide resolved

turhantolgaunal added 2 commits August 18, 2025 00:26

Updated DBLPFetcher and DBLPQueryTransformerTest to use the new parse…

2f9ea97

…r and node logic

Updated DOABFetcher and DOABFetcherTest to use the new parser and nod…

f3d40af

…e logic

trag-bot bot reviewed Aug 17, 2025

View reviewed changes

jablib/src/test/java/org/jabref/logic/importer/fetcher/DOABFetcherTest.java Show resolved Hide resolved

Updated DOAJFetcher to use the new parser and node logic

184a721

trag-bot bot reviewed Aug 18, 2025

View reviewed changes

jablib/src/main/java/org/jabref/logic/importer/fetcher/DOAJFetcher.java Show resolved Hide resolved

turhantolgaunal added 7 commits August 18, 2025 22:02

Updated GoogleScholar and SearchBasedFetcherCapabilityTest to use the…

3ce8865

… new parser and node logic

Updated GvkFetcher GvkFetcherTest and GVKQueryTransformerTest to use …

b28df9c

…the new Search.g4 logic. Fixed how AbstractQueryTransformer handles fields.

Updated IEEE and IEEEQueryTransformerTest to use Search.g4 based parser

9c978dc

Updated InfixTransformerTest to use Search.g4 based parser.

60b6db9

Updated INSPIREFetcher to use Search.g4 based parser.

98aa35f

Updated ISIDOREFetcher and ISIDOREFetcherTest to use Search.g4 based …

efcd08f

…parser

Updated JstorFetcher to use Search.g4 based fetcher

45245b8

turhantolgaunal added 4 commits August 23, 2025 22:45

Merge remote-tracking branch 'origin/fetchertosearchg4' into fetchert…

e2f2ae2

…osearchg4

Changed how "not comparison expressions" are handled

138746f

Added the change to CHANGELOG.md

e83c1fe

Merge branch 'main' into fetchertosearchg4

a95b3d0

turhantolgaunal marked this pull request as ready for review August 24, 2025 18:21

calixtus added the status: ready-for-review Pull Requests that are ready to be reviewed by the maintainers label Aug 24, 2025

calixtus reviewed Aug 24, 2025

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

trag-bot bot reviewed Aug 24, 2025

View reviewed changes

jablib/src/main/java/org/jabref/logic/importer/fetcher/INSPIREFetcher.java Show resolved Hide resolved

trag-bot bot reviewed Aug 24, 2025

View reviewed changes

jablib/src/main/java/org/jabref/logic/importer/fetcher/JstorFetcher.java Show resolved Hide resolved

Update CHANGELOG.md

b8c190f

Siedlerchr previously approved these changes Aug 25, 2025

View reviewed changes

Reword

9f4953d

calixtus dismissed Siedlerchr’s stale review via 9f4953d August 25, 2025 20:13

Merge branch 'main' into fetchertosearchg4

668a1d4

calixtus approved these changes Aug 25, 2025

View reviewed changes

calixtus added this pull request to the merge queue Aug 25, 2025

Merged via the queue into JabRef:main with commit 768c86a Aug 25, 2025
1 check passed

github-actions bot mentioned this pull request Aug 25, 2025

Migrate Fetcher to Search.g4 #13607

Closed

turhantolgaunal mentioned this pull request Aug 29, 2025

Renamed DefaultLuceneQueryTransformer class to DefaultSearchQueryTran… #13787

Merged

6 tasks

turhantolgaunal mentioned this pull request Sep 1, 2025

Update the search syntax highlight for web search #13801

Merged

3 tasks

Uh oh!

Migrate fetchers to Search.g4 ANTLR parser. #13691

Migrate fetchers to Search.g4 ANTLR parser. #13691

Uh oh!

Conversation

turhantolgaunal commented Aug 13, 2025 • edited by calixtus Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps to test

Documentation issue JabRef/user-documentation#584

Mandatory checks

Uh oh!

Uh oh!

calixtus commented Aug 14, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Siedlerchr left a comment

Choose a reason for hiding this comment

Uh oh!

Siedlerchr commented Aug 25, 2025

Uh oh!

trag-bot bot commented Aug 25, 2025

Uh oh!

calixtus commented Aug 25, 2025

Uh oh!

calixtus commented Aug 25, 2025

Uh oh!

calixtus left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

turhantolgaunal commented Aug 26, 2025

Uh oh!

turhantolgaunal commented Aug 26, 2025

Uh oh!

koppor commented Aug 31, 2025

Uh oh!

turhantolgaunal commented Aug 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

turhantolgaunal commented Aug 13, 2025 •

edited by calixtus

Loading