Project centric indexing and management

[vladak]

In our deployment syncing source code before indexing takes a lot time for some projects, so a scheme is used where syncing is done in parallel and then indexpart is used for individual projects and once all these partial indexes are done, one global index run is done to discover newly added projects and remove data of deleted projects. The global index takes non-trivial time to run and can be avoided by having addproject and delproject sub-commands of the OpenGrok shell script.

addproject would scan project's repos, add the project to the configuration. The indexpart could then index it and the config can be sent to the webapp (e.g. via the changes for #1355 or more fine grained Message).

delproject would remove the project from configuration (and update the webapp) and remove project's index/xref/history data. This should take care of #1307.

To make this fully functional, some other changes are needed, like #1187.

[TotoXe]

Very interested by this great feature!
We scan with OpenGrok more than 800 projects for more than 9500 repos. At the moment, I am running daily 1 indexing job but I would like using indexpart plus other features to add easily new project and delete useless ones.

[vladak]

Also, something needs to be done with the Indexes created ... timestamp in the page footer. Once a setup is fully project driven, this will cease to have significance.

[vladak]

Here's a proposal of how I think it can work:

• addproject will be a Message sent to the webapp. The webapp will add the project to the list and do the scan of project's repositories and add them to the config as well.
• delproject will also be a Message. The webapp will remove the project and its repositories from config, perform the cleanup of indexSearchers/searcherManagers. It can also try remove project's data from data root (but not the actual source code).
• to sync the config from the webapp to configuration.xml, the configuration can be downloaded from the webapp via new Message (get configuration from webapp #1426) so that indexpart can use it.
• It should be possible to send multiple addproject/delproject messages to the webapp in parallel. If needed, create new thread that will be handling the work of scanning repositories etc.

[tulinkry]

Maybe better only one message project with text "add" or "delete" and appropriate tags.

[vladak]

Yep, new project message will be introduced that has 3 flavors:

• add
• delete
• indexed (internal; to mark a project as searchable, sent automatically once indexing is done for the first time for given project)