Hosted ChromaDB instance, infer the project-root automatically.

[redoxahmii]

Hello, hopefully you're doing well, and thank you for the wonderful project.

I recently got into the MCP and AI world (or extending myself from just using tab completions) and ended up on VectorCode from the documentation of codecompanion.nvim.

So far the experience has been amazing and the only issue I have right now is the retrieval part from the DB as it runs on CPU, and I am using this on an old 8th gen laptop and the repos I work on have near enough 200+ files.
How much would you estimate the performance can improve with the free tier provided by chroma themselves.

Currently, I tested with 30+ files and the response time is close to 12 secs which is a lot for only 30+ files.
I want to test and extend this as much as possible by indexing every single repo I work on and also include notes to see how far it can be pushed without the LLMs hallucinating lol.

How cumbersome would it be to use Chroma Cloud to setup a dedicated instance and connect to it?
What version of ChromaDB specifically is compatible with VectorCode right now I'm using the uv install and the version is 0.6.11.
One last thing is that do we have to always mention the specific project-root for it to only query that?

I suspect that currently when I query it, it tries to use all the project-root that have been indexed, so this might be a mistake on my mind so a little clarification on how this can be configured without having to use local scope configuration because adding the folder to .gitignore of each repo is something I'm trying to avoid.

[Davidyz]

How cumbersome would it be to use Chroma Cloud to setup a dedicated instance and connect to it?

This is something that I kept in mind when writing the db-related code, but never had the chance to really test to make sure it worked. All I tested was to host a chroma server in LAN on my workstation and connect my laptop to it. Without authentication (because it was in LAN), this is trivial to do. All you need is to open the port in your firewall and make sure the server is listening to an appropriate IP (not 127.0.0.1).

What version of ChromaDB specifically is compatible with VectorCode right now I'm using the uv install and the version is 0.6.11.

Up until now, VectorCode works with chromadb 0.6.3. The chromadb 1.0.x is being tested, but I haven't been able to get it to work on a database created by chroma 0.6.x, so I won't make the version bump just yet.

One last thing is that do we have to always mention the specific project-root for it to only query that?

If you don't specify project-root, it'll work like git. It tries to find a .git or .vectorcode directory from your cwd upwards, and if it finds a parent directory that contains either of the directories (or root markers), VectorCode'll use it as the project-root. It doesn't query all projects.

As for the performance issue, I'd suggest that you use some resource monitoring tool (if you're on linux, stuff like htop will suffice) to figure out whether you're bounded by the chromadb or the reranker model. If it's chromadb, some of the configuration options allow you to improve query time at the cost of potentially worse accuracy. See here and here. If it's the reranker model, you can either just disable it, or use hardware acceleration if you've got a GPU. To do this, you'd need to configure the following:

{
  "reranker_params": {
    "device": "cuda", // or "mps", "npu", depending on your hardware
  }
}

See more about reranker model parameters here.

[redoxahmii]

I am on a Linux machine and using htop the process seems to work alot more quicker when working directly through the cli instead of using codecomapanion.nvim in the retrieval from the nvim plugin.

ChromaDB seems to be up quick so I doubt that's the issue here.

I'm running this command to mimic what the command from the nvim plugin looks like :

time vectorcode query "Could you see what the widgets are doing and how are they being used" -n 10
vectorcode query -n 10 95.22s user 6.96s system 266% cpu 38.312 total

This is the output of the vectorcode ls :

Project Root                         Collection Size    Number of Files  Embedding Function
---------------------------------  -----------------  -----------------  ------------------------------------
~/Code/encoder421                281                 27  SentenceTransformerEmbeddingFunction

I'm not too well versed with how reranker adds to the equation here but there is definitely quite a lot of time difference when running directly from the CLI.

The laptop does have a dedicated Intel GPU and I did try using the openvino backend resulting in this error :

Exception: Using the OpenVINO backend requires installing Optimum and OpenVINO. You can install them with pip: pip install optimum[openvino].

This was after using uv tool install vectorcode[intel].

Not really sure how to install this using uv as I've never used it before (the speed was amazing btw) as I don't indulge in python much.

I'm debating now reinstalling everything with pipx to check again but that should not be the issue tbh.

One last thing i forgot to add to it is that I did use this to switch to CPU only but this was before the vectorcode[intel] command.
uv tool install --upgrade vectorcode --index https://download.pytorch.org/whl/cpu

Thank you for your time with helping!

[Davidyz]

The laptop does have a dedicated Intel GPU and I did try using the openvino backend resulting in this error :

I only have an intel igpu, and setting {"embedding_params": {"backend": "openvino"}} was sufficient to use the openvino backend (on a clean install with uv tool install vectorcode[intel]). The same applies to reranker_params.

I'm debating now reinstalling everything with pipx to check again but that should not be the issue tbh.

Just to clarify, did you have a previous VectorCode install from pipx on your system? If you did, you should uninstall it before using uv to install. uv is an alternative package manager that installs the package from the same registry (pypi.org) as pipx. The difference is that they maintain the python stuff in different directories, and create symlinks of the executable to the same dir (~/.local/bin/ by default), so unexpected things could happen if VectorCode from pipx and uv co-existed at some point.

As for the reranker, it analyses your query and try to refine the search results. The default neural network reranker (from sentence-transformers) can be slower, but it should be consistent between the codecompanion tool and the CLI query. You can set reranker to NaiveReranker to see if it helps (this uses the embeddings to sort the results, and will be faster than the defaults).