[BUG] VectorCode process killed due to timeout

[edmondop]

"VectorCode process killed due to timeout." appears to me when running :lua require("vectorcode").query("mysql") or similar commands. I am running chromadb 0.6.3 in docker-compose mode using 1 core.

vectorcode query mysql -n 3  36.64s user 4.87s system 137% cpu 30.214 total

 vectorcode ls
Project Root      Collection Size    Number of Files  Embedding Function
--------------  -----------------  -----------------  ------------------------------------
~/code/my_proj                176                 88  SentenceTransformerEmbeddingFunction

sounds like a really long time, how can I troubleshoot it?

[Davidyz]

Hi, if you've got spare cpu cores it might help to allocate more cpu cores to the chromadb process because the query is compute-intensive.

Alternatively, you can also try a different embedding model that produces lower embedding dimensions (the default all-MiniLM-L6-v2 model produces 384-dimension vectors, which is already on the lower end of the spectrum). Some models that produce high-dimensional embeddings claims to perform well even if you truncate the embeddings (that is, getting rid of some dimensions so that they become low-dimensional). snowflake-arctic-embed-m-v2.0 mentioned this on their huggingface page. To use this model with truncation, you can use the following config:

{
  "embedding_params": {
    "model_name": "Snowflake/snowflake-arctic-embed-m-v2.0",
    "truncate_dim": 256,  // this truncate the embeddings to 256 dimensions
  }
}

[Davidyz]

Moving this to discussion for now. We can open a new issue if we find a specific thing (eg. a poorly written piece of code) that is slowing things down.

[edmondop]

Thank you @Davidyz I am trying to compare between the Ubuntu Machine and MacOS,

• on Macbook vectorcode query "mysql" 2.09s user 0.57s system 56% cpu 4.743 total
• on Ubuntu vectorcode query "mysql" 133.08s user 15.92s system 765% cpu 19.469 total

This is my macbook configuration

OS: macOS 15.5 24F74 arm64
Host: Mac16,5
Kernel: 24.5.0
Uptime: 3 hours, 22 mins
Packages: 89 (brew)
Shell: zsh 5.9
Resolution: 1728x1117, 1920x1080
DE: Aqua
WM: Quartz Compositor
WM Theme: Blue (Dark)
Terminal: tmux
CPU: Apple M4 Max
GPU: Apple M4 Max
Memory: 5574MiB / 65536MiB

And this is my Linux Box configuration

eporcu@devrestricted-eporcu
---------------------------
OS: Ubuntu 24.04.2 LTS x86_64
Host: m7a.4xlarge
Kernel: 6.8.0-1026-aws
Uptime: 12 days, 23 hours, 35 mins
Packages: 1568 (dpkg)
Shell: zsh 5.9
Resolution: 800x600
Terminal: /dev/pts/0
CPU: AMD EPYC 9R14 (16) @ 3.700GHz
GPU: 00:03.0 Amazon.com, Inc. Device 1111
Memory: 18756MiB / 62920MiB

What I have discovered is that there is no GPU on my machine (The above being the output of neofetch)

lspci | grep -A 10 1111
00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 (prog-if 00 [VGA controller])
        Physical Slot: 3
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        NUMA node: 0
        Region 0: Memory at c0000000 (32-bit, prefetchable) [size=4M]
        Expansion ROM at 000c0000 [disabled] [size=128K]

maybe one should just update the Readme of VectorCode suggesting not to try without GPU?

[Davidyz]

If there's no available GPU, VectorCode (or specifically, sentence transformers) won't try to use it. This means that VectorCode is probably already on the CPU. Aside from the reranker model params that I mentioned above, here are a couple of things that you can try:

1. Disable crossencoder rerankers: This is the default reranking method, which uses transformer-based models to rerank the search results. While this can improve the quality, it could also be much slower,r especially if you don't have a GPU. You can achieve this by setting "reranker": "NaiveReranker" in the JSON file.
2. Change the query multiplier: This determines the number of chunks that will be analysed by the reranker before it produces a list of the best matching files. By default, it processes all chunks from the database, which might be overwhelming for your machine. You can change this to a smaller value (a good starting point would be number of chunks divided by number of files in your collection). You can experiment with different values for multipliers in CLI via the -m flag, and once you find a sweet spot, you can set it in the config JSON too, so that it'll be effective in all queries for this collection.