Would use of ctags help with the embedding?

[bbli]

Hi there, I know the aider project(https://aider.chat/) uses ctags to create its repomap. So I am wondering what the strategy is here and whether it would help with the embedding. I don't know too much about how chromaDB works under the hood, so posting this as an open question

[Davidyz]

When I started programming, LSP was already a thing (as well as treesitters), so I never really learnt much about ctags. As far as I know, ctags is a way to generate symbol outlines that can be used for other tools (completions, etc.). Since this project already uses treesitter for chunking (splitting large documents into smaller pieces so that embedding models can process without truncation), I think I might be able to implement some sort of node filters that achieve this without introducing extra dependencies for ctags. This would also reduce the number of embeddings for large files, which will make queries faster.

The risk of this, though, is that the embedding may lose information about implementation-level details, which IMO is also important (if all you need is a symbol map, you can do that with LSP inside neovim).

[bbli]

already uses treesitter for chunking (splitting large documents into smaller pieces so that embedding models can process without truncation)

Ohh, if that's the case, why does the vectorise command have options for the chunk size/overlap?

I see. Thanks for the insights!

[Davidyz]

Ohh, if that's the case, why does the vectorise command have options for the chunk size/overlap?

The chunk size is there because the embedding model is configurable. If you use an embedding model with a larger context window, you'd want to split a document into fewer chunks.

The overlap ratio is there for files that Treesitter can't handle. This can be a programming language not covered by the Python tree-sitter-language-pack or a long text file that simply isn't structured in any way. In this case, having an overlap between chunks helps avoid splitting mid-sentence.

Also, both of these options were introduced back when tree-sitter chunking hadn't been implemented.

[bbli]

Ok. So if I pass those options, vectorcode will still use tree-sitter as the default chunk strategy?

[Davidyz]

Ok. So if I pass those options, vectorcode will still use tree-sitter as the default chunk strategy?

Yes. It uses Treesitter as long as it finds a suitable parser (based on file name extension). If not, it falls back to a naive chunker, which simply counts the number of characters and yields a chunk at chunk_size. The overlapping only applies to the naive chunker.