Cleaning up stale parts

[ruslandoga]

👋

Just wanted to quickly check if chdb automatically cleans stale MergeTree parts after optimize table commands. And if so, how does it work?

[auxten]

No, we haven't do this. To be fixed

[ruslandoga]

In that case this chat with GPT that explains the naming might be useful: https://chat.openai.com/share/94a7fa2d-5f73-4f9f-b2ba-681ed324ae35

[ruslandoga]

Seems like we can find the part with the highest LEVEL, check its MAX_BLOCK_NUMBER, and delete all parts with MAX_BLOCK_NUMBER less than that.

[auxten]

There should be some code in clickhouse doing that in background thread pool. We just need to find it out, run it immediately on Optimize. Any interest to make a patch? @ruslandoga

[ruslandoga]

Yes, I'm interested and will try doing it today! :)

[ruslandoga]

The logic seems to be in MergeTreeData::clearOldPartsFromFilesystem but I'm a bit stuck at compiling ClickHouse, it's already taking over four hours and I'm at [7671/8349] Building CXX object src/CMakeFiles/dbms.dir/Interpreters/DatabaseAndTableWithAlias.cpp.o 😅

I'm new to C++ so I wonder if there is a faster way to build ClickHouse / chdb just for tests?

[lmangani]

Hey @ruslandoga I'm afraid the first build is as painful as you're experiencing (our action takes > 5h to build) but if you add ccache to the mix the next compile and link rounds with minor modifications will be much faster. If you're on our discord feel free to ping on the dev channel and we'll try to assist.

[ruslandoga]

I tried to find your discord server but failed :)

[auxten]

I tried to find your discord server but failed :)

Imangani == qxip

[ruslandoga]

Thank you @auxten, but google doesn't return anything for "qxip discord" either.

[auxten]

Here are we: https://discord.com/channels/1098133460310294528/1125668654965604422