Skip to content

Conversation

@frankmcsherry
Copy link
Member

The columnar trie datastructures used in examples/columnar.rs are grouped by key, and also need to be exchanged by (a hash of) the key. We can do this without departing from the columnar representation, which this PR introduces. It also inlines a hot function in ord_neu.rs.

@frankmcsherry
Copy link
Member Author

Qualitatively, performance ticks up on larger batching sizes where we spend relatively more time exchanging, but is harder to notice for small batch sizes when we spend most of our time maintaining arrangements (which this does not affect).

On master:

     Running `target/release/examples/columnar 10000000 1000000 -w4`
5.86842775s     loading complete
5.8684315s      loading complete
5.8684335s      loading complete
5.8684325s      loading complete
15.462424042s   queries complete
15.462429375s   queries complete
15.462428875s   queries complete
15.462431208s   queries complete
15.464183083s   shut down

vs this branch:

     Running `target/release/examples/columnar 10000000 1000000 -w4`
5.175452958s    loading complete
5.17545875s     loading complete
5.175458208s    loading complete
5.17545875s     loading complete
13.752099208s   queries complete
13.752102208s   queries complete
13.75210275s    queries complete
13.752103208s   queries complete
13.762354208s   shut down

@frankmcsherry frankmcsherry merged commit 11d7155 into TimelyDataflow:master Nov 18, 2025
5 checks passed
@frankmcsherry frankmcsherry deleted the columnar_tweaks branch November 18, 2025 00:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant