feat: Add scalar quantization dataset #97

ahuber21 · 2025-03-21T14:36:33Z

This PR adds a new class SQDataset which implements global scalar quantization.

examples/cpp/vamana.cpp

This reverts commit be95013.

Co-authored-by: ibhati <[email protected]>

dian-lun-lin · 2025-04-10T17:55:56Z

We need to make SQDataset resizable in order to make it work on dynamic vamana index. Dynamic vamana index requires a dataset to be resizable as this index will call compact and resize. (see this). Please implement the two functions in SQDataset. I think calling data_.compact() and data_.resize() internally should be enough.

For reference, see the SimpleData compact and resize implementation.

razdoburdin · 2025-04-29T08:35:10Z

I have took a look at ARM CI failure. It looks like some problem with EVE. Is it possible to update EVE to the last stable version ? There is a chance, that the problem is already solved on there side.

Please go ahead and update EVE. As long as it does not break our existing codebase, it's fine

I have created PR to dev branch: #117

This reverts commit d18182c.

ahuber21 · 2025-04-29T10:02:25Z

Thank you @razdoburdin for the PR against this branch. I've checked my usage of eve and I don't think I should have used it in the fix_argument() function. I've replaced it with a simple std::reduce() and thereby removed the necessity to #include eve/algo.h, which hopefully fixes the build error on ARM.

@ibhati @dian-lun-lin as discussed I reverted the re-compression changes (and the check for trivial data). My last action for this PR would be clean up the RNG generator in scalar.cpp, which I'm hoping to finish before your AM so that we can merge quickly.

…antization

ibhati · 2025-04-30T14:44:39Z

include/svs/quantization/scalar/scalar.h

+        // Thread-local accumulators
+        std::vector<MinMaxAccumulator> tls(threadpool.size());
+
+        // Compute mean and squared sum


Suggested change

// Compute mean and squared sum

// Compute min and max values in dataset

ibhati · 2025-04-30T14:45:20Z

include/svs/quantization/scalar/scalar.h

+    }
+};
+
+// operator to find global min and max in dataset


Suggested change

// operator to find global min and max in dataset

// Operator to find global min and max in dataset

ibhati · 2025-04-30T14:45:59Z

include/svs/quantization/scalar/scalar.h

+    }
+};
+
+// operator to compress a dataset using a threadpool


Suggested change

// operator to compress a dataset using a threadpool

// Operator to compress a dataset using a threadpool

ibhati · 2025-04-30T14:56:45Z

include/svs/quantization/scalar/scalar.h

+        // Compute mean and squared sum
+        threads::parallel_for(
+            threadpool,
+            threads::DynamicPartition(data.size(), batch_size),


Where did we get this batch_size value from? If there is no strong reason for using DynamicParition, we can simply use StaticPartition as used in most of the code. See an example here for instance: https://github.com/intel/ScalableVectorSearch/blob/main/include/svs/index/vamana/index.h#L553

Can we use StaticPartion here as well?

ibhati · 2025-04-30T15:01:42Z

include/svs/quantization/scalar/scalar.h

+            std::move(compressed), scale, bias};
+    }
+
+    /// @brief Compact the dataset


Just to keep in mind we need to dd the Doxygen docstrings for the other functions and some more documentation for SQDataset in next PR

ibhati · 2025-04-30T15:02:38Z

tests/integration/vamana/scalar_build.cpp

+    CATCH_REQUIRE(svs_test::prepare_temp_directory());
+    size_t num_threads = 2;
+
+    // use uncompressed reference results which should be identical


Suggested change

// use uncompressed reference results which should be identical

// Use uncompressed reference results which should be identical

ibhati · 2025-04-30T15:07:28Z

tests/svs/quantization/scalar/scalar.cpp

+    // Scale is calculated from (max_data - min_data) / (max_quant - min_quant)
+    // The dataset features values [-127, 127], the quantization range is given by the MIN
+    // and MAX elements of the provided type.
+    constexpr float MIN = std::numeric_limits<T>::min();


Is not the minimium value for int8 -128 instead of -127?

Yes, but the dataset is symmetric, not sure why, but it doesn't cause issues.

ibhati · 2025-04-30T15:08:23Z

tests/svs/quantization/scalar/scalar.cpp

+    // and MAX elements of the provided type.
+    constexpr float MIN = std::numeric_limits<T>::min();
+    constexpr float MAX = std::numeric_limits<T>::max();
+    constexpr float exp_scale = 254.0F / float(MAX - MIN);


Suggested change

constexpr float exp_scale = 254.0F / float(MAX - MIN);

constexpr float exp_scale = 255.0F / float(MAX - MIN);

No the dataset min max max values are -127 and 127, respectively. I could have used min() and max(), but figured I would use this knowledge about our test data in the test case.

tests/svs/quantization/scalar/scalar.cpp

ibhati · 2025-04-30T15:11:51Z

tests/svs/quantization/scalar/scalar.cpp

+    // Calculations are performed in float everywhere and should therefore produce the exact
+    // same results
+    CATCH_REQUIRE(sq_dataset.get_scale() == exp_scale);
+    CATCH_REQUIRE(sq_dataset.get_bias() == exp_bias);


Why are these checks not failing if the int8 minimum is -128? Not sure

Yeah as explained, these are the values that are not determined from the compression range but rather from data. So all good!

ibhati

Looks good, just see if we can use StaticParition for the other instance as well

ibhati · 2025-04-30T21:56:17Z

include/svs/quantization/scalar/scalar.h

+        // Compute mean and squared sum
+        threads::parallel_for(
+            threadpool,
+            threads::DynamicPartition(data.size(), batch_size),


Can we use StaticPartion here as well?

ahuber21 · 2025-05-05T10:25:38Z

@ibhati everything should be there now. Please update your "change requested" if you think we're ready to merge.

ibhati

Thanks @ahuber21 for enabling this capability in SVS.

ibhati · 2025-05-05T15:54:15Z

/intelci

Initial version of SQDataset

0b8f7e8

ahuber21 force-pushed the dev/ahuber/default-quantization branch from 075f55f to 0b8f7e8 Compare March 21, 2025 14:51

ahuber21 added 10 commits March 21, 2025 10:16

more dynamic max error calculation

9314910

clean up tests

254cceb

refactor to a single scalar.h

d464e23

revert some changes

5a45645

remove _impl file

01b3d80

Add compressed distances

a386aaa

add compressed distances

083a9b8

refactor: move some helpers to detail namespace

ba8e71e

adding more to compressed distance metrics

247e8d2

template compression data type

1ba7c2e

ibhati reviewed Mar 27, 2025

View reviewed changes

examples/cpp/vamana.cpp Outdated Show resolved Hide resolved

ahuber21 and others added 6 commits March 28, 2025 01:49

Use linear transformation s*v+b for compression

be95013

Revert "Use linear transformation s*v+b for compression"

f2e8f0d

This reverts commit be95013.

Compute everything in float

3c1811e

refactor directories and concepts

21bae3c

fix compressed distance calculations

9051128

Update examples/cpp/vamana.cpp

e8b9284

Co-authored-by: ibhati <[email protected]>

ahuber21 added 10 commits April 17, 2025 02:35

make dataset resizeable

1ea1894

add first distance computation checks

13062e1

evaluate IP with SQDataset

e6916d4

unit test for all distances with updated reference calculations

20c6f9f

fix argument for reference distance

13f20f8

dedicated test for compressed queries

d0cec00

add first set of vamana search integration tests

d7e4e55

add first set of vamana search integration tests

283cfa1

fix reload test

6233152

passing scalar build tests

b6df7f8

ahuber21 added 3 commits April 29, 2025 02:57

remove eve

3aeb779

Revert "set_datum: recalculate scale and bias if necessary"

0530b07

This reverts commit d18182c.

fix MinMax initial values

7f70373

ahuber21 added 7 commits April 29, 2025 03:03

fix license

91475d3

Merge remote-tracking branch 'origin/main' into dev/ahuber/default-qu…

be39340

…antization

improve RNG generation to avoid precision loss

d998692

clean up comments

9206310

final newline

58e999a

clean up

00ba9a2

newline

5b8b0cf

ibhati requested a review from dian-lun-lin April 30, 2025 14:29

ahuber21 added 2 commits April 30, 2025 07:52

Merge remote-tracking branch 'origin/main' into dev/ahuber/default-qu…

0d009fa

…antization

further clean up comments

0dc38e8

ibhati requested changes Apr 30, 2025

View reviewed changes

ahuber21 added 4 commits April 30, 2025 09:40

BatchIterator updates

007da9e

Review suggestions

f2aa689

fix staticpartition

605f39b

update expected recall

84b633f

ahuber21 requested a review from ibhati April 30, 2025 21:06

ibhati reviewed Apr 30, 2025

View reviewed changes

rfsaliev mentioned this pull request May 5, 2025

Fix SVS factory issue on non-SVS and non-LVQ platforms [MOD-9413] RedisAI/VectorSimilarity#664

Merged

2 tasks

dian-lun-lin and others added 2 commits May 5, 2025 10:43

Solve compression issue on ARM (#119)

7d91f77

static partition for MinMax

5481b68

ibhati approved these changes May 5, 2025

View reviewed changes

ahuber21 merged commit 94e24ba into main May 6, 2025
18 checks passed

ahuber21 deleted the dev/ahuber/default-quantization branch May 6, 2025 07:09

	// Compute mean and squared sum
	// Compute min and max values in dataset

	// operator to find global min and max in dataset
	// Operator to find global min and max in dataset

	// operator to compress a dataset using a threadpool
	// Operator to compress a dataset using a threadpool

	// use uncompressed reference results which should be identical
	// Use uncompressed reference results which should be identical

	constexpr float exp_scale = 254.0F / float(MAX - MIN);
	constexpr float exp_scale = 255.0F / float(MAX - MIN);

feat: Add scalar quantization dataset #97

feat: Add scalar quantization dataset #97

Uh oh!

Conversation

ahuber21 commented Mar 21, 2025

Uh oh!

Uh oh!

dian-lun-lin commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

razdoburdin commented Apr 29, 2025

Uh oh!

ahuber21 commented Apr 29, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ibhati left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ahuber21 commented May 5, 2025

Uh oh!

ibhati left a comment

Choose a reason for hiding this comment

Uh oh!

ibhati commented May 5, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dian-lun-lin commented Apr 10, 2025 •

edited

Loading