HBASE-28621 PrefixFilter should use SEEK_NEXT_USING_HINT #6361

PDavid · 2024-10-09T11:33:30Z

No description provided.

hbase-client/src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java

PDavid · 2024-10-10T14:35:07Z

hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestPrefixFilter.java

  static final char FIRST_CHAR = 'a';
  static final char LAST_CHAR = 'e';
  static final String HOST_PREFIX = "org.apache.site-";
-  static final byte[] GOOD_BYTES = Bytes.toBytes("abc");


This is also a bit unrelated - an unused constant field.

stoty · 2024-10-14T08:38:18Z

please remove the colon from the commit message / PR description

hbase-client/src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java

PDavid · 2024-10-24T14:49:36Z

hbase-client/src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java

+  private byte[] increaseLastNonMaxByte(byte[] bytes) {
+    byte[] result = Arrays.copyOf(bytes, bytes.length);
+    for (int i = bytes.length - 1; i >= 0; i--) {
+      byte b = bytes[i];
+      if (b < Byte.MAX_VALUE) {
+        result[i] = (byte) (b + 1);
+        break;
+      }
+    }
+    return result;
+  }


Unfortunately I did not yet found an existing util method which would do the same.
What I tried for example PrivateCellUtil.createFirstOnNextRow(Cell) which is similar but not the same (and tests are failing if that is used).

That's OK.
We can keep this here, or move it to PrivateCellUtil before commit.

stoty

The code looks good, I did not check the tests yet.

hbase-client/src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java

…atchFilter usage

…ed methods so that the real logic is together.

to avoid re-computing it several times in the corner case where there are a lot of cells between the hint and the first real match.

stoty · 2024-10-30T09:50:24Z

hbase-client/src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java

    // On reversed scan hint should be the prefix with last byte incremented
    byte[] reversedHintBytes = increaseLastNonMaxByte(this.prefix);
-    return PrivateCellUtil.createFirstOnRow(reversedHintBytes, 0, (short) reversedHintBytes.length);
+    this.reversedNextCellHint =


I was about to say that we could choose the direction here, but interestingly the direction is not available here yet.
Another API awkwardness.

Exactly. The direction can be influenced with setReversed() which can be invoked only after the filter creation.

So it is available but can be changed after creation.

Practically, I'm sure it's not changed after any of the Cell processing methods have been called.
It is called once when the Scan is set up. (perhaps also after reset() is called)

stoty · 2024-10-30T09:56:35Z

hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestPrefixFilter.java

+
+    assertTrue(filter.filterRowKey(afterCell));
+    assertEquals(Filter.ReturnCode.NEXT_ROW, filter.filterCell(afterCell));
+    assertTrue(filter.filterAllRemaining());


This API is so awkward...

stoty

+1 LGTM

PDavid · 2024-10-30T11:43:13Z

+1 LGTM

Many thanks. Unfortunately I still have to fix TestFilterList.testMPONE(). It will still fail but I'm on it.

Before PrefixFilter.filterRowKey() had an early return with true when the cell length was smaller than prefix. In that case the filterRow boolean field was not changed to true (while it should have). Now that this early return was removed from PrefixFilter.filterRowKey() (to be able to provide hint even when the cell length is smaller than prefix) the filterRow boolean field is properly set also in this case.

hbase-client/src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java

…Key()

hbase-client/src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java

For reverse scans, smaller/bigger is not accurate. Use before/after instead.

Apache-HBase · 2024-10-30T22:22:36Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 45s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+0 🆗	mvndep	0m 16s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 37s		master passed
+1 💚	compile	4m 43s		master passed
+1 💚	checkstyle	1m 3s		master passed
+1 💚	spotbugs	2m 34s		master passed
+1 💚	spotless	0m 54s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 42s		the patch passed
+1 💚	compile	4m 30s		the patch passed
+1 💚	javac	4m 30s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 23s		hbase-client: The patch generated 0 new + 0 unchanged - 5 fixed = 0 total (was 5)
+1 💚	checkstyle	0m 46s		hbase-server: The patch generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
+1 💚	spotbugs	3m 18s		the patch passed
+1 💚	hadoopcheck	12m 32s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 55s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 19s		The patch does not generate ASF License warnings.
		48m 36s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6361/16/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6361
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux f0d3ee8876df 5.4.0-192-generic #212-Ubuntu SMP Fri Jul 5 09:47:39 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `ba21710`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	83 (vs. ulimit of 30000)
modules	C: hbase-client hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6361/16/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-10-31T01:54:24Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 17s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+0 🆗	mvndep	0m 14s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 0s		master passed
+1 💚	compile	1m 18s		master passed
+1 💚	javadoc	0m 48s		master passed
+1 💚	shadedjars	5m 23s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 13s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 53s		the patch passed
+1 💚	compile	1m 20s		the patch passed
+1 💚	javac	1m 20s		the patch passed
+1 💚	javadoc	0m 51s		the patch passed
+1 💚	shadedjars	5m 54s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	1m 47s		hbase-client in the patch passed.
+1 💚	unit	232m 9s		hbase-server in the patch passed.
		260m 26s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6361/16/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6361
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 55f290b297e3 5.4.0-192-generic #212-Ubuntu SMP Fri Jul 5 09:47:39 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `ba21710`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6361/16/testReport/
Max. process+thread count	5011 (vs. ulimit of 30000)
modules	C: hbase-client hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6361/16/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

stoty · 2024-10-31T06:00:32Z

If there are no outstanding issues, please remove the Draft flag so that I can merge this.

Please also create a backport PR for branch-2, so that the CI tests can run on that.

PDavid · 2024-10-31T08:48:14Z

If there are no outstanding issues, please remove the Draft flag so that I can merge this.

Please also create a backport PR for branch-2, so that the CI tests can run on that.

Many thanks! I just waited for the CI build to finish. Now that the build is green, this PR is ready to review.

PDavid · 2024-10-31T11:38:20Z

Prepared #6424 to backport this to branch-2.

Co-authored-by: Dávid Paksy <[email protected]> Signed-off-by: Istvan Toth <[email protected]> (cherry picked from commit 1687018)

This comment has been minimized.

Sign in to view

PDavid commented Oct 9, 2024

View reviewed changes

hbase-client/src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

PDavid marked this pull request as ready for review October 10, 2024 08:52

PDavid commented Oct 10, 2024

View reviewed changes

hbase-client/src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java Show resolved Hide resolved

PDavid commented Oct 10, 2024

View reviewed changes

This comment has been minimized.

Sign in to view

PDavid marked this pull request as draft October 14, 2024 08:10

PDavid changed the title ~~HBASE-28621: PrefixFilter should use SEEK_NEXT_USING_HINT~~ HBASE-28621 PrefixFilter should use SEEK_NEXT_USING_HINT Oct 14, 2024

PDavid force-pushed the HBASE-28621-PrefixFilter-SEEK_NEXT_USING_HINT branch from 09fc843 to 02ed23a Compare October 14, 2024 12:38

This comment has been minimized.

Sign in to view

stoty requested changes Oct 15, 2024

View reviewed changes

hbase-client/src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java Show resolved Hide resolved

This comment has been minimized.

Sign in to view

stoty reviewed Oct 18, 2024

View reviewed changes

hbase-client/src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

PDavid force-pushed the HBASE-28621-PrefixFilter-SEEK_NEXT_USING_HINT branch from 5085fd8 to 0989fa1 Compare October 24, 2024 07:02

This comment has been minimized.

Sign in to view

PDavid commented Oct 24, 2024

View reviewed changes

hbase-client/src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java Show resolved Hide resolved

PDavid commented Oct 24, 2024

View reviewed changes

This comment has been minimized.

Sign in to view

stoty reviewed Oct 30, 2024

View reviewed changes

hbase-client/src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java Outdated Show resolved Hide resolved

hbase-client/src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java Outdated Show resolved Hide resolved

PDavid added 3 commits October 30, 2024 09:25

HBASE-28621 Restored some tests in TestPrefixFilter related to WhileM…

5779bd4

…atchFilter usage

HBASE-28621 Moved hint-related methods before the serialization-relat…

3225197

…ed methods so that the real logic is together.

HBASE-28621 Also pre-compute forward hint at PrefixFilter creation

039c896

to avoid re-computing it several times in the corner case where there are a lot of cells between the hint and the first real match.

stoty reviewed Oct 30, 2024

View reviewed changes

stoty approved these changes Oct 30, 2024

View reviewed changes

This comment has been minimized.

Sign in to view

PDavid commented Oct 30, 2024

View reviewed changes

hbase-client/src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java Show resolved Hide resolved

HBASE-28621 Adjusted comment about behavior of PrefixFilter.filterRow…

1aa84f5

…Key()

stoty reviewed Oct 30, 2024

View reviewed changes

hbase-client/src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

HBASE-28621 Fixed comment: use before/after

ba21710

For reverse scans, smaller/bigger is not accurate. Use before/after instead.

This comment has been minimized.

Sign in to view

PDavid marked this pull request as ready for review October 31, 2024 08:47

PDavid mentioned this pull request Oct 31, 2024

HBASE-28621 PrefixFilter should use SEEK_NEXT_USING_HINT #6424

Merged

stoty merged commit 1687018 into apache:master Nov 6, 2024
1 check passed

stoty pushed a commit that referenced this pull request Nov 6, 2024

HBASE-28621 PrefixFilter should use SEEK_NEXT_USING_HINT (#6361)

3239628

Co-authored-by: Dávid Paksy <[email protected]> Signed-off-by: Istvan Toth <[email protected]> (cherry picked from commit 1687018)

PDavid deleted the HBASE-28621-PrefixFilter-SEEK_NEXT_USING_HINT branch November 6, 2024 09:57

HBASE-28621 PrefixFilter should use SEEK_NEXT_USING_HINT #6361

HBASE-28621 PrefixFilter should use SEEK_NEXT_USING_HINT #6361

Conversation

PDavid commented Oct 9, 2024

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Uh oh!

This comment has been minimized.

Uh oh!

PDavid Oct 10, 2024

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

stoty commented Oct 14, 2024

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Uh oh!

This comment has been minimized.

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Uh oh!

PDavid Oct 24, 2024

Choose a reason for hiding this comment

Uh oh!

stoty Oct 25, 2024

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

stoty left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

stoty Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PDavid Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stoty Oct 30, 2024

Choose a reason for hiding this comment

Uh oh!

stoty Oct 30, 2024

Choose a reason for hiding this comment

Uh oh!

stoty left a comment

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

PDavid commented Oct 30, 2024

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

Apache-HBase commented Oct 30, 2024

Uh oh!

Apache-HBase commented Oct 31, 2024

Uh oh!

stoty commented Oct 31, 2024

Uh oh!

PDavid commented Oct 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stoty Oct 30, 2024 •

edited

Loading

PDavid Oct 30, 2024 •

edited

Loading

PDavid commented Oct 31, 2024 •

edited

Loading