Add a MockS3Server plus an s3client and bulkdumpings3 simulation test #12279

saintstack · 2025-07-25T18:47:41Z

Register it at 127.0.0.1:8080 on the simulated network (simulated network intercepts requests for 127.0.0.1:8080 and does appropriate forwarding to s3 handler). Add two workloads, one for s3client against 's3' and another that swaps in s3 as backend for the BulkDumping test.

The Mock S3 Server is mostly generated (claude-sonnet-4) code. It supports:

Basic GET/PUT/DELETE/HEAD object operations
Multipart uploads (initiate, upload parts, complete, abort)
Object tagging (put/get tags)
In-memory storage with deterministic behavior
S3-compatible XML responses

fdbserver/workloads/BulkDumping.actor.cpp Change this workload so it reads the transport to use from configuration file so can be used to test file-based and s3 bulk loading. If the transport is blobstore/s3, start up the mocks3server in _setup.
fdbserver/workloads/S3ClientWorkload.actor.cpp Simple workload to exercise s3client. Starts up the mocks3server in _setup.
fdbclient/S3BlobStore.actor.cpp Check BUGGIFY before messing w/ status codes.
fdbrpc/HTTP.actor.cpp Add defines and log if failed parse of status line.


j start --tarball ~/build_output/packages/correctness-7.4.0.tar.gz  --max-run 100000  --env TH_ARCHIVE_LOGS_ON_FAILURE=true && j tail --errors
....

  20250725-194324-stack-d0bf81a817222ed4             compressed=True data_size=41362786 duration=5883808 ended=100000 env=TH_ARCHIVE_LOGS_ON_FAILURE=true fail=1 fail_fast=10 max_runs=100000 pass=99999 priority=100 remaining=0 runtime=0:57:29 sanity=False started=100000 stopped=20250725-204053 submitted=20250725-194324 timeout=5400 username=stack

And I think I fixed the one fail ... its global buggify.

network (simulated network intercepts requests for 127.0.0.1:8080 and does appropriate forwarding to s3 handler). Add two workloads, one for s3client against 's3' and another that swaps in s3 as backend for the BulkDumping test. The Mock S3 Server is mostly generated (claude-sonnet-4) code. It supports: - Basic GET/PUT/DELETE/HEAD object operations - Multipart uploads (initiate, upload parts, complete, abort) - Object tagging (put/get tags) - In-memory storage with deterministic behavior - S3-compatible XML responses * fdbserver/workloads/BulkDumping.actor.cpp Change this workload so it reads the transport to use from configuration file so can be used to test file-based and s3 bulk loading. If the transport is blobstore/s3, start up the mocks3server in _setup. * fdbserver/workloads/S3ClientWorkload.actor.cpp Simple workload to exercise s3client. Starts up the mocks3server in _setup. * fdbclient/S3BlobStore.actor.cpp Check BUGGIFY before messing w/ status codes. * fdbrpc/HTTP.actor.cpp Add defines and log if failed parse of status line.

foundationdb-ci · 2025-07-25T19:12:50Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 4783393
Duration 0:24:56
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-25T19:27:56Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 4783393
Duration 0:40:02
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-25T19:36:46Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 4783393
Duration 0:48:52
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-25T19:50:18Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 4783393
Duration 1:02:24
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-25T19:54:56Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 4783393
Duration 1:07:02
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-07-25T19:55:55Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 4783393
Duration 1:07:59
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-25T19:56:30Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 4783393
Duration 1:08:36
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-25T20:53:28Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: e23b25e
Duration 0:24:15
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-25T21:08:19Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: e23b25e
Duration 0:39:08
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-25T21:17:27Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: e23b25e
Duration 0:48:15
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-25T21:26:54Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: e23b25e
Duration 0:57:39
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-25T21:32:45Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: e23b25e
Duration 1:03:33
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-25T21:36:54Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: e23b25e
Duration 1:07:43
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-07-25T21:40:54Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: e23b25e
Duration 1:11:39
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

kakaiu · 2025-07-28T19:15:17Z

fdbserver/fdbserver.actor.cpp

@@ -2028,6 +2028,8 @@ int main(int argc, char* argv[]) {
 			enableGeneralBuggify();
 		} else {
 			disableGeneralBuggify();
+			// When buggify is disabled, also disable global fault injection
+			opts.faultInjectionEnabled = false;


Why do we want to disable the fault injection? I think the buggify is not exactly same as the fault injection. Please correct me if I'm wrong. Thanks!

This is here because though I have fault injection set to false in the two toml tests files, there was still fault injection happening in joshua. Let me undo this for now. I'll post new joshua run numbers... Get an example of what fault injection I was seeing.

kakaiu · 2025-07-28T19:19:03Z

fdbclient/S3BlobStore.actor.cpp

@@ -1186,12 +1186,13 @@ ACTOR Future<Reference<HTTP::IncomingResponse>> doRequest_impl(Reference<S3BlobS
 			}

 			Reference<HTTP::IncomingResponse> _r = wait(timeoutError(reqF, requestTimeout));
-			if (g_network->isSimulated() && deterministicRandom()->random01() < 0.1) {
+			if (g_network->isSimulated() && BUGGIFY && deterministicRandom()->random01() < 0.1) {


Why do we do Buggify here? Thanks!

Only do the random if BUGGIFY is enabled. I don't to turn off any variance for the moment in s3 client behaviors till we have stable determinism.

kakaiu · 2025-07-28T19:22:12Z

fdbserver/MockS3Server.actor.cpp

+}
+
+// Public Interface Implementation
+ACTOR Future<Void> startMockS3Server(NetworkAddress listenAddress) {


When is this startMockS3Server used? Thanks!

Not currently. I thought I would need it until tripped over the register http server thingy. Seems useful though if someone wanted to start up a mock s3 instance.

foundationdb-ci · 2025-07-28T20:43:42Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: be1d620
Duration 0:23:55
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-28T21:00:26Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: be1d620
Duration 0:40:36
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-28T21:09:20Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: be1d620
Duration 0:49:32
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-28T21:22:51Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: be1d620
Duration 1:03:03
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-28T21:26:33Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: be1d620
Duration 1:06:41
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-07-28T21:37:43Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: be1d620
Duration 1:17:50
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-28T21:38:04Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: be1d620
Duration 1:18:16
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

saintstack · 2025-07-28T21:47:33Z

Here is 10k tests of BulkDumpingS3 only

j start --tarball ~/BulkDumpingS3.tar.gz  --max-run 10000 --env TH_ARCHIVE_LOGS_ON_FAILURE=true --env TH_UNSEED_CHECK_RATIO=1.0
...
  20250728-202325-stack-06303b24331eafcc             compressed=True data_size=41406270 duration=422162 ended=10000 env=TH_ARCHIVE_LOGS_ON_FAILURE=true:TH_UNSEED_CHECK_RATIO=1.0 fail_fast=10 max_runs=10000 pass=10000 priority=100 remaining=0 runtime=0:17:15 sanity=False started=10000 stopped=20250728-204040 submitted=20250728-202325 timeout=5400 username=stack

and here are 100k of general tests

j start --tarball ~/build_output/packages/correctness-7.4.0.tar.gz  --max-run 100000  --env TH_ARCHIVE_LOGS_ON_FAILURE=true && j tail --errors
Note: Ensemble will complete after 10 failed results.
Note: Ensemble will complete after 100000 runs.
Starting ensemble
...
  20250728-204821-stack-d63f1d74060032c4             compressed=True data_size=41361024 duration=5864025 ended=100000 env=TH_ARCHIVE_LOGS_ON_FAILURE=true fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=0:57:58 sanity=False started=100000 stopped=20250728-214619 submitted=20250728-204821 timeout=5400 username=stack

Looks like its fine w/o the global disabling of fault injection.

kakaiu · 2025-07-29T04:39:33Z

tests/slow/S3Client.toml

+
+[[knobs]]
+# Disable failure injection for deterministic behavior
+bulkload_sim_failure_injection = false


Seems that this knob is irrelevant to this toml file test?

Yes. Removed (was mirroring changes in BulkDumpingS3 to this file)

kakaiu · 2025-07-29T04:40:12Z

tests/slow/S3Client.toml

+# When done, run:
+#  shutdown_weed ~/weed
+
+# Disable all fault injection and network failures


Why do we want to disable all fault injection? Will we enable those in the future? Thanks!

Yeah. Currently, fault injection produces failures. In follow-on, would introduce faults.

kakaiu · 2025-07-29T04:41:02Z

tests/slow/BulkDumpingS3.toml

+# DETERMINISM FIX: Combined knobs section to avoid duplicates
+[[knobs]]
+# S3/Bulkload determinism fixes
+bulkload_sim_failure_injection = false


Can you try with enabling this knob bulkload_sim_failure_injection? Since we have a mocked http server, we do not have a reason to disable the bulkload failure injection? Thanks!

When I disable this flag, I get failures:

20250729-152416-stack-b70ff5b181db9da0 compressed=True data_size=41406205 duration=898424 ended=10000 env=TH_ARCHIVE_LOGS_ON_FAILURE=true:TH_UNSEED_CHECK_RATIO=1.0 fail=2 fail_fast=10 max_runs=10000 pass=9998 priority=100 remaining=0 runtime=0:22:44 sanity=False started=10000 stopped=20250729-154700 submitted=20250729-152416 timeout=5400 username=stack

Ok. We can set the knob here and get the PR merged. We will remove the bulkload_sim_failure_injection later. Maybe better to add a "TODO" here.

kakaiu · 2025-07-29T04:42:33Z

tests/slow/BulkDumpingS3.toml

+# Simple network configuration - single region, no satellites
+minimumRegions = 1
+# Use ssd-2 storage engine instead of rocksdb to avoid RocksDB bulk loading crashes
+# The RocksDB assertion failure indicates incompatibility with current bulk loading implementation


Which assertion failure? The current bulkload implementation should be compatible to RocksDB. Please let me know more detail about this. Thanks!

This is old at your suggestion IIRC. We can look at this in a follow-on?

kakaiu · 2025-07-29T04:47:18Z

fdbserver/workloads/BulkDumping.actor.cpp

+				// Register MockS3Server with IP address - simulation environment doesn't support hostname resolution
+				// See in HTTPServer.actor.cpp how the MockS3RequestHandler is implemented. Client connects to
+				// connect("127.0.0.1", "8080") and then simulation network routes it to MockS3Server.
+				wait(g_simulator->registerSimHTTPServer("127.0.0.1", "8080", makeReference<MockS3RequestHandler>()));


kakaiu

LGTM. Only a few nits. Thank you for the nice PR!

foundationdb-ci · 2025-07-29T16:32:19Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 25ea3d0
Duration 0:25:49
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-29T16:46:19Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 25ea3d0
Duration 0:39:49
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-29T16:54:34Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 25ea3d0
Duration 0:48:07
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-29T17:08:48Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 25ea3d0
Duration 1:02:20
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-29T17:19:44Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 25ea3d0
Duration 1:13:18
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-07-29T17:59:05Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 25ea3d0
Duration 1:52:38
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-29T18:02:46Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 25ea3d0
Duration 1:56:20
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

saintstack · 2025-07-29T21:30:38Z

Thanks for the review @kakaiu

saintstack changed the title ~~Add a MockS3Server and an s3client and bulkdumpings3 simulationn test~~ Add a MockS3Server and an s3client and bulkdumpings3 simulation test Jul 25, 2025

Disable global buggify if the toml file turns off buggify

e23b25e

saintstack requested review from kakaiu and jzhou77 July 28, 2025 15:24

saintstack changed the title ~~Add a MockS3Server and an s3client and bulkdumpings3 simulation test~~ Add a MockS3Server plus an s3client and bulkdumpings3 simulation test Jul 28, 2025

kakaiu reviewed Jul 28, 2025

View reviewed changes

Remove disable of fault injection if BUGGIFY set

be1d620

kakaiu reviewed Jul 29, 2025

View reviewed changes

Remove an irrelevant configuration

25ea3d0

kakaiu self-requested a review July 29, 2025 17:55

kakaiu approved these changes Jul 29, 2025

View reviewed changes

saintstack merged commit d45a17b into apple:main Jul 29, 2025
7 checks passed

Add a MockS3Server plus an s3client and bulkdumpings3 simulation test #12279

Add a MockS3Server plus an s3client and bulkdumpings3 simulation test #12279

Uh oh!

Conversation

saintstack commented Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

foundationdb-ci commented Jul 25, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jul 25, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jul 25, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 25, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jul 25, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jul 25, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jul 25, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jul 25, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jul 25, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jul 25, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 25, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jul 25, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jul 25, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jul 25, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kakaiu Jul 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

foundationdb-ci commented Jul 28, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jul 28, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jul 28, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 28, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jul 28, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jul 28, 2025

Result of foundationdb-pr on Linux RHEL 9

saintstack commented Jul 25, 2025 •

edited

Loading

kakaiu Jul 28, 2025 •

edited

Loading

kakaiu Jul 29, 2025 •

edited

Loading