Fix remote call of s3Cluster function #583

ianton-ru · 2025-01-09T09:37:20Z

Changelog category (leave one):

Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fix remote call of s3Cluster function

Documentation entry for user-facing changes

remote('remote_host', ''s3Cluster(.....)') did not work with error

2024.11.24 09:07:37.307264 [ 11 ] {5548e29a-ab3e-4306-9925-c5d39e47fa9c} <Error> executeQuery: Code: 49. DB::Exception: Distributed task iterator is not initialized: While executing Remote. (LOGICAL_ERROR) (version 24.11.1.1) (from 172.16.1.1:55326) (in query: SELECT * FROM remote('s0_0_1', s3Cluster('cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', '[HIDDEN]', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))')) ORDER BY (name, value, polygon) ASC LIMIT 1), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x0000000014cd1e72
1. /home/iantonspb/ch-build/./src/Common/Exception.cpp:109: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ba7cad9
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x00000000069abf0c
3. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x00000000069b5e8b
4. /home/iantonspb/ch-build/./src/QueryPipeline/RemoteQueryExecutor.cpp:715: DB::RemoteQueryExecutor::processReadTaskRequest() @ 0x000000000fecbf12
5. /home/iantonspb/ch-build/./src/QueryPipeline/RemoteQueryExecutor.cpp:614: DB::RemoteQueryExecutor::processPacket(DB::Packet) @ 0x000000000feca25a
6. /home/iantonspb/ch-build/./src/QueryPipeline/RemoteQueryExecutor.cpp:553: DB::RemoteQueryExecutor::readAsync() @ 0x000000000fecb60c
7. /home/iantonspb/ch-build/./src/Processors/Sources/RemoteSource.cpp:182: DB::RemoteSource::tryGenerate() @ 0x000000001295b590
8. /home/iantonspb/ch-build/./src/Processors/ISource.cpp:108: DB::ISource::work() @ 0x0000000012648b45
9. /home/iantonspb/ch-build/./src/Processors/Executors/ExecutionThreadContext.cpp:49: DB::ExecutionThreadContext::executeTask() @ 0x000000001265fc00
10. /home/iantonspb/ch-build/./src/Processors/Executors/PipelineExecutor.cpp:289: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000012655c3f
11. /home/iantonspb/ch-build/./src/Processors/Executors/PipelineExecutor.cpp:255: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001265541d
12. /home/iantonspb/ch-build/./src/Processors/Executors/PipelineExecutor.cpp:126: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000012655207
13. /home/iantonspb/ch-build/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000012662d8c
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000bb49da3
15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:359: ? @ 0x000000000bb4f07b
16. ? @ 0x00007f08de1c4ac3
17. ? @ 0x00007f08de256850

The reason is this:
https://github.com/ClickHouse/ClickHouse/blob/master/src/TableFunctions/TableFunctionObjectStorageCluster.cpp#L34
s3Cluster function has two stages, and stage is choose depends of query_kind - INITIAL_QUERY for first stage and SECONDARY_QUERY for second stage. When s3Cluster called from remote function, both stages were with SECONDARY_QUERY.

Now requests from remote function use INITIAL_QUERY.

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

CI Settings (Only check the boxes if you know what you are doing):

Allow: All Required Checks
Allow: Stateless tests
Allow: Stateful tests
Allow: Integration Tests
Allow: Performance tests
Allow: All Builds
Allow: batch 1, 2 for multi-batch jobs
Allow: batch 3, 4, 5, 6 for multi-batch jobs

Exclude: Style check
Exclude: Fast test
Exclude: All with ASAN
Exclude: All with TSAN, MSAN, UBSAN, Coverage
Exclude: All with aarch64, release, debug

Run only fuzzers related jobs (libFuzzer fuzzers, AST fuzzers, etc.)
Exclude: AST fuzzers

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

CI Settings (Only check the boxes if you know what you are doing):

Allow: All Required Checks
Allow: Stateless tests
Allow: Stateful tests
Allow: Integration Tests
Allow: Performance tests
Allow: All Builds
Allow: batch 1, 2 for multi-batch jobs
Allow: batch 3, 4, 5, 6 for multi-batch jobs

Exclude: Style check
Exclude: Fast test
Exclude: All with ASAN
Exclude: All with TSAN, MSAN, UBSAN, Coverage
Exclude: All with aarch64, release, debug

Run only fuzzers related jobs (libFuzzer fuzzers, AST fuzzers, etc.)
Exclude: AST fuzzers

ilejn · 2025-01-10T15:46:13Z

src/Interpreters/Context.cpp


-    if (client_info.query_kind == ClientInfo::QueryKind::INITIAL_QUERY)
+    if (client_info.query_kind == ClientInfo::QueryKind::INITIAL_QUERY
+        && (getApplicationType() != ApplicationType::SERVER || client_info.initial_query_id.empty()))


According to the comment in ClientInfo.h
when query_kind == INITIAL_QUERY
initial_query_id is equal to current.
Does not it contradict with the condition?

Technically it must be some new kind, something like REMOTE_INITIAL_QUERY, but this breaks backward compatibility on protocol level.

ilejn · 2025-01-10T15:48:28Z

src/Interpreters/ClusterProxy/executeQuery.cpp

            not_optimized_cluster->getName());

        read_from_remote->setStepDescription("Read from remote replica");
+        read_from_remote->setRemoteFunction(is_remote_function);


To me, "setIsRemoteFunction" is slightly more natural.

ilejn · 2025-01-10T15:58:42Z

tests/integration/test_s3_cluster/test.py

+        """
+    )
+
+    assert TSV(pure_s3) == TSV(s3_distributed)


Is the PR really s3Cluster specific?

No, affects all *Cluster object storage functions. Suggest to make test for others?

Not sure about new tests, may be. Depends on your feeling how probably is to break something that worked before or accidentally create a "bridge" that e.g. bypasses security.

Added test for iceberg, most actual for us.

altinity-robot · 2025-01-29T17:16:55Z

This is an automated comment for commit 85495bd with description of existing statuses. It's updated for the latest CI running

❌ Click here to open a full report in a separate page

Check name	Description	Status
Integration tests	The integration tests report. In parenthesis the package type is given, and in square brackets are the optional part/total tests	❌ failure
Sign aarch64	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	❌ error
Sign release	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	❌ error
Stateful tests	Runs stateful functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	❌ failure
Stateless tests	Runs stateless functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	❌ failure
Stress test	Runs stateless functional tests concurrently from several clients to detect concurrency-related errors	❌ failure

Successful checks

Check name	Description	Status
Builds	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Compatibility check	Checks that clickhouse binary runs on distributions with old libc versions. If it fails, ask a maintainer for help	✅ success
Docker keeper image	The check to build and optionally push the mentioned image to docker hub	✅ success
Docker server image	The check to build and optionally push the mentioned image to docker hub	✅ success
Install packages	Checks that the built packages are installable in a clear environment	✅ success
Ready for release	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success

…emote-s3-cluster

…s3-cluster Fix remote call of s3Cluster function

…_calls 25.3 Antalya port #583, #584, #703, #720 - fixes for distributed calls

…s3-cluster Fix remote call of s3Cluster function

…_calls 25.3 Antalya port #583, #584, #703, #720 - fixes for distributed calls

…s3-cluster Fix remote call of s3Cluster function

…te_calls 25.6.5 Antalya port #583, #584, #703, #720 - fixes for s3Cluster distributed calls

ilejn reviewed Jan 10, 2025

View reviewed changes

ianton-ru force-pushed the project-antalya-24.12.2-remote-s3-cluster branch from 7473648 to 8b6064f Compare January 14, 2025 12:32

ianton-ru added 8 commits January 29, 2025 15:20

Test for remote+s3Cluster

ce467f0

Use INITIAL_QUERY for remote() call

fc0eef8

Avoid CLIENT_INFO_DOES_NOT_MATCH error

812d4b3

Workaroung for remote with multiple shards

abc65f3

Workaround to CLIENT_INFO_DOES_NOT_MATCH with 'TCP not equal to HTTP'

b5e6dde

Keep initial_query_id for remote with INITIAL_QUERY

ed31a9f

Left QueryID for server only

46fa2d1

Fixes after few comments

8ac4341

Enmk force-pushed the project-antalya-24.12.2-remote-s3-cluster branch from 8b6064f to 8ac4341 Compare January 29, 2025 14:20

ianton-ru mentioned this pull request Jan 30, 2025

Project antalya backport remote s3 cluster #565

Closed

29 tasks

ianton-ru and others added 2 commits February 4, 2025 11:28

Merge branch 'project-antalya-24.12.2' into project-antalya-24.12.2-r…

9abc929

…emote-s3-cluster

Merge branch 'project-antalya-24.12.2' into project-antalya-24.12.2-r…

85495bd

…emote-s3-cluster

Enmk merged commit 5a4f3bf into project-antalya-24.12.2 Feb 6, 2025
185 of 248 checks passed

ianton-ru mentioned this pull request Apr 4, 2025

Antalya 25.2 ivashkin #716

Merged

Enmk mentioned this pull request Apr 14, 2025

Antalya Feature Matrix #736

Merged

3 tasks

ianton-ru pushed a commit that referenced this pull request May 23, 2025

Merge pull request #583 from Altinity/project-antalya-24.12.2-remote-…

9f81e2d

…s3-cluster Fix remote call of s3Cluster function

ianton-ru mentioned this pull request May 23, 2025

25.3 Antalya port #583, #584, #703, #720 - fixes for distributed calls #796

Merged

Enmk added a commit that referenced this pull request May 29, 2025

Merge pull request #796 from Altinity/feature/antalya-25.3/fix_remote…

0d9a1de

…_calls 25.3 Antalya port #583, #584, #703, #720 - fixes for distributed calls

ianton-ru pushed a commit that referenced this pull request Jul 17, 2025

Merge pull request #796 from Altinity/feature/antalya-25.3/fix_remote…

a616152

…_calls 25.3 Antalya port #583, #584, #703, #720 - fixes for distributed calls

ianton-ru added a commit that referenced this pull request Jul 17, 2025

Merge pull request #583 from Altinity/project-antalya-24.12.2-remote-…

3cc7033

…s3-cluster Fix remote call of s3Cluster function

ianton-ru mentioned this pull request Jul 17, 2025

25.6 Antalya port #583, #584, #703, #720, #839 - fixes for distributed calls #922

Closed

13 tasks

ianton-ru pushed a commit that referenced this pull request Aug 5, 2025

Merge pull request #796 from Altinity/feature/antalya-25.3/fix_remote…

db934dc

…_calls 25.3 Antalya port #583, #584, #703, #720 - fixes for distributed calls

ianton-ru added a commit that referenced this pull request Aug 5, 2025

Merge pull request #583 from Altinity/project-antalya-24.12.2-remote-…

274d7c8

…s3-cluster Fix remote call of s3Cluster function

ianton-ru mentioned this pull request Aug 5, 2025

25.6.5 Antalya port #583, #584, #703, #720 - fixes for s3Cluster distributed calls #950

Merged

13 tasks

Enmk added a commit that referenced this pull request Sep 9, 2025

Merge pull request #950 from Altinity/feature/antalya-25.6.5/fix_remo…

dfa56f4

…te_calls 25.6.5 Antalya port #583, #584, #703, #720 - fixes for s3Cluster distributed calls

ianton-ru pushed a commit that referenced this pull request Oct 1, 2025

Merge pull request #950 from Altinity/feature/antalya-25.6.5/fix_remo…

06e34b3

…te_calls 25.6.5 Antalya port #583, #584, #703, #720 - fixes for s3Cluster distributed calls

ianton-ru mentioned this pull request Oct 1, 2025

25.8 Antalya port: fixes for s3Cluster distributed calls #1052

Open

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix remote call of s3Cluster function #583

Fix remote call of s3Cluster function #583

Uh oh!

ianton-ru commented Jan 9, 2025

Uh oh!

ilejn Jan 10, 2025

Uh oh!

ianton-ru Jan 13, 2025

Uh oh!

ilejn Jan 10, 2025

Uh oh!

ianton-ru Jan 14, 2025

Uh oh!

ilejn Jan 10, 2025 •

edited

Loading

Uh oh!

ianton-ru Jan 13, 2025

Uh oh!

ilejn Jan 13, 2025

Uh oh!

ianton-ru Jan 14, 2025

Uh oh!

altinity-robot commented Jan 29, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Fix remote call of s3Cluster function #583

Fix remote call of s3Cluster function #583

Uh oh!

Conversation

ianton-ru commented Jan 9, 2025

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

CI Settings (Only check the boxes if you know what you are doing):

CI Settings (Only check the boxes if you know what you are doing):

Uh oh!

ilejn Jan 10, 2025

Choose a reason for hiding this comment

Uh oh!

ianton-ru Jan 13, 2025

Choose a reason for hiding this comment

Uh oh!

ilejn Jan 10, 2025

Choose a reason for hiding this comment

Uh oh!

ianton-ru Jan 14, 2025

Choose a reason for hiding this comment

Uh oh!

ilejn Jan 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ianton-ru Jan 13, 2025

Choose a reason for hiding this comment

Uh oh!

ilejn Jan 13, 2025

Choose a reason for hiding this comment

Uh oh!

ianton-ru Jan 14, 2025

Choose a reason for hiding this comment

Uh oh!

altinity-robot commented Jan 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ilejn Jan 10, 2025 •

edited

Loading

altinity-robot commented Jan 29, 2025 •

edited

Loading