[Feature]: Support AMS horizontal performance expansion capabilities

[wardlican]

Description

Currently, AMS uses a master-slave mode. If a large number of Iceberg tables are managed, the master-slave mode will have performance bottlenecks, and a solution that can support horizontal performance expansion is needed.

Use case/motivation

Multiple catalogs need to be registered on amoro. Under each catalog, there are many Iceberg tables that need to be managed automatically. If only a single-node AMS is working, the single node will consume a lot of resources. It is necessary to support a horizontally scalable performance expansion solution. By expanding the capacity of AMS nodes, each node can be responsible for processing a part of the Iceberg table. In this way, the resources of each node can be controlled and horizontal performance expansion can be achieved.

Describe the solution

Extended support for a new master-slave mode. Each AMS node is only responsible for processing a part of the Iceberg table. It supports dynamic expansion and contraction of the number of AMS nodes and automatic migration of the Iceberg table. The optimizer will pull tasks from each AMS node for processing. The AMS master node is responsible for evenly distributing the Iceberg table to all AMS nodes in the cluster and managing the life cycle of the AMS nodes in the cluster.

Subtasks

-[1]#3845 Use a new configuration item to control whether master & slave mode is enabled.
-[2]#3782 add "bucket_id" field to the table_identifier table to support balanced table partitioning in master-slave mode.
-[3]#3919 Add a registration function for table allocation in master-slave mode.
-[4]#3921 add AmsAssignService to implement balanced bucket allocation in master-slave mode.
-[5]#3923 Modify DefaultTableService to be compatible with master-slave mode.
-[6]#3928 Modify the optimizer to support obtaining tasks from each AMS node for processing.
-[7]#3929 In master-slave mode, each AMS should automatically senses the optimizer.
-[8]# Supports forwarding OpenAPI requests to the master node in master-slave mode.
-[9]# In master-slave mode, for table migration scenarios, supports committing routes to the new node.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct