A design to support multi-instances frontend deployment

[baiyangtx]

Background

As this issue described #2810.

The current Master/Standby architecture is becoming a bottleneck. In addition to providing Optimizing task scheduling capabilities, Amoro also provides valuable capabilities such as Dashboard and RestCatalog.

The purpose of this discussion is to propose a solution to make these valuable features not limited to the master/standby architecture.

Goals

• Make RestCatalogService, DashboardService and TableThrift.Iface supports multi-instance deployment.

Current architecture and the problems

There are 4 services in the API layer:

• Rest Catalog Service, which provides iceberg rest catalog api, and depends on
  • TableService:
    • Creates or deletes table metadata via TableService.
    • Trigger to create a new TableRuntime when creating a new table.
• TableThrift.Iface: which provides catalog metadata api and table blocker api, and depends on
  • TableService:
    • TableThrift.Iface gets catalog metadata information via TableService
    • TableThrift.Iface creates or releases a table blocker, which is managed by TableRuntime.
• DashboardService, which provides all http api for web fortend, and depends on
  • TableService:
    • To manage catalog metadata.
    • Visit ServerCatalog to load AmoroTable
    • Visit TableRuntimes to cancel a process.
    • Visit TableRuntimes to list table status in optimizer tabs.
  • OptimizingService:
    • Visit metadata of optimizer / optimizer group
    • Scale out or release an optimizer
    • Create or delete an optimizer group
  • Terminal Manager
    • To execute sqls on a TerminalSession
• OptimizerThrift.Iface, which provides thrift api for optimizers to fetch tasks, and depends on
  • DefaultOptimizingService
    • Register optimizer to OptimizerGroup
    • Poll task or trigger an optimizing process plan
    • Complete task persist result to db, and update TableRuntime status.

The core reason for currently using the Master/Standby architecture is the need to manage TableRuntime and OptimizerQueue objects in memory.

Based on the above function sorting, we can conclude that if we want to get rid of the master/standby architecture, what we need to do：

1. How to trigger table evaluation when internal table is created.
2. How to implement a stateless table blocker, which has nothing related to TableRuntime.
3. How to cancel a process without TableRuntime object.
4. How to check if a group can be deleted.
5. How to implement a stateless TerminalSession.

[baiyangtx]

A design

I extracted the most basic stateless functionality from the current TableService and OptimizingService, and extracted as some Manager interfaces.

So that services are divided into two groups:

1. Stateless Services, those services will not keep anything in memory. The only thing there can do is provide methods for visiting metadata in database, and those methods are defined in the interface named XXManger.

2. Stateful Services, those services will keep objects in memory, and could only be visited by a master.

Now we can answer the above questions:

1. How to trigger table evaluation when internal table is created.

A: Both internal catalog and external catalog, the TableRuntime objects are discovered by TableExplorer threads. There will be a delay to trigger table evaluation after an internal table is created, but it is acceptable.

2. How to implement a stateless table blocker, which has nothing related to TableRuntime.

A: TableBlocker can be achieved through the database based CAS operations.

3. How to cancel a process without TableRuntime object.

A: We can't do this, but we can do this by proxy the Web Call to master.

4. How to check if a group can be deleted.

A: A group could not be deleted if:

i. It is the default group of any catalog, this could be checked by database queries.
ii. It has an active optimizer, this could be checked by database queries.
iii. Some tables refer to the group by setting properties directly. In the current implementation, this check is done by maintaining the tableRuntime and group correspondence in memory. We can persist group information of tables to the database by adding a field to the table_runtime table. And maintain the value via table refresh threads.

5. How to implement a stateless TerminalSession.

A: TerminalSession currently provides a range of values:

i. Saving the current database when executing a sql use $database
ii. Saving properties when executing a sql set xxx=xxx

There are some ways to achieve the same effect in a stateless way:

1. Persist context information to Amoro Database and initialize them when creating TerminalSession.
2. Although TerminalSession is stateful, it does not depend on the master node. When configuring the nginx proxy, you can use the ip_hash strategy to make the session fall into the same node.

upstream backend {    
    ip_hash;    
    server amoro1.example.com;    
    server amoro2.example.com;
}

[link3280]

@baiyangtx are you working on it currently? I'm planning to push it forward in Q4.

[baiyangtx]

At present, the lock-free design part has been promoted #3079 , and other content is not being continued for the time being.

[link3280]

I have 2 questions in addition:

1. how to divide workloads on multiple instances? An optimizing process is planned when an optimizer pulls a new task, does that mean we need to introduce a dividing strategy on the optimizer end, like pulling tasks from different ams based on groups/catalogs or just round-robinly.
2. how to migrate in-mem locks to distributed locks? As you mentioned before, we could use DB CAS locks instead (e.g. StatedPersistentBase would need to migrate to a db lock), however it breaks the assumption that in-mem objects are always the ground truth and may lower the efficiency. Moreover, we need to carefully think about the table runtimes consistency problem, when one ams makes a change to the table runtimes while the others don't know about it.

[baiyangtx]

A:

1. In this design, the optimizing service still runs in an active/standby architecture, and only the REST API interface runs in multiple instances.
2. Currently, locks that do not involve the Optimizing part have been refactored into database implementations. [AMORO-3078]: A lock-free TableBlocker implementation and does not rely on TableRuntime anymore. #3079