Capacity Planning for Storage-Compute Decouple mode

[Nj-kol]

I am setting up doris 3 in comute storage decoupled mode in k8s.

My target doris cluster will only used by a single application and will host around 100 TB of data. What config would you recommend for

• Foundationdb ( log and storage )
• Doris FE and BE ( fe and be storage for filecache and logs )

[ixzc]

You can first refer to the general model of 1 CPU core : 4GB Memory and 1 CPU core : 50GB Data for estimation. However, there are no universal stress test metrics for cluster scale evaluation—it is closely tied to your actual scenario and highly relevant to your business. For example:

Is the data log-based, report-based, or of other types?
Do queries mostly involve multi-table joins or single-table operations?

These factors can lead to significant differences in data scan volumes. Another example: if data is stored on a daily basis, even if the total volume reaches 1TB (accumulated over a year), the daily data volume may only be around 3GB. In this case, if queries are filtered by day, the actual scan volume will be very small.

Therefore, cluster scale must be evaluated through stress testing based on business data. It usually depends on two key factors: concurrent access in the scenario and computational complexity of data requirements.

for detail recommendation, we can have a deep discussion.

[Nj-kol]

Sure @ixzc would love to have a discussion. Yes, I do agree there cannot be a one-size-fits-all all approach. But, some I feel some general guidelines targetted to use-cases can help

[ixzc]

yes, that's a good idea.