[Feature Request] Multi NUMA CPU Tensor Parallel

[rankaiyx]

Feature Description

By splitting the weights between NUMA nodes, and then doing tensor parallel between those nodes, bandwidth utilization can be significantly improved on multi socket systems, circumventing the problem of overloading the link between them when weights are shared globally. This was also recently implemented by sglang: https://lmsys.org/blog/2025-07-14-intel-xeon-optimization/#multi-numa-parallelism

Motivation

Likely faster