[LLamaSharp.Backend.Cuda12 v0.10.0] Unable to load the model onto multiple GPUs

[ChengYen-Tang]

In version cuda12 v0.10.0, the model will only load onto a single GPU, resulting in cudaMalloc failed: out of memory
There is no such issue in cuda12 v0.9.1.
OS: Ubuntu22.04

[martindevans]

There was a new "GPU Split Mode" parameter introduced in 0.10.0, see here.

I don't use multiple GPU inference myself, but I would guess you need to set that to something other than "None" when loading the model.

[ChengYen-Tang]

Thanks, I will try it tomorrow.

[ChengYen-Tang]

Thank you, it worked