Tensor parallelism
Splitting each layer across several GPUs so they compute together, rather than taking turns.
The default multi-GPU mode in llama.cpp splits by layer, which means the GPUs run sequentially — one computes while the others wait. Capacity adds up; bandwidth does not. Two cards give you the combined memory at roughly one card's token rate.
Tensor parallelism instead splits each layer's work across cards so they compute simultaneously, which does aggregate bandwidth. It costs a synchronization step at every layer, and consumer cards lack the fast interconnect that makes that cheap.