Add a card to run bigger, not to run faster
If a model already fits on your GPU at the context you want, a second GPU will not speed it up — under llama.cpp's default placement it makes things marginally slower. If a model does not fit, a second GPU is the whole ballgame, because a model that spills to system RAM runs 5–20x slower, not slightly slower.
The question is never "will two cards be twice as fast." It is "what does 48GB let me load that 24GB does not."
The default split is a pipeline with one worker at a time
-sm layer is the default, and llama.cpp's help text in common/arg.cpp is honest about it: "split layers and KV across GPUs (pipelined)". GPU 0 holds layers 0–39 and their KV, GPU 1 holds layers 40–79 and theirs.
ggml_backend_sched_compute_splits() in ggml/src/ggml-backend.cpp makes the mechanism plain — a for (int split_id = 0; split_id < sched->n_splits; split_id++) loop that copies each split's inputs to that split's backend and runs it. Layer 40 cannot start until layer 39's hidden state exists. While GPU 0 works, GPU 1 waits, then they swap. Bandwidth does not add up because the cards never read weights at the same time.
There is one real exception, and it isn't the one people assume. llama-context.cpp enables pipeline parallelism when n_devices > 1, split mode is layer, KV is offloaded and you set no tensor overrides; ggml_backend_sched_new then sets n_copies = GGML_SCHED_MAX_COPIES, which is 4. That puts four ubatches in flight. Prefill submits four (defaults n_batch 2048, n_ubatch 512), so prompt processing genuinely overlaps across cards. Decode submits one ubatch per token — nothing to overlap. The second card helps prefill and concurrent server slots, not single-stream token rate.
Time per token is the sum of each card's read time, t = Σ bᵢ/BWᵢ. Split bytes proportional to bandwidth (which is optimal) and every term becomes B/ΣBW, so t = nB/ΣBW and effective bandwidth is ΣBW/n. The mean of your cards' bandwidths, not the sum. Two 3090s average to 936 GB/s, which is one 3090.
What that costs on a real file
bartowski's Llama-3.3-70B-Instruct-Q4_K_M.gguf is 42,520,398,816 bytes, of which 42,512,531,712 (39.593 GiB) is tensor data across 80 blocks — 513,245,184 bytes per layer, plus a 591,003,648-byte token_embd (a row lookup, not read in bulk) and an 861,880,320-byte output.weight (read in full every token). Per-token read: 41.92 GB.
Split 40/40, GPU 0 reads 20.53 GB and GPU 1 reads 21.39 GB, serially. Using our fitted consumer-CUDA constants — 79% of advertised bandwidth plus 0.74 ms fixed per token per device — that is 27.8 ms then 28.9 ms: about 58 ms per token, 17 tok/s. If bandwidth aggregated, the same 41.92 GB over 2 × 739 GB/s would be 29 ms and 34 tok/s. No flag recovers that gap under -sm layer.
The gain is real, though. This model has 8 KV heads × 128, so 320 KiB of KV per token: weights plus 16K of context is 39.59 + 5.00 = 44.59 GiB and fits in 48GB, while 32K needs 10.00 GiB of KV, totals 49.59 GiB, and does not. A third card reaches 64K.
Now the counterexample. Qwen3-32B-Q4_K_M.gguf holds 19,756,174,336 bytes of tensor data, 19.32 GB read per token. On one 3090: 26.9 ms, 37.2 tok/s. Layer-split across two: identical bytes, same serial reads, plus a second device's fixed overhead — 27.6 ms, 36.2 tok/s. What the second card buys is context, since 18.40 GiB of weights plus 8.00 GiB of KV at 32K is 26.4 GiB and never fit on one.
-sm row is gone; -sm tensor replaced it
If you are copying flags from a 2024 forum post, stop. PR #24216 (6 July 2026) removed row-split from CUDA; ggml_backend_split_buffer_type now exists only in SYCL, so -sm row on CUDA throws device CUDA0 does not support split buffers. docs/multi-gpu.md marks it Deprecated.
The replacement is -sm tensor, added in PR #19378 (April 2026) and still EXPERIMENTAL. It is textbook Megatron-style tensor parallelism: llama_meta_device_get_split_state() splits attn_q/attn_k/attn_v and ffn_up/ffn_gate on axis 1, attn_output and ffn_down on axis 0, and cache_k_l*/cache_v_l* on axis 0 as well. Every card holds a slice of every matrix and of the KV, reads concurrently, and bandwidth genuinely sums — at the price of an all-reduce after each row-split matmul, two per layer.
The payload is smaller than folklore suggests: for Llama-70B two-way, 160 reductions per token of 8192 F32 values, 5.24 MB total, about 210 µs over a real 25 GB/s PCIe 4.0 x16 link against a ~29 ms token. What hurts is 160 synchronisation points per token on a bus with microsecond latency — hence llama.cpp's own troubleshooting line, that if multi-GPU is slower than single-GPU "the performance is bottlenecked by GPU interconnect speed… try --split-mode layer (less communication)."
The constraints are strict. -fa on is mandatory, quantized KV is a startup error, and --fit is disabled so you size -c yourself. There is an architecture allow-list: DeepSeek2, Grok, OLMoE, Mamba/Mamba2, Jamba, Nemotron-H, Minimax-M2, Kimi-Linear and Gemma-3n among others fail with LLAMA_SPLIT_MODE_TENSOR not implemented for architecture. Build with NCCL (-DGGML_CUDA_NCCL=ON is default, but NCCL does not ship with CUDA — check your CMake log).
llama-cli -m Llama-3.3-70B-Instruct-Q4_K_M.gguf -sm tensor -fa on -ctk f16 -ctv f16 -c 16384
GGML_CUDA_P2P=1 lets the GPUs DMA to each other rather than through system memory; it is opt-in because driver support is "usually restricted to workstation/datacenter GPUs" and it "may cause crashes or corrupted outputs" on some boards, usually with IOMMU on.
NVIDIA's spec comparison lists NVLink Yes for the RTX 3090, No for the 4090, and the 5090 page states "NVIDIA NVLink™ (SLI-Ready): No". A bridged pair of 3090s is the only consumer configuration where tensor parallelism has an interconnect worth the all-reduces; on 4090s and 5090s it runs over PCIe, and vLLM's own guidance for single-node setups without NVLink is to prefer pipeline parallelism for "higher throughput and lower communication overhead." If you do reach for --tensor-parallel-size 2, note there is no CPU-offload escape hatch — weights and KV must all be resident — and vLLM raises Total number of attention heads (N) must be divisible by tensor parallel size (M) at startup.
Mixed cards, and the split ratio nobody changes
You inherit the average, so a slow card drags a fast one down. Qwen3-32B Q4_K_M on a 5090 alone is 14.4 ms per token, 69.5 tok/s. Spread across a 5090 and a 3090, mean bandwidth is (1792 + 936)/2 = 1364 GB/s: 19.4 ms, 51.5 tok/s. 35% slower for adding a GPU. If the model fits on the fast card, pin it with -dev CUDA0 and give the second card a different job.
When a model does need both, -ts defaults to splitting proportional to free memory, which is the wrong axis. For Llama-70B on that pair, 32:24 memory-proportional gives 42.7 ms per token; 1792:936 bandwidth-proportional gives 40.4 ms. Proportions are normalised, so type the bandwidths — -ts 1792,936 — then check it still fits, since you have just moved KV onto the fast card too.
Lanes matter less than power does
Under -sm layer with two GPUs there is one graph-split boundary per forward pass, and what crosses it is a single hidden state: 8192 F32 values, 32,768 bytes per token for Llama-70B. That is 1.0 µs on PCIe 4.0 x16, 4.2 µs on x4, 33 µs on a PCIe 3.0 x1 mining riser — 0.06% of a 58 ms token. Prefill scales it by ubatch (512 × 32,768 = 16.8 MB), so a x1 riser starts to show there. Get x4 Gen4 and stop thinking about it. Under -sm tensor the interconnect is the bottleneck, and that is the only case where lanes earn their money.
Power is what people trip over. NVIDIA rates the 3090 at 350W, the 4090 at 450W and the 5090 at 575W, and recommends a 750W supply for one 3090; two is 700W of GPU before the CPU exists. Layer-split decode loads one card at a time so sustained draw is mild, but prefill and tensor-parallel light both up, and a PSU sized for the average trips on the peak. Stack two triple-slot cards and the top one inhales the bottom one's exhaust, and a card throttling its memory clocks loses exactly the bandwidth you bought.
The rule
Add a second GPU when a model you want does not fit — weights plus KV at the context you actually use. Expect capacity to scale, token rate to stay flat, prefill and multi-user throughput to improve. If you want the same model to generate faster, the purchase is one card with more bandwidth, not two with more VRAM.