NVIDIA benchmarks confidential inference overhead on Blackwell
NVIDIA reports that DeepSeek-R1 inference on eight B200 GPUs under Intel TDX retains 96.1–98.2% of non-confidential throughput, a figure that matters to teams weighing private inference against performance cost.
NVIDIA's performance engineering team published benchmarks showing that confidential computing on Blackwell GPUs costs relatively little throughput for a large-scale inference workload, alongside the runtime changes needed to get there.
The test ran nvidia/DeepSeek-R1-0528-NVFP4 on a single DGX B200 system (8 B200 GPUs) under Intel TDX, using TensorRT LLM 1.3.0rc22 with the PyTorch backend, TP=8, FP8 KV cache, and a 32K-input/1K-output sequence at concurrency levels of 1, 2, 4, 8, and 16. Holding model, hardware, framework version, and concurrency constant and toggling only confidential-compute state, the team measured 96.1–98.2% of baseline output-token throughput retained, with mean time per output token 1.2–4.3% above baseline.
NVIDIA deliberately chose a workload designed to expose overhead: long context stresses prefill data movement, extended generation amplifies per-token costs during decode, and low concurrency removes the chance to hide stalls behind other requests.
The post also details three CC-specific adaptations in TensorRT LLM. Host-to-device transfers pass through a software-encrypted bounce buffer, so pinned memory loses its asynchronous advantage; the framework now selects pageable memory on affected paths and moves token/sampling readback to an async worker (PR #11573). CUDA-event timestamps proved unstable under CC, so the kernel autotuner uses the GPU %globaltimer instead (PR #11657). NVLink SHARP multicast is unavailable in B200 CC configurations, which NVIDIA says frameworks should detect before choosing collective algorithms.
The figures cover one model, one platform, and one hardware configuration; NVIDIA presents the methodology for teams to reproduce on their own workloads.