CPU vs H100
━━ CPU
━━ H100
Input and output token throughput on CPU and GPU
Measured completed-request tokens per second in ten-second buckets during the first 120 seconds of a median-throughput trial.
Replay
Elapsed
120 s