Benchmark Testing

This page summarizes the vGPU overhead results that TensorFusion has published and tells you how to rerun them on your own hardware. The numbers come from the NexusGPU/benchmark repository and date from July 2025.

Read this first

The results measure the API-remoting path (client library plus worker), where every GPU call crosses a process or network boundary. This is the path used by remote vGPU and by hard isolation.
There are no published results yet for soft isolation, the default mode, in which a limiter runs inside the application process and no worker is involved. Do not apply the numbers below to it.
The published results do not record the GPU model, driver version or instance type. Treat them as relative overhead between modes on one machine, not as absolute performance.
The "Same AZ" and "Cross AZ" columns were produced on a single node with artificial latency, not on two machines.

If you need numbers for a purchasing or capacity decision, run the benchmark on your own cluster.

What was measured

Each test runs the same workload in up to five setups and compares wall-clock time with the native run. "Loss" is the extra time relative to native; lower is better and a negative value means the run was faster than native.

Setup	Meaning
Native	The application talks to the GPU driver directly, without TensorFusion.
NGPU	Client and worker run inside the application process.
Local	Client and worker are separate processes on the same node.
Same AZ	Remote worker, with 0.3 ms of added network latency.
Cross AZ	Remote worker, with 1 ms of added network latency.

TorchBench inference (2025-07-06)

Models from TorchBench, eval mode on CUDA. Time is total seconds for the configured number of iterations.

Model	Native	NGPU	Loss	Local	Loss	Same AZ	Loss	Cross AZ	Loss
basic_gnn_edgecnn	41.15 s	40.95 s	-0.49%	43.48 s	5.66%	46.07 s	11.96%	54.97 s	33.58%
BERT_pytorch	249.02 s	248.84 s	-0.07%	251.26 s	0.90%	253.71 s	1.88%	261.62 s	5.06%
basic_gnn_gcn	15.05 s	15.24 s	1.26%	19.63 s	30.43%	29.70 s	97.34%	64.39 s	327.84%
basic_gnn_gin	9.47 s	9.53 s	0.63%	9.78 s	3.27%	12.66 s	33.69%	21.83 s	130.52%
hf_Albert	24.73 s	24.00 s	-2.95%	29.19 s	18.03%	39.19 s	58.47%	73.00 s	195.19%
hf_Bart	39.88 s	38.68 s	-3.01%	54.96 s	37.81%	94.17 s	136.13%	211.68 s	430.79%
hf_Bert	24.15 s	24.35 s	0.83%	29.55 s	22.36%	42.00 s	73.91%	75.86 s	214.12%
llama	39.91 s	41.20 s	3.23%	42.90 s	7.49%	45.80 s	14.76%	52.55 s	31.67%
hf_distil_whisper	170.61 s	170.87 s	0.15%	172.16 s	0.91%	178.75 s	4.77%	189.45 s	11.04%
hf_clip	191.60 s	191.70 s	0.05%	194.52 s	1.52%	197.51 s	3.08%	208.90 s	9.03%
hf_Whisper	58.98 s	59.18 s	0.34%	63.50 s	7.66%	66.66 s	13.02%	72.63 s	23.14%
Average loss			0.00%		12.37%		40.82%		128.36%

What the table shows:

In-process mode (NGPU) is within about 3% of native on every model.
Once calls cross a process boundary, overhead varies widely by model. hf_Bart, basic_gnn_gcn and hf_Bert lose 20–40% locally, while BERT_pytorch, hf_clip and hf_distil_whisper lose under 2%. This is consistent with a fixed cost per GPU call: short runs made of many small calls pay the most.
Network latency widens the gap. With 1 ms of latency the first group takes 3–5 times as long as native, while the second group stays within about 11%.

MLPerf Inference (2025-07-04)

bert-99, reference PyTorch implementation, edge category, SingleStream scenario.

Setup	Time	Loss
Native	27.008 s	-
Local	29.930 s	10.82%
Same AZ	33.341 s	23.45%
Cross AZ	41.597 s	54.02%

Run it yourself

Prerequisites: a Kubernetes cluster with TensorFusion installed, a GPU node with NVIDIA drivers, and Helm 3.

Clone the benchmark repository and install its chart. It creates a Deployment for manual runs and a CronJob for scheduled runs.
```
git clone https://github.com/NexusGPU/benchmark.git
cd benchmark
helm install tf-bench ./helm/torchbench -f custom-values.yaml
```
Set the image, node selector and TensorFusion Pod annotations for your cluster in custom-values.yaml. The chart defaults point at the authors' test environment.

Run one TorchBench model inside the benchmark Pod:

cd benchmark
python3 test.py -k "test_${model_name}_eval_cuda" -t ${eval_times}

Run the MLPerf test:

mlcr run-mlperf,inference,_full,_r5.0-dev \
    --model=bert-99 \
    --implementation=reference \
    --framework=pytorch \
    --category=edge \
    --scenario=SingleStream \
    --execution_mode=valid \
    --device=cuda \
    --quiet --rerun

Repeat each run natively (without the TensorFusion annotations) and with each vGPU setup you want to compare, then compute (vgpu_time - native_time) / native_time.

To reproduce the latency columns on one node, add delay to the loopback interface before the run and remove it afterwards:

tc qdisc add dev lo root netem delay 0.3ms   # "Same AZ"; use 1ms for "Cross AZ"
tc qdisc del dev lo root

Next steps

Read this first

The results measure the API-remoting path (client library plus worker), where every GPU call crosses a process or network boundary. This is the path used by remote vGPU and by hard isolation.
There are no published results yet for soft isolation, the default mode, in which a limiter runs inside the application process and no worker is involved. Do not apply the numbers below to it.
The published results do not record the GPU model, driver version or instance type. Treat them as relative overhead between modes on one machine, not as absolute performance.
The "Same AZ" and "Cross AZ" columns were produced on a single node with artificial latency, not on two machines.

If you need numbers for a purchasing or capacity decision, run the benchmark on your own cluster.

What was measured

Setup	Meaning
Native	The application talks to the GPU driver directly, without TensorFusion.
NGPU	Client and worker run inside the application process.
Local	Client and worker are separate processes on the same node.
Same AZ	Remote worker, with 0.3 ms of added network latency.
Cross AZ	Remote worker, with 1 ms of added network latency.

TorchBench inference (2025-07-06)

Models from TorchBench, eval mode on CUDA. Time is total seconds for the configured number of iterations.

Model	Native	NGPU	Loss	Local	Loss	Same AZ	Loss	Cross AZ	Loss
basic_gnn_edgecnn	41.15 s	40.95 s	-0.49%	43.48 s	5.66%	46.07 s	11.96%	54.97 s	33.58%
BERT_pytorch	249.02 s	248.84 s	-0.07%	251.26 s	0.90%	253.71 s	1.88%	261.62 s	5.06%
basic_gnn_gcn	15.05 s	15.24 s	1.26%	19.63 s	30.43%	29.70 s	97.34%	64.39 s	327.84%
basic_gnn_gin	9.47 s	9.53 s	0.63%	9.78 s	3.27%	12.66 s	33.69%	21.83 s	130.52%
hf_Albert	24.73 s	24.00 s	-2.95%	29.19 s	18.03%	39.19 s	58.47%	73.00 s	195.19%
hf_Bart	39.88 s	38.68 s	-3.01%	54.96 s	37.81%	94.17 s	136.13%	211.68 s	430.79%
hf_Bert	24.15 s	24.35 s	0.83%	29.55 s	22.36%	42.00 s	73.91%	75.86 s	214.12%
llama	39.91 s	41.20 s	3.23%	42.90 s	7.49%	45.80 s	14.76%	52.55 s	31.67%
hf_distil_whisper	170.61 s	170.87 s	0.15%	172.16 s	0.91%	178.75 s	4.77%	189.45 s	11.04%
hf_clip	191.60 s	191.70 s	0.05%	194.52 s	1.52%	197.51 s	3.08%	208.90 s	9.03%
hf_Whisper	58.98 s	59.18 s	0.34%	63.50 s	7.66%	66.66 s	13.02%	72.63 s	23.14%
Average loss			0.00%		12.37%		40.82%		128.36%

What the table shows:

In-process mode (NGPU) is within about 3% of native on every model.
Once calls cross a process boundary, overhead varies widely by model. hf_Bart, basic_gnn_gcn and hf_Bert lose 20–40% locally, while BERT_pytorch, hf_clip and hf_distil_whisper lose under 2%. This is consistent with a fixed cost per GPU call: short runs made of many small calls pay the most.
Network latency widens the gap. With 1 ms of latency the first group takes 3–5 times as long as native, while the second group stays within about 11%.

MLPerf Inference (2025-07-04)

bert-99, reference PyTorch implementation, edge category, SingleStream scenario.

Setup	Time	Loss
Native	27.008 s	-
Local	29.930 s	10.82%
Same AZ	33.341 s	23.45%
Cross AZ	41.597 s	54.02%

Run it yourself

Prerequisites: a Kubernetes cluster with TensorFusion installed, a GPU node with NVIDIA drivers, and Helm 3.

Clone the benchmark repository and install its chart. It creates a Deployment for manual runs and a CronJob for scheduled runs.
```
git clone https://github.com/NexusGPU/benchmark.git
cd benchmark
helm install tf-bench ./helm/torchbench -f custom-values.yaml
```
Set the image, node selector and TensorFusion Pod annotations for your cluster in custom-values.yaml. The chart defaults point at the authors' test environment.

Run one TorchBench model inside the benchmark Pod:

cd benchmark
python3 test.py -k "test_${model_name}_eval_cuda" -t ${eval_times}

Run the MLPerf test:

mlcr run-mlperf,inference,_full,_r5.0-dev \
    --model=bert-99 \
    --implementation=reference \
    --framework=pytorch \
    --category=edge \
    --scenario=SingleStream \
    --execution_mode=valid \
    --device=cuda \
    --quiet --rerun

Repeat each run natively (without the TensorFusion annotations) and with each vGPU setup you want to compare, then compute (vgpu_time - native_time) / native_time.

To reproduce the latency columns on one node, add delay to the loopback interface before the run and remove it afterwards:

tc qdisc add dev lo root netem delay 0.3ms   # "Same AZ"; use 1ms for "Cross AZ"
tc qdisc del dev lo root

Read this first

What was measured

TorchBench inference (2025-07-06)

MLPerf Inference (2025-07-04)

Run it yourself

Next steps

Table of Contents

Benchmark Testing

Read this first

What was measured

TorchBench inference (2025-07-06)

MLPerf Inference (2025-07-04)

Run it yourself

Next steps

Table of Contents