Business-standard LLM benchmarks in DataRobot

Each LLM deployment has a ceiling, a latency curve, and a unit price. Most groups function blindly, discovering their deployment limits solely when over-provisioning exhausts their GPU funds or peak site visitors causes a catastrophic failure.

Three numbers matter: most sustained concurrency earlier than GPU saturation, end-to-end latency at that concurrency, and value per million tokens at sustained load. These metrics emerge from how the mannequin interacts along with your {hardware}, runtime, tokenizer, and site visitors combine.

DataRobot 11.8 adjustments that with LLM Profiling Jobs: a local integration of NVIDIA AIPerf, the industry-standard generative AI benchmarking software. One authenticated POST benchmarks any DataRobot LLM deployment serving an OpenAI-compatible net server, sweeps the concurrency vary and use instances you outline, and returns the empirical inputs to Quota Reservations (obtainable in DataRobot 11.9).

Why LLM capability is difficult to foretell

LLM inference doesn’t scale linearly. Compute and reminiscence calls for per request rely dynamically on immediate size, response size, sampling parameters, and KV cache utilization.A deployment that serves 50 brief chat turns per second can stall at 5 long-context RAG requests per second on the identical {hardware}. 4 distinct behaviors make static or speculative capability estimates unreliable:

• Latency is non-linear in concurrency. Time to first token and inter-token latency keep roughly flat throughout a large concurrency vary, then rise sharply as soon as GPU reminiscence bandwidth or compute saturates. TTFT rises when prefill compute saturates; inter-token latency rises when decode reminiscence bandwidth saturates. Which one bites first relies on the workload combine and the deployment’s GPU configuration (single card or a cluster). The saturation knee is the working level that issues, and it could possibly’t be inferred from a single low-load measurement.
• Throughput and latency commerce off. You’ll be able to squeeze extra complete tokens per second out of a deployment by operating it at increased concurrency, at the price of slower per-user response. The suitable trade-off relies on your SLO, not on a generic suggestion.
• Use case combine issues. Two deployments operating the identical mannequin on the identical {hardware} can have very totally different capability if one serves brief Q&A and the opposite serves long-context summarization. The combo must be within the take a look at, or the take a look at is mistaken.
• Caching and routing change the reply. Prefix caching (frequent in agentic coding with periodic compaction) and KV-aware routing can carry efficient throughput dramatically. Profiles run in opposition to a chilly deployment with random inputs symbolize the ground, not the ceiling.

LLM Profiling Jobs make these curves seen.

How LLM benchmarks assist

• Defend capability and quota selections with measured information. When finance questions a four-H100 footprint, or when cross-functional groups negotiate shared capability, you may justify the structure with empirical profiling information. Saturation knee, SLO goal, and forecast site visitors make GPU sizing an evidence-based line merchandise. The identical numbers feed Quota Reservations immediately.
• Account for price per client. Whole token throughput plus the GPU occasion price offers a cost-per-million-tokens determine that helps chargeback or showback. Attribute spend to shoppers proportionally to their reservations, not by guesswork.
• Evaluate fashions and {hardware} on equal phrases. Maintain the workload profile fixed and differ one dimension at a time: the identical mannequin on totally different GPU configurations (a B200 node vs a B300 node, or 4×H100 vs 8×H100), or totally different fashions on the identical configuration (Qwen3.6 35B-A3B MoE vs Qwen3.6 27B dense). As a result of AIPerf metrics match NVIDIA’s printed NIM benchmarks, the numbers are additionally immediately similar to public benchmarks for a similar mannequin and {hardware} mixtures. The suitable enter for procurement and capacity-sizing selections earlier than a {hardware} order.
• Show a change is secure earlier than you ship it. Earlier than a mannequin improve, vLLM bump, driver swap, or GPU migration, rerun the identical profile and examine in opposition to the prior baseline. Regressions present up within the metrics, not in incident stories.

What LLM benchmark metrics imply

The 4 headline metrics AIPerf returns map on to person expertise and to GPU economics:

• Time to first token (TTFT, ms). Measures how lengthy a person waits between submitting a immediate and seeing the primary character; this metric is dominated by prefill compute.
• Inter-token latency (ITL, ms). Common time between successive output tokens as soon as era has began. Units the perceived “typing velocity” of the response.
• Request throughput (requests/sec). Full request-and-response cycles per second on the examined concurrency. The premise for the Capability (RPM) worth on Quota Reservations.
• Whole token throughput (tokens/sec). Whole tokens (enter plus output) processed per second throughout all concurrent requests. The premise for cost-per-token economics.

For every metric, AIPerf stories averages and percentiles (p50, p90, p99). When GPU saturation is detected in the course of the sweep, estimatedCapacity stories the iteration instantly earlier than it. When saturation isn’t detected (the frequent case, for the reason that profiler isn’t co-located with the deployment), estimatedCapacity stories the final iteration examined. Sweep large sufficient that the curve clearly bends, or deal with the end result as a decrease sure.

Submitting a job

A profiling request takes 4 parameters: a deploymentId (the ID of the DataRobot LLM deployment you wish to profile), an inventory of concurrency ranges to comb, a request depend scalar (what number of requests every concurrent employee points), and a number of use instances. Every use case defines an enter sequence size (ISL), an output sequence size (OSL), normal deviations for each, and a weight (prob). Weights throughout all use instances should sum to 100.

export DATAROBOT_ENDPOINT="https://app.datarobot.com"
export DR_API_KEY=""
export HUGGINGFACE_DR_CRED_ID=""
export DEPLOYMENT_ID=""
export CONCURRENCIES="[1,10,50,100]"
export REQUEST_COUNT_SCALAR=2
export MODEL_TOKENIZER="openai/gpt-oss-20b"
export USE_CASES='[{"isl":200,"islStddev":15,"osl":1000,"oslStddev":15,"prob":100}]'
 
curl -X POST -H "Authorization: Bearer ${DR_API_KEY}" 
     -H "Content material-Sort: utility/json" 
     "${DATAROBOT_ENDPOINT}/api/v2/llmProfilingJobs/" 
     -d @- <

A 202 Accepted response returns the job ID, an execution ID, and a standing ID:

{
  "id": "69e09f9e25fdfdfab0d27925",
  "jobExecutionId": "69e09f9f25fdfdfab0d27926",
  "statusId": "5633f028-3f68-4f83-bddc-560d266d6bd2"
}

Monitoring and retrieving LMM benchmark outcomes

Ballot the Standing API with the returned statusId. When the job finishes, the API returns 303 See Different and the Location header factors to the outcomes endpoint:

curl -s -L -i 
  -H "Authorization: Bearer ${DR_API_KEY}" 
  "${DATAROBOT_ENDPOINT}/api/v2/standing/${STATUS_ID}/"

Fetch the total outcomes with the profiling job id:

curl -H "Authorization: Bearer ${DR_API_KEY}" 
     "${DATAROBOT_ENDPOINT}/api/v2/llmProfilingJobs/${LLM_PROFILING_JOB_ID}/profilingResults/"

Instance payload (truncated):

{
  "estimatedCapacity": {
    "metrics": [
      { "name": "request_throughput",     "units": "requests/sec", "measurements": [{ "name": "avg", "value": 8.84    }] },
      { "title": "inter_token_latency",    "models": "ms",           "measurements": [{ "name": "avg", "value": 23.79   }] },
      { "title": "time_to_first_token",    "models": "ms",           "measurements": [{ "name": "avg", "value": 833.06  }] },
      { "title": "total_token_throughput", "models": "tokens/sec",   "measurements": [{ "name": "avg", "value": 4524.80 }] }
    ]
  },
  "outcomes": [ "...per-iteration benchmark data..." ]
}

estimatedCapacity is the sustained working level. outcomes comprises one entry per concurrency stage examined, with the total metric set.

Studying the curve

The estimated-capacity numbers let you know the sustained ceiling. The per-iteration outcomes present you ways the deployment behaves as load climbs towards that ceiling. The desk beneath is an illustrative instance.

When AIPerf detects GPU saturation in the course of the sweep, it identifies the iteration earlier than it (concurrency 50 right here) and returns these metrics as estimatedCapacity. When saturation isn’t detected, estimatedCapacity is just the final iteration examined, which is why the sweep wants to increase previous the knee. Something previous that time trades user-perceived latency for marginal throughput positive factors. If the product spec requires TTFT beneath 1 second, the curve exhibits the deployment helps as much as roughly 50 concurrent requests with margin: provision GPU so peak concurrent demand stays at or beneath that stage.

From profiling end result to Quota Reservations config

The bridge from a profiling run to a Quota Reservations configuration is direct:

For a primer on how Capability, Utilization Threshold, and Reserved % work together beneath load, see Price Limiting vs. Quota Reservations.

A labored price instance

Take the pattern end result: 4,524 complete tokens per second sustained (enter plus output). That’s roughly 16.3 million tokens per hour from one deployment.

If the underlying GPU occasion prices $X per hour, the associated fee per million tokens is $X / 16.3. For an occasion at $4 per hour, that’s about $0.25 per million tokens. For $12 per hour, about $0.74. To calculate price per million output tokens—the usual benchmark for public API comparisons—divide the whole price by the workload’s output share. For instance, given an ISL of 200 and an OSL of 1000, output accounts for roughly 83% of complete tokens. At a $4 hourly occasion worth, this interprets to roughly $0.30 per million output tokens.

Each benchmark run offers you a contemporary, correct cost-per-token determine for the precise mannequin, {hardware}, and quantization mixture you’re operating. After a vLLM improve or a {hardware} swap, re-run the identical profile and make sure your unit economics improved as a substitute of trusting a vendor declare. That is the muse for per-token and per-agent price transparency in chargeback.

Selecting your inputs

A helpful profile begins with two questions: what concurrency vary do you count on in manufacturing, and what does your site visitors truly seem like?

• Concurrencies to comb. Begin large ([1, 10, 50, 100]) to find the saturation knee, then slender (reminiscent of [40, 50, 60, 70]) for an SLO-grade studying round that time.
• Request depend scalar. Set it excessive sufficient that every iteration runs lengthy sufficient to clean out noise. A scalar of two is an affordable place to begin. Elevate it if variance seems excessive.
• Use instances. Match your actual site visitors combine. For those who serve 70% brief chat turns (ISL 200, OSL 300) and 30% long-context RAG (ISL 4000, OSL 800), outline two use instances with prob: 70 and prob: 30. Testing a blended site visitors combine exposes tail-latency habits (reminiscent of p99 spikes) {that a} single-use-case common obscures.
• Tokenizer. Set it explicitly. The benchmark relies on correct token counts, so the matching tokenizer is a part of an accurate measurement.

Operational notes

• Profiling generates artificial load. Run jobs in opposition to a non-production LLM deployment or throughout a upkeep window.
• As a result of the site visitors is artificial, prefill cache hits gained’t seem in token metrics.
• Profiling treats the deployment as a black field. Whether or not the deployment runs on one GPU or many, and no matter mixture of tensor, pipeline, information, or knowledgeable parallelism it makes use of, the profile measures the externally observable end result.
• Jobs will be canceled with a DELETE to the profiling job ID. Cancellation is best-effort and will not cease a run that’s almost full.
• Earlier than you submit, retailer your Hugging Face token in DataRobot Credential Administration as an “API Token (API Key)” credential. AIPerf makes use of it to fetch the mannequin tokenizer, and the saved credential prevents rate-limit errors.

Get entry

LLM Profiling Jobs are in personal preview in DataRobot 11.8. To allow in your tenant, contact your DataRobot account workforce. They may activate the Allow Dynamic Quota Capability Profiling function flag (the inner title for LLM Profiling Jobs) and configure the profiling job picture in your cluster.

Study extra