This guide is about running open weight models - DeepSeek, Kimi, Qwen, GLM, etc, specifically about LLM inference - which is the process of running a LLM to generate response from a input prompt.
Unlike closed source models like Claude and GPT, you have different options on how to run these models.
The two main options for you to use these models is-
1. Serverless Inference - pay by token (similar to how openai and anthropic models are consumed via api)
2. Dedicated Inference - pay per GPU minute (or hour) of usage. Eg- You could choose to host Deepseek V4 Flash on H100s, B200s, etc via an inference platform like Nava
(Note - other options such as self hosting, PTU and batch processing are discussed later)
So, which inference form should you choose? A common misconception is that serverless is more expensive than dedicated inference, but is it really?
The answer, like most things, is: it depends.
As AI workloads move from prototypes to production, inference can become one of the largest line items in an AI company's infrastructure bill [AI product companies have 50-60% gross margins vs. 80-90% for SaaS primarily due to inference costs (BVP)]. Hence, it is tempting to reduce the serverless vs dedicated decision to a simple question of: Which is cheaper?
That is a useful starting point, but it is not enough.
The right choice depends on the model you are running, the shape of your workload, the hardware available, the optimizations in the serving stack, and what you actually care about from a serving system.

Infrastructure choice is a design decision, not a price comparison — six inputs shape the choice; five properties come out of it.

Serverless and dedicated inference both have a place; the right choice depends on six factors.
We have put together a spreadsheet calculator alongside this post that lets you plug in your own workload and get a rough estimate of which approach makes more sense. More on that later.
There are six things we have found consistently matter when deciding between serverless and dedicated inference.
The model determines the basic economics of inference.
A 4B model and a 400B model have completely different hardware requirements. Architecture matters too. Dense models, MoE models, models with long context windows, and models with different attention mechanisms can have very different throughput characteristics on the same GPU.
Two models with a similar parameter count can therefore have very different economics.
On the exact same agentic coding workload, Muse Glimmer 30B is able to produce 1500 out tok/s on 2xH100, whereas Qwen 27B is able to produce 300 out tok/s. Both are dense ~30B models. Most of this is due to architectural differences (other than the fact that both of these are dense models!)

Model Architecture Drives Real-World Inference Performance — two dense models, similar parameter counts, very different throughput and economics.
The metric - requests per second (rps) doesn’t cover the full workload picture
What matters is the actual token workload:
- Input sequence length (ISL)
- Output sequence length (OSL)
- Cache hit rate (how much of your input tokens remain the same - this affects new computations in the prefill stage)
- Reasoning depth (hence number of reasoning tokens - this is a part of osl)
- Distribution of request lengths
For example, an agentic coding workload may have relatively modest user-facing requests but generate thousands of tokens across many tool calls.
A general chat workload might have much shorter outputs and substantially different economics.
This is why our calculator asks for ISL, OSL and cache hit rate rather than simply asking for requests per second.

The Anatomy of an LLM Inference Workload — requests per second doesn’t tell the whole story — token workload does. Same requests per second. ~290× more tokens to serve.
The workload matters even more than requests per second. Consider these workload categories-
Agentic codingtends to have: High ISL to OSL ratio; Long contexts; Multiple sequential requests; Potentially high reasoning depth; Significant prefix/cache reuse (>95% in many cases). This can make dedicated infrastructure particularly attractive once there is a reasonable baseline of traffic. It also makes serving architecture important. Prefix caching, prefill/decode separation and predictable latency can have a significant impact on both cost and user experience.
General chat typically has: Shorter interactions; Lower output lengths; Less predictable session behavior. Serverless can be attractive at lower volumes, particularly when traffic is genuinely intermittent. As usage grows, however, the baseline can become large enough that dedicated capacity starts making economic sense.
The GPU matters enormously.
A model that requires four H100s might require only two B200s. But the decision is not simply "which GPU is cheaper?"
The relevant question is:
How much does it cost to produce the performance I need?Memory capacity, memory bandwidth, compute, interconnect, and the ability to efficiently run the model's parallelism strategy all matter.
A simple example here basis inferenceX (from semianalysis) LLama 3.3 70B

Same model, Both are FP8 - at similar E2E latency, and same workload - Per chip output throughput is 2.4X for B200 vs H100. Given B200s are ~2x the cost of H100 rental, it may seem like B200 will be the better deal. But infact, what it depends on more is, which usage is your baseline requirement closer to.
If you only need 150 out-tok/s, it does not make any sense to take B200s over H100s.
Though older gpus are less efficient per dollar - as seen in above infographic, they can make more sense at certain usage requirements
This is probably the variable people think about most when considering dedicated inference.
You need to understand three numbers:
- Minimum sustained traffic
- Average traffic
- Peak traffic
Dedicated infrastructure becomes increasingly attractive as the baseline becomes a larger fraction of the peak.
But burstiness does not automatically mean serverless is better. Most dedicated deployments use either autoscaling capability or serverless fallbacks when dedicated capacity reaches a certain threshold.

Traffic Shape Determines Inference Infrastructure Economics — right-size dedicated capacity to your baseline; use serverless to absorb the rest.
Imagine you need enough capacity to handle 100 requests per second at peak. If you only need that capacity for five minutes a day, paying for dedicated capacity all day is probably not attractive.
But imagine your traffic looks more like this:
- Median: 45 req/s
- Average: 65 req/s
- Peak: 100 req/s
You are already consuming a meaningful amount of capacity most of the time. In that situation, continuously provisioned infrastructure can make sense even if the workload has substantial bursts. This is where the difference between peak traffic and baseline traffic becomes important.
The inference optimisations matters almost as much as the hardware. Same model, same GPU can have very different economics due to different inference optimisations.
Some of these techniques are - Speculative decoding, prefix caching, disaggregated prefill and decode, quantization, batching strategies, custom kernels.
A comparison between serverless and dedicated infrastructure is therefore only meaningful if you are comparing similar levels of optimization.
In our testing, we have seen improvements of more than 300% in throughput, when choosing a particular use case and optimising for that.
For instance, we optimised inference for Gemma 4 26B-A4B running on 8*H100 for their user facing production agentic chatbot, we went from 700 output-tok/s to 2900 output-tok/s
Finally, what are you optimizing for? Different buyers have different priorities:
- Lowest possible cost
- Tight P95 latency
- Predictable latency
- High availability
- Data isolation
- Geographic control
- Capacity guarantees
- Ability to customize the runtime
There is no single optimal infrastructure choice without knowing which of these matters.
Here are some comparisons derived from InferenceX by SemiAnalysis - specifically AgentX dataset (agentic coding simulation)
Note- InferenceX published numbers uplifted 50% for the optimisation a provider adds on top, GPU rates from SemiAnalysis three year rental rates, and serverless at the median price across live OpenRouter providers.
First, the same workload on the same GPU (B200s in this case), across models.

Break-even utilization · InferenceX / SemiAnalysis AgentX · agentic coding
Now the same model and workload, across GPUs.
This is the hardware section again, in money. On H200 you need 72% utilization. On GB300 you need 4%. But the GB300 replica here is 56 GPUs producing 86k output tok/s - if your workload cannot fill that, the 4% is meaningless and the H200 number is the one that applies to you.
Advantages of Pure Serverless-
Serverless gives you something valuable: flexibility.
You can:
- Start using a model immediately
- Scale down to zero
- Experiment with many models
- Avoid capacity planning
- Absorb unpredictable demand
For an early-stage product, a new feature, or a workload with genuinely low utilization, those benefits can easily outweigh the higher per-token price.
The question is whether you are still paying for that flexibility after your workload has become predictable.
So how to decide?
Calculator - Serverless vs Dedicated Inference Calculator
We are making the spreadsheet we use for this analysis available for free. You can enter: Model · GPU · ISL · OSL · Cache Hit Rate · Minimum Traffic · Average Traffic · Peak Traffic · Serving Priorities · Estimated Serverless Price · Estimated Dedicated GPU Cost
The calculator then gives you a rough comparison of: Estimated serverless cost · Estimated dedicated cost · Approximate break-even utilization · Potential savings · Capacity requirements · The sensitivity of the result to different workload assumptions
This decision is better understood as a spectrum than as a binary.
At one end is completely elastic serverless inference - pay per token, scale to zero, no capacity planning. At the other is fully provisioned, highly optimised, dedicated infrastructure that keeps running whether or not traffic shows up.
Almost every production setup sits somewhere between the two.
This post compares the two ends plus the hybrid in the middle - dedicated sized to your baseline, serverless absorbing the peaks - because that hybrid is what most teams end up running, and because all three can be priced from public numbers. There are other points on the spectrum we left out of the calculator on purpose.
Batch and off-peak tiers. If the work can wait, the same tokens often cost less. For open weight models it is less tidy - on the aggregators a batch listing is frequently no cheaper than the cheapest live provider, so it has to be checked model by model rather than assumed.
Provisioned throughput. You reserve a guaranteed rate of tokens per minute and pay for that rate rather than for GPUs or for tokens consumed. We left it out because it is quote based, carries a high minimum term, and model coverage is still thin.
Self-hosting. Buying or long-leasing GPUs and running your own serving stack. This is not a cheaper version of dedicated, it is a different cost structure - capex and depreciation over three to five years, power, cooling, spares, networking, the engineers to run it, and the residual value of the hardware at the end.
As inference becomes a larger part of the cost structure of AI products, spending a little time understanding that workload can pay for itself very quickly.
The goal is not to choose dedicated because it is cheaper.
The goal is to choose the infrastructure model that is cheapest for the performance, reliability, predictability and security you actually need.
"My usage keeps changing, so dedicated inference won't make sense for me, right?"
Not necessarily. Dedicated inference does not have to mean a long-term commitment. You can provision dedicated capacity for short periods. If the model you use changes frequently, you can still evaluate and switch deployments relatively quickly. The relevant question is whether each workload has enough baseline demand to justify dedicated capacity while it is running.
"I have very bursty traffic. Shouldn't I always use serverless?"
No. Look at your baseline first. If you have meaningful traffic throughout the day and occasional peaks, dedicated capacity for the baseline combined with serverless overflow can be a better architecture. The important distinction is between bursty and zero-baseline. They are not the same thing.
"Is serverless always more expensive?"
Usually, you should expect to pay a premium for elasticity and operational simplicity. But that premium can be worthwhile if your utilization is low enough. The only useful comparison is the total cost of achieving the performance and reliability your application requires.
"How accurate is the calculator?"
It is intentionally a rough planning tool. Real production economics depend on batching, scheduling, model architecture, runtime versions, GPU availability, utilization, latency requirements and many other factors. Use it to identify where a deeper analysis is worthwhile, not as a precise infrastructure quote.

