Over the past two years, model architecture has quietly taken over work that once belonged to the serving stack. Managing the KV cache, quantising the weights, running a draft model for speculative decoding, routing between languages and modalities- much of this is steadily moving into the model.
If we look at this year's major model releases, we see a consistent pattern: each one has absorbed functionality that was, until recently, a serving-side concern.
- At a one-million-token context, DeepSeek-V4-Pro runs on 27% of the per-token inference FLOPs and 10% of the KV cache of DeepSeek-V3.2, the flagship it replaced. It gets there through a hybrid attention design - Compressed Sparse Attention paired with Heavily Compressed Attention - so most of the cache management is a property of the architecture rather than a job for the serving layer. Its MoE experts also ship natively in FP4, which means the quantisation calibration has already been done.
- Tencent’s Hy4 ships a 10-billion-parameter draft model inside the same checkpoint as the main model, so turning on speculative decoding is a configuration flag rather than a second model to source, align, and keep in sync.
- Gemma 4 combines cross-layer KV sharing, p-RoPE, and reuse of the key projection as the value projection to cut its global KV cache by up to 37.5%. This saving is already coming from the weights.
- NVIDIA’s Nemotron 3.5 ASR transcribes 40 language-locales from one 600M checkpoint through prompt conditioning, deleting the language-ID router and the per-language model pool that voice platforms once used to be built around.
This raises the question: with so much moving into the weights, what does the runtime still own?
To provide a short answer, runtime’s job is not disappearing; it is being redefined. And it is not being reformulated evenly: how much moves, and what gets left behind, also depends on what you are serving- the workload.
In May 2024, DeepSeek-V2 introduced Multi-head Latent Attention, which compresses every attention head’s keys and values into one small shared vector instead of storing a pair per head. The result was a staggering 93.3% smaller KV cache and 5.76x the maximum generation throughput of the model it replaced.
KV memory management had always been a serving-side problem - PagedAttention, cache offloading, and eviction policies all exist to deal with it. DeepSeek did not build a better cache manager; it changed the architecture so there was less cache to manage.
The work that is gradually transitioning from the serving layer into the weights can be grouped into six jobs.
Every token leaves behind a key and a value, kept for as long as the request lives. Architectures now shrink that before any runtime sees it, and most checkpoints combine several of the available mechanisms: compressing the cache into a smaller shared representation (Multi-head Latent Attention); bounding it, so a layer keeps only the last N tokens instead of the full history (sliding-window attention); halving it by reusing the key projection as the value projection (Gemma 4); replacing it with a fixed-size recurrent state that does not grow with the sequence at all (linear-attention layers); and sharing it, so several layers read one cached set of keys and values rather than computing their own (cross-layer KV sharing, which Gemma 4 uses alongside a 5:1 local-to-global layer schedule and p-RoPE to reach its 37.5% figure).
Two further decisions now shape the cache as much as the attention formula does. The first is storage format: DeepSeek-V4 keeps the RoPE dimensions of each KV entry in BF16 and the remainder in FP8, which nearly halves the cache against pure BF16. The second is the draft head - the small multi-token-prediction module that proposes tokens for the main model to verify. Gemma 4's drafter reads the target model's existing keys and values rather than building a cache of its own, so turning on speculative decoding no longer allocates a second cache per request.
Hybrid sparse designs such as DeepSeek-V4's Compressed Sparse Attention (CSA) sit across two of these jobs. Their primary effect is on retrieval, since they decide which positions get read - but because CSA pools groups of tokens into single compressed entries before anything is selected, it also reduces what has to be stored. Selection is covered under Retrieval below; the pooling belongs here. Attention design still supplies most of the saving, but the cache is now shaped by the layer schedule, the storage format, and the draft head as well.
What is left for the runtime is a different problem. The cache is smaller, and it is shaped differently in each model family, which matters most for capacity planning. In hybrid and linear-attention models, the unit of accounting changes: the recurrent layers hold one fixed-size state per concurrent request, and that state costs the same whether the conversation is 2K tokens or 200K, while the minority of full-attention layers still grow with context length. So the ceiling is set mostly by how many requests run at once and only partly by how long they are - the reverse of what most capacity planning assumes.
Attention across a million tokens is unaffordable if every position must consult every other position, and the serving-side answer was largely heuristic: trim the context, drop unimportant tokens, reuse shared prefixes, or move retrieval outside the model entirely. Sparse-attention architectures replace those heuristics with a trained indexer that scores positions and selects a subset - the top 2,048 in both GLM-5.2 and Hy4 - so expensive attention runs only over the selection.
Because adjacent layers tend to want similar positions, GLM-5.2’s IndexShare and Hy4’s IndexCache compute that selection once and reuse it across several layers, which is where a substantial part of the saving actually comes from. DeepSeek-V4 stages the same idea: Compressed Sparse Attention pools every four tokens into one compressed entry before the indexer chooses from it, so selection runs over a sequence already four times shorter, while Heavily Compressed Attention compresses 128x and attends densely over what remains.
Kimi K3’s Attention Residuals let each layer read selectively from the representations of all preceding layers rather than receiving everything through one accumulated residual stream - reading across depth rather than across sequence - and DeepSeek-V4’s Manifold-Constrained Hyper-Connections and Hy4’s Identity Hyper-Connections widen the residual path for the same purpose. What a model can reach is now a property of the residual topology as much as of the attention pattern.
The runtime is relieved of guessing which context matters. In exchange, it inherits something harder: the selection happens inside the model, so when a sparse model overlooks a fact that was demonstrably in the prompt, there is no cache-hit log to inspect and no threshold to adjust.
Mixture-of-Experts severs the link between how large a model is and what a forward pass costs. Hy4 activates 49B of its 770B parameters; Kimi K3 fires 16 of 896 routed experts plus 2 shared. This is the one job the runtime never held, so what comes back is not a saving but a constraint: the whole model still has to be resident in memory even though only a fraction of it runs per token, and the experts that do run have to be reached across devices. A model with a small active footprint and an enormous total footprint has a very different memory-to-compute ratio from a dense model of comparable quality, and that changes how it should be sharded, how many replicas fit on a node, and how much interconnect bandwidth the expert routing will demand. Sparsity did not remove the bottleneck so much as move it out of arithmetic and into memory capacity and the network between devices.
Autoregressive decoding yields one token per forward pass, and speculative decoding was the standard way around it: run a small, fast draft model, let it propose several tokens, and have the target model verify them in a single pass. Making that work was a serving project in itself - find or train a compatible drafter, match its tokeniser, keep two checkpoints in step through every update, tune the acceptance threshold. The drafter now arrives in the box. Hy4 carries a dedicated 10B multi-token-prediction (MTP) layer; Gemma 4 a 430–500M drafter built from three local and one global attention layer that cross-attends to the target model’s own KV cache; MiMo-V2.5 a three-layer MTP head, and GLM-5.2 and Qwen3.8 their own equivalents. This is native speculative decoding: the drafter was trained alongside the model it drafts for, so the provisioning and alignment problem disappears.
What remains for the serving layer is smaller than what it replaced. Draft depth - how many tokens to speculate per step - is still a tuning decision, and the right value depends on traffic, because speculation pays at low batch sizes and can cost throughput at high ones. That makes acceptance rate and batch size something to watch together, and the drafter still has to be supported by whichever attention backend you are running. Even so, this is one of the few places where the runtime’s job got genuinely easier.
Until about the end of 2024, quantisation was a post-hoc serving step: start from BF16 weights, pick a format, calibrate on sample data, measure what quality was lost, and decide whether to accept it. The turning point was DeepSeek-V3 in December 2024, the first open model at scale to be pre-trained in FP8 mixed precision rather than quantised afterwards, which meant the weights had learned to live in low precision instead of being squeezed into it. By 2026, shipping a first-party low-precision checkpoint has become routine. DeepSeek-V4 ships its MoE experts natively in FP4, with the remaining attention, norm, and router parameters in FP8. MiMo-V2.5 ships block-wise FP8 weights and was likewise pre-trained in FP8 mixed precision. Gemma 4 releases quantisation-aware-trained checkpoints alongside the raw ones, and Hy4 and Qwen3.5 publish first-party FP8 and NVFP4 variants.
So the serving layer no longer owns the quantisation recipe, which removes a real source of quality risk. It has not stopped owning precision. KV-cache precision is still a serving choice, and low-precision weights interact with parallelism in ways that often surface under load.
A multimodal system used to be assembled from parts: a vision encoder running as its own service, an audio front end, adapters to translate between representations, a language-identification step, a pool of per-language models, and the routing logic to choose between them. Those parts are now being folded into the weights as omnimodal architectures, where one checkpoint ingests several modalities natively instead of being wrapped in converters. MiMo-V2.5 carries a 729M-parameter vision transformer and a 261M audio encoder inside the model, so text, image, video, and audio arrive in one architecture instead of four services. Gemma 4’s 12B variant dispenses with encoders altogether and ingests raw audio and image patches. Nemotron 3.5 ASR replaces a language-ID router and a per-language model pool with prompt conditioning.
What is left is orchestration of whatever stays separate, and what stays separate differs by modality. Vision-language systems shed a vision encoder and cross-modal adapters; speech systems shed an audio front end, a language-ID step, and a per-language model pool. Absorbing them leaves very different work behind. In vision-language, the encoder is compute-bound while the language decoder is memory-bandwidth-bound, so folding the encoder into the checkpoint does not make the two halves want the same hardware: the runtime inherits the job of splitting the model across hardware tiers at the modality boundary, plus the volume of visual KV that text models never generate. In speech, what disappeared was routing and duplication rather than compute, so the residue is genuinely smaller and different in kind - stream and session lifecycle, concurrency per chunk size, endpointing and barge-in.
Each of these six jobs leaves a different residue behind. Latent compression changes what attention stores, sparse attention changes which positions it reads, and Mixture-of-Experts changes how much it computes. Multi-token prediction changes how many tokens leave per forward pass, native low precision changes the numeric contract the weights arrive under, and in-model encoders change the shape of the pipeline around the model.
Essentially, these are six separate bargains. Each buys its efficiency by giving something up - precise recall, dense arithmetic, numeric headroom, architectural independence - and what it gives up decides what the runtime has to watch. A model that compressed its cache has made a very different promise from one that sparsified its attention or shrank its numeric format, and the two will fail in different places. So it is not only how much the architecture absorbed, but also which of these bargains it struck for the workload you actually run.
For different workloads, model architecture absorbs different jobs, thereby leaving different amounts of work for the runtime in each case.
Agentic coding is one of the hardest cases: repo-scale context, million-token windows, thousands of sequential tool calls. It is also where the architecture has absorbed the most. The flagship coding models - DeepSeek-V4, Kimi K3, GLM-5.2, Hy4, and MiMo-V2.5-Pro - each take on four or five of the six jobs. They shrink their own KV cache (storage), select which positions to attend to rather than reading everything (retrieval), activate a fraction of their parameters per token (activation), ship their own draft head (emission), and arrive in a first-party low-precision format (precision). Only pipeline does not apply, because these are text-only models. Across the group, five of the six jobs have moved into the weights.
The residue is largest here. Each of these models buys its saving by reading less of the context in full - either by selecting a subset of positions to attend to, or by interleaving fixed-state layers with a minority of full-attention ones. The saving is real, but it means long-range recall is now a property of the specific checkpoint and the specific ratio it was built at. For coding, where a fix may depend on a definition two hundred thousand tokens back, that has to be properly tested per model.
The same fixed-state layers also change what limits capacity. A recurrent state slot is held for each concurrent request whether that request is short or long, so the ceiling is set by concurrency rather than by context length. Expert sparsity shifts the bottleneck again: at Kimi K3’s 16-of-896 routing, interconnect bandwidth rather than arithmetic becomes the binding resource, and Moonshot open-sourcing its attention kernels and MoE communication library alongside the weights is a cue. And DeepSeek-V4 and Hy4 each need their own tool-call and reasoning parsers, a category of serving work that did not exist two years ago.
That is five of the six jobs - storage, retrieval, activation, emission, and precision - absorbed by the architecture, with only pipeline left untouched, because these are text-only models.
So for agentic coding, the runtime keeps five things: validate long-range recall, provision by concurrency rather than context length, shard for interconnect rather than for compute, tune draft depth against live acceptance rates, and maintain per-model parsers and attention backends.
Chat makes the same architectural trade as coding, and comes out ahead on it. The trade is cheaper memory and cheaper arithmetic in exchange for weaker precise recall for long context. Gemma 4 26B-A4B routes 8 of 128 experts, so inference cost tracks a 4B model rather than a 25B one; it interleaves local and global attention at 5:1; and it stacks three KV optimisations for up to a 37.5% cut in the global cache. What a 5:1 schedule gives up is the ability to retrieve one specific token from far back in the context - and a chat turn needs the last few exchanges and not a token buried 200K positions back.
The architecture is not more efficient for chat than for coding. It made the identical trade in both cases. Chat simply does not need much of what was traded away.
Jobs absorbed: storage, activation, and emission. What remains is mostly pertaining to economics - batching, cache-aware routing, right-sized concurrency, cold start, and autoscaling.
Vision-language is where the simple story breaks. Fewer jobs move, and the ones that do barely touch the dominant cost. Vision encoding is compute-bound; language generation is memory-bandwidth-bound. Shrinking the decoder's KV cache makes the memory-bound half cheaper and does nothing for the compute-bound half, so in image- and video-heavy traffic the encoder can remain the larger share of the cost after every attention optimisation has landed.
The runtime picks up a job here that text serving never had: running the two halves as separate services, on hardware chosen for each. Here the serving job got larger while the architecture got more efficient.
Streaming speech is the cleanest absorption in the set, and a completely different job from any of the text cases. Streaming ASR used to re-encode overlapping windows of audio at every step, so the same audio was processed several times over, and the serving stack existed largely to manage that buffer. NVIDIA's Nemotron 3.5 ASR keeps an internal cache of its own attention and convolution activations and encodes each frame once. Two components that earlier voice platforms were built around, the language-ID router and the per-language model pool, were made redundant by that one release. It also runs the migration backwards: the look-ahead setting is a latency-versus-accuracy control the architecture exposed.
What is left for the serving layer is stream and session lifecycle, concurrency per chunk size, endpointing (deciding when a speaker has finished) and barge-in (letting a caller interrupt playback), and orchestration with the downstream language model and speech synthesis.
Table 1. How the shift lands, by modality. Models per row: agentic coding (DeepSeek-V4, Kimi K3, GLM-5.2, Hy4, MiMo-V2.5-Pro); chat (Gemma 4 26B-A4B); vision-language (Gemma 4 31B, Qwen3.8-27B, MiMo-V2.5); speech (Nemotron 3.5 ASR). The last column is the finding: the residual work does not shrink uniformly, and in one row it grows.
Everything the model absorbed is per-model work. Most of what stayed with the runtime is cross-request or cross-model. The per-model job shrank; the between-models job did not. Each new architecture arrives with its own memory profile, its own parallelism requirement and its own attention backend, and a cluster has to hold all of those at once - so each release adds another configuration the fleet has to accommodate, mostly not something that it can standardise on. The centre of gravity now moves to the fleet, and four things should be considered.
Utilisation. A GPU costs the same idle or saturated, so real cost per token is hardware-hours divided by tokens actually served. Batch continuously; right-size parallelism so you neither strand memory nor over-shard; cache and route so shared context is computed once; autoscale to the real curve rather than to peak. Prefill-decode disaggregation (pays when you have enough sustained traffic to keep both pools busy, and strands capacity when you do not). For vision-language, add a third pool for the encoder and split earlier, because the encoder wants different hardware rather than just a different batch shape.
Multi-tenancy and routing. Any platform that offers a handful of open-weight models is already running dense, MoE and hybrid architectures side by side, and the published serving recipes make the divergence concrete: each model prescribes its own parallelism layout, its own attention backend and its own cache settings. Serving one model well is a solved problem. Keeping a mixed set of them at high utilisation on shared hardware is the part that is not, and it is where the levers sit: cache-aware routing, so a request lands on the replica that already holds its prefix; per-model capacity policy, meaning how many replicas each model gets and how each is sharded; admission control when demand exceeds capacity; prefill-decode segregation, so long prompts do not stall short ones; and placement that keeps each model on the hardware tier it actually suits.
Security, isolation and operations. As inference handles more sensitive data and more autonomous tool use, the runtime owns a widening surface: the input and output guardrails, model-supply-chain integrity, tenant isolation, and at the deepest level confidential execution that keeps data and weights encrypted while the GPU computes. Around it sits the operational work - rollout, observability, autoscaling, and SLAs.
These levers also influence how two providers serving the identical open-weight model land at very different cost, throughput, and latency.
Models and technical reports. DeepSeek-V2 / Multi-head Latent Attention (arXiv:2405.04434); DeepSeek-V3 (arXiv:2412.19437); DeepSeek-V4 model card and technical report (arXiv:2606.19348); Kimi K3 technical report (arXiv:2607.24653) and Kimi Linear (arXiv:2510.26692); GLM-5.2 (zai-org/GLM-5.2); Tencent Hy4-preview model card; MiMo-V2.5 and MiMo-V2.5-Pro model cards (XiaomiMiMo); Gemma 4 technical report (arXiv:2607.02770); Qwen3.8-27B model card and serving recipes; Nemotron 3.5 ASR model card (NVIDIA).
Research. Wang et al., “A Systematic Analysis of Hybrid Linear Attention” (arXiv:2507.06457, preprint); Zhou et al., “Attention Amnesia in Hybrid LLMs” (arXiv:2606.11052, preprint); Yu, “Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity” (arXiv:2603.12707, preprint); Hsieh et al., RULER (arXiv:2404.06654; COLM 2024); Liu et al., “Lost in the Middle” (TACL 2024).
Vendor-reported figures are as published by the labs and have not been independently reproduced.

