VZU Insights · VZU essay · 2026-06-20
On-prem AI, in 2026.
Why some operators still need on-prem. The data residency argument. The audit trail argument. The deployment model VZU ships.
The essay
On-prem AI, in 2026.
On-prem AI is making a comeback in 2026. Three reasons: data residency, audit trail, and predictable cost. This essay is the operator's decision tree for whether on-prem is the right answer for any given workload.
Data residency is the easy case. If the regulator requires that customer data never leave a specific jurisdiction, and the cloud provider does not have a region in that jurisdiction, on-prem is the answer. The same data residency argument applies to data sovereignty (the data must stay inside the operator's perimeter for security reasons) and to data gravity (the data is too large to move).
Audit trail is the harder case. If the auditor requires that the inference log, the model weights, and the prompt history all live on infrastructure the operator physically controls, on-prem is the answer. The cloud provider will give part of this. On-prem gives all of it. The operators in regulated industries (healthcare, finance, government) are increasingly choosing on-prem by default, not by exception.
Predictable cost is the hardest case. At scale, the cloud bill for inference is variable and can spike. Some operators prefer the fixed cost of hardware to the variable cost of tokens. The break-even for replacing a cloud inference bill with on-prem hardware is around 800 hours/month of inference, or roughly 11 hours/day. Most production workloads run 24/7, so on-prem wins on cost above that threshold.
The hybrid case is the most common in production. The model runs in the cloud for most queries. A small subset of queries (the ones that touch regulated data) are routed to an on-prem inference server. The on-prem server may be a smaller model (a fine-tuned 7B instead of a 70B), running on commodity hardware, inside the operator's data center. The Netwit runtime supports this pattern natively — the orchestration layer can route individual tool calls to on-prem or cloud endpoints based on data classification.
The decision tree: data residency required → on-prem. Audit trail required → on-prem or private cloud. Predictable cost required → on-prem above the break-even. Otherwise → cloud. The Netwit runtime ships the same code in all three modes, so the operator can change their mind later without rewriting the agent.
Disclosure
Dhiraj Chatpar is the founder of VZU. The VZU team wrote this essay. The runtime described is in production at a small number of operators. The benchmarks, where cited, are from internal testing or from operators who have given permission to publish.