Operating and Governing AI-Native Infrastructure: Metrics, Budget, Isolation, Sharing, SLO to Cost

The key to governing AI-native infrastructure lies in how to institutionalize the closed-loop management of costs and risks arising from uncertainty.

In the cloud-native era, system operations were typically considered “basically deterministic”: request paths were predictable, resource curves were relatively stable, and scaling could respond promptly to load changes. However, entering the AI era, this assumption no longer holds—uncertainty has become the norm.

This chapter aims to provide CTOs/CEOs with key conclusions for architecture reviews:

The starting point of AI-native infrastructure is to treat uncertainty as the default input; The goal is to achieve closed-loop governance of the resource consequences (cost, risk, experience) arising from uncertainty.

This is also why “becoming AI-native” in organizational contexts increasingly points to the reshaping of operational methods and governance models: when system consequences are amplified, governance must be institutionalized.

What is an “Uncertain System”

In this handbook, “uncertainty” does not refer to randomness in the probabilistic sense, but to three types of phenomena in engineering practice:

• Unpredictable behavior: execution paths change dynamically with model inference, especially evident in agentic processes (Agent intelligent workflows).
• Unpredictable resource consumption: tokens, KV cache, tool calls, I/O, and network overhead exhibit long-tail and burst characteristics.
• Non-linear consequences: the same “intent” can produce cost and risk outcomes differing by orders of magnitude.

Therefore, the infrastructure problem of AI-native infrastructure has shifted from “how to make the system more elegant” to:

How to ensure the system maintains economic viability, controllability, and recoverability when worst-case scenarios occur.

During architecture reviews, if you cannot answer “what is the worst case, where are the upper bounds, and how to degrade/rollback when triggered,” you are still reviewing the 惯性 extension of deterministic systems, not true AI-native systems.

Major Sources of Uncertainty

The following table summarizes common sources of uncertainty in AI-native infrastructure and their specific manifestations, facilitating quick reference for CTOs/CEOs.

Behavior Uncertainty

Behavior uncertainty is mainly reflected in changes to agent task decomposition paths, dynamic adjustment of tool selection and call sequences, and path explosion caused by failure retry, reflection, and multi-round planning. Tools and contexts are combined through standard interfaces (such as MCP protocol integration), significantly expanding system capability surfaces while making branch space a governance challenge.

More critically, tool calls are not “free external functions” - they occupy context windows and consume token budgets, amplifying cost and tail latency pressures. Therefore, behavior uncertainty is not merely “feature flexibility” at the product layer, but “cost and risk elasticity” at the platform layer, which must be budgeted, capped, and made auditable.

Demand Uncertainty

Demand uncertainty includes concurrency and burst (peaks), long-tail requests (ultra-long contexts, complex reasoning), and mutual interference under multi-tenancy (noisy neighbor). This drives capacity planning from “average capacity” to “tail capacity + governance strategies.”

In AI-native infrastructure, experience and cost are often determined not by average requests, but by the combination of tail requests: a small number of long-chain, long-context, tool-intensive requests can overwhelm shared resource pools. Therefore, demand uncertainty requires answering: which requests deserve guarantees, which must be throttled, and which should be isolated.

State/Context Uncertainty

State uncertainty is the most underestimated category in the AI era: context is a state asset, and it often exists across requests. When inference state / KV cache is elevated to a reusable, shareable, migratable system capability, it is no longer an application detail but a decisive variable for throughput and unit cost. NVIDIA in public materials identifies Inference Context Memory Storage as a new infrastructure layer, pointing to state reuse and sharing requirements for long-context and agentic workloads.

The conclusion is: “context/state” has changed from optional optimization to a critical infrastructure asset that must be meterable, allocable, and governable.

Infrastructure Uncertainty

AI workloads are far more sensitive to network, interconnect, and storage than traditional microservice workloads. Congestion, packet loss, and I/O jitter are amplified into tail latency and job completion time instability, creating “non-linear consequences” for experience and cost.

This type of uncertainty usually cannot be solved through “component selection” but requires end-to-end path engineering constraints: from topology, bandwidth, and queuing, to transport protocols, isolation strategies, and congestion control—all must be incorporated into the governance plane, not just the operations plane.

How Uncertainty Amplifies Across Layers

The diagram below illustrates the closed-loop relationship between metrics, budgets, and isolation strategies, emphasizing that governance must be rewritable.

The flowchart below demonstrates the cross-layer amplification path of uncertainty in AI-native infrastructure:

Typical phenomena include:

• Agent branch explosion: more tools and composable paths make tail costs increasingly uncontrollable.
• Context inflation: long contexts and multi-round reasoning make KV cache a performance bottleneck and cost black hole.
• Resource contention distortion: GPU/network contention under multi-tenancy makes “average performance” meaningless—tails must be governed.

Therefore, the core of AI-native is not “making execution stronger,” but enabling you to stably answer three questions:

• Where are the upper bounds (budgets, steps, call counts, state occupancy)
• What to do when crossing boundaries (degradation, rollback, isolation, blocking)
• How results are rewritten (policy iteration and cost correction)

Engineering Response of AI-Native Infrastructure

Enterprises can refer to the following five “hard standards” during reviews—missing any one means inability to achieve closed-loop governance of uncertainty.

Admission: Ingress Admission Control

• Implement tiered admission for requests with ultra-long contexts, oversized tool graphs, or ultra-high budgets
• Bind “budget, priority, compliance” as part of intent (policy as intent)
• Clearly communicate rejection reasons and explain why requests are denied

Translation: Intent Translation to Governable Execution Plans

• Select runtime, routing/batching strategies, and caching strategies for requests
• “Cap” agent workflows: maximum steps, maximum tool calls, maximum tokens
• Include fallback paths: deterministic alternatives, cached answers, manual/rule-based fallbacks

Metering: End-to-End Metering and Attribution

• Meter tokens, GPU time, KV cache footprint, I/O, and network for each request/agent task
• Attribute by tenant, project, model, and tool to form cost and quality metrics
• Separately label “tail overhead” so long-tail costs no longer hide in averages

Enforcement: Budget and Degradation Mechanisms

• Budget triggers: rate limiting, degradation, preemption, queuing (by priority and tenant isolation)
• Risk triggers: isolation

Conclusion

The core of AI-native infrastructure governance lies in front-loading uncertainty, layered metering, policy feedback, and institutionalized constraints to form a closed loop of cost and risk. Only with engineering mechanisms such as Admission, Translation, Metering, and Enforcement can systems achieve economically viable, controllable, and recoverable operations under normalized uncertainty.

References

• Google SRE Book - Service Level Objectives - google
• FinOps Foundation - AI Cost Management - finops.org
• OpenAI Usage Policies - openai.com