Sovereign AI: What Changes When The Cloud Is No Longer Enough

Karan Alang, principal software engineer at Versa Networks with 25 years of experience in AI, cloud and big data.

gettyFor most companies, cloud computing is a sensible way to start developing AI systems. On-demand resources and managed services make it easier to build data pipelines, try use cases and scale as adoption increases.

However, public clouds don’t always satisfy the legal, operational and security requirements of a sovereign environment.

Gartner defines sovereignty as an organization’s ability to control its data, technology and digital operations. It’s “about who operates and supports your IT systems.” This includes control over provider operations, personnel and access paths, not just where the data is located.​

This is becoming increasingly important in the AI/ML age, especially for regulated and security-sensitive organizations. Financial institutions face privacy, record-keeping and jurisdictional requirements; healthcare systems must protect sensitive workloads, and government agencies may require FedRAMP-authorized or fully air-gapped environments​.

AI makes this complex because data is no longer confined to traditional databases and applications. It can flow through training pipelines, vector databases and inference systems. With agentic AI, autonomous agents may also access and act on it. Organizations therefore need to understand not only where the data resides but where it’s processed, and what systems and agents access it.​​

Sovereignty can be achieved on the cloud as well as an air-gapped on-premise environment. A sovereign cloud environment provides a middle ground between using a cloud-based environment and an air-gapped sovereign environment. The comparison below shows how sovereign cloud and on-premise environments satisfy sovereignty.​

• Sovereign Cloud: High, depending on the provider and region. Data can be constrained to an approved country or jurisdiction, with regional storage or processing and controls over data movement.

• On-Premise: Very high. Data is stored within the organization’s air-gapped controlled environment.

• Sovereign Cloud: Partial to high. The customer has significant controls, but the cloud provider still operates parts of the underlying infrastructure. Support personnel, privileged access, telemetry and management-plane dependencies need be evaluated.

• On-Premise: Very high. The customer controls administrators, physical infrastructure, networking, patching, deployment, monitoring, keys and access.

• Sovereign Cloud: Partial. The customer generally remains dependent on the provider’s hardware, cloud platform, APIs, managed services and service roadmap. For example, GPU availability on the cloud might be restricted to specific regions.

• On-Premise: Potentially very high, but not absolute. The customer selects and operates its compute, storage, networking, Kubernetes/platform stack, models and inference environment. However, dependencies can remain on GPU vendors, commercial software, firmware and externally developed models.​

The sovereign cloud provides on-demand provisioning while satisfying the sovereignty requirement of many organizations. But for highly sensitive data and AI workloads, some organizations require stronger isolation and operational controls provided by air-gapped environments.

Air-gapped sovereign AI has traditionally been cost-prohibitive and difficult to manage. However, technological changes have made air-gapped AI more practical.

• AI compute is increasingly accessible outside hyperscale data centers. Enterprise servers with modern GPUs now support meaningful inference workloads inside a customer-controlled environment.

• Local LLM inference has matured. Open-weight models can run on customer hardware without organizations having to send sensitive data to an external model API. Additionally, quantization techniques help reduce model size to run efficiently on enterprise server GPUs.

• The open-source infrastructure stack is production-ready and portable. Technologies like Kubernetes, Kafka, Airflow and Spark run across cloud and on-premise environments, bringing cloud-native principles to sovereign environments.

I’ve spent the last few years building cloud-native data platforms for network and security intelligence. We started with the cloud, which was the right starting point. It let us move quickly, iterate and scale. However, for some customers, an air-gapped sovereign AI solution is the correct one.

Moving the platform into an air-gapped environment taught us that sovereign AI isn’t simply cloud architecture deployed on-premises. The operational considerations and requirements change significantly.​

There are several key points to be considered while formulating an enterprise AI strategy and in the design/implementation of an air-gapped sovereign AI solution.

• Capacity is now finite. Every microservice and Kubernetes workload must be right-sized upfront—CPU limits, memory requests, persistent and ephemeral storage, and GPU allocation. The elasticity of the cloud isn’t available in an air-gapped on-premise environment; hence, the size needs to account for the peak loads.

• The upfront infrastructure cost may be significant and needs to be calculated appropriately. The lag between the hardware procurement and delivery must be factored into capacity planning.

• Observability is a key part of the design. The pre-built metrics visualizations for managed services like Composer, Mongo Atlas and Dataproc needs to be recreated.

• Log aggregation needs to be built using technologies like Grafana and Loki, This is critical for observability and compliance.

• CI/CD must be redesigned for the air-gapped environment. Internal artifact registries need to be sized and set up to securely store and distribute docker images and other software artifacts.

• Support teams need to be equipped to handle infrastructure and product issues. The cost estimate needs to include 24/7 support.

• Model and hardware selection must match the workload. NetOps and SecOps environments combine high-volume structured telemetry, real-time processing and language-based investigation. Model size, quantization, inference throughput and GPU memory all affect the architecture—and in an air-gapped deployment, getting those choices correct is critical.

As AI becomes embedded in organizations, sovereignty needs to be part of the architecture design considerations from the beginning. For many organizations, a sovereign cloud environmemt provides the right balance of control and flexibility. For highly sensitive or mission-critical workloads, an air-gapped infrastructure may be more appropriate.

The technological advancements in AI models and Infrastructure now makes these deployments increasingly practical, but greater control also means greater responsibility. Capacity, observability, security, software distribution and operational support need to be incorporated into the platform design considerations.

That’s the key lesson of sovereign AI: Organizations should design for sovereignty from day one rather than trying to retrofit it later.​

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?