Designing Global Sovereign Data Platforms

For years our default assumption — usually unspoken — was that all data eventually lands in one place. One lake, one platform. That assumption is breaking now. And for data teams specifically, it flips something fundamental: your model can be global even when your raw data cannot be.

Hold that thought, because it is the hinge the whole architecture turns on. Once you accept that raw data may stay put, every downstream decision — where you store, where you compute, how you build features, where you train — changes.

This is mainstream, not hypothetical: Gaia-X and France’s SecNumCloud in Europe, Microsoft’s EU Data Boundary, AWS and Google sovereign-cloud offerings — all exist because customers now demand that data location be a contractual fact, not a footnote.

Where the one-platform model breaks down

Let me be fair to the centralized model first. It is not obsolete. A single platform is the simplest thing that works, and if you can centralize, you probably should. What is breaking is the assumption that you can always centralize.

Picture a global raw data lake on one side. Now push four things against it: local legal boundaries, sector-specific rules, procurement constraints, and sovereign-cloud requirements. Any one of those, and the neat picture starts to crack.

So, the redesign is not about throwing away centralization. It is about making the architecture answer three questions out loud: where data is processed, who is allowed to read it, and what is allowed to leave the country. If your current design cannot answer those three clearly, it is not ready for a regulated market.

The detail that kills naive designs: remote access counts as export. A data scientist in Stockholm running a notebook against raw China storage can be treated as a cross-border transfer even if nothing is downloaded. So ‘just give analysts read access’ is off the table.

China is just one example, but a clear one

China is the clearest example — not the only one, but the clearest. And again, I am not here to give legal advice. The engineering takeaway is what matters: China forces you to treat data classification and export boundaries as first-class artifacts in your platform — the same way you treat a schema or a pipeline — not as paperwork you bolt on at the end.

Look at what the rules actually target: cross-border transfers, the category they call “important data,” exposure of critical infrastructure, and raw operational datasets. Our response was consistent across all of them — keep the raw data in-country, do the heavy processing locally, export only approved aggregates, and record lineage and policy evidence as you go.

However, there is one nuance I’d like to emphasize, because it trips up data scientists: even machine data with no personal information can require classification. Once the scale is large enough, the precision is high enough, or the sector is sensitive enough, “no PII” does not mean “free to export.” We assumed machine data was safe to move. It was not automatically. Design for that.

The legal backbone is the Data Security Law (2021), the CAC’s 2024 cross-border provisions, and the outbound data-transfer security-assessment measures. ‘Important data’ is the catch-all — it is sector-defined and can include large-scale industrial or health telemetry, which is exactly our case.

Sweden and the EU push the same way

The natural reaction is, “fine, but that is China.” So, let me be careful and precise: Sweden is not China. There is no blanket data-localization law in Sweden.

What there is — and what is growing across Europe — is sovereignty pressure, and it is strongest in the public sector and regulated industries. Cloud decisions there increasingly weigh privacy, foreign access, procurement rules, and a real question of who actually operates the system. Across the EU more broadly, you have the Data Act, the Data Governance Act, GDPR transfer rules, NIS2, and CER — all shaping how data is accessed, switched, shared, and kept resilient.

The point is not that these are the same laws. They are not. The point is that the rules differ by jurisdiction, but the architectural demand is identical everywhere: make data location and access explicit in the design. China just makes it impossible to ignore; Europe lets you pretend a little longer.

Sweden’s IMY has flagged public-sector use of US hyperscalers over CLOUD Act / FISA foreign-access risk, and some municipalities moved off foreign email and cloud. NIS2 (2024) and CER raise resilience and supply-chain duties; the Data Act adds switching and access rules from 2025. Different levers, same direction.

The business problem

Let me translate this into business terms, because that is how you get buy-in. The fear in the room is always the same: if the data cannot move, do we lose our dashboards, our forecasting, our anomaly detection, our ML products? The answer has to be no.

The goal is to preserve global insight without moving raw data globally. In practice, that means four things: keep your global KPIs and ML signals whole across regions; keep raw data under local control; avoid duplicating your product logic in every region; and still give the business one coherent global view.

The line I use with stakeholders is simple — raw data does not have to be global for the insight to be global. Once leadership believes that is achievable, the architecture conversation gets a lot easier.

Put a number on it: if ~90% of global dashboards and models consume L2/L3 features rather than raw events, exporting features instead of raw cuts cross-border volume by orders of magnitude while keeping those products alive. In our case Qlik apps and forecasting ran on exported features alone.

The technical problem

For the engineers, let me make the constraint concrete, because “sovereignty” is abstract until it hits your code. The platform boundary is not a single cluster you can point to. It is the combination of five things: storage, compute, catalog, access, and governance.

And the things that have to stay portable across that boundary are exactly the things data teams touch every day — your code, your catalogs, your schemas, and your features. So, the target state is one logical platform spread across several physical boundaries, producing AI outputs that still line up at the end.

Here is the practical tell: if your notebooks assume one region, one metastore, one identity provider — that is the first thing that breaks when you cross a border. Portability is not a nice-to-have here; it is the whole game.

Portability in practice: Apache Spark 3.5.x to match Databricks’ Spark 3.5, Delta 3.1.x or Iceberg 1.5+, S3A paths instead of dbfs:/, and no dbutils or Unity-Catalog-only calls. The day a Databricks-only API lands in a shared job, the local plane can no longer run it.

You can’t bolt on compliance afterwards

I want to be clear that I am not anti-tool. Tools are necessary. They are just not sufficient. The mistake — and I have watched teams make it — is treating compliance as something you patch in later.

The fragile version looks like this: a catalog built after the fact, approvals happening over email, one-off masking jobs, and direct access to storage that nobody is really tracking. The durable version builds the controls into the shape of the platform: raw zones with hard limits, an explicit export gate, policy expressed as code, and lineage captured by default rather than reconstructed afterward.

The principle I would write on the wall: the control that actually works removes the unsafe path. It does not just discourage it.

The export gate is a real component, not a metaphor. It enforces a table and column allowlist, minimum-cohort thresholds, and high-cardinality-identifier checks, and it emits an export manifest — export_id, policy_version, source_table, watermark range, row_count, file list — plus an audit log and a rejection report for anything that fails.

Pattern 1: clean separation

So, let us look at the two architectures. Pattern one is clean separation. It is the easiest to explain and the easiest to defend to a regulator: two fully independent platforms, joined only by a controlled export path. The local side runs its own BI and AI; the global side runs its own; approved exports are the only bridge.

The strength is a beautifully simple compliance story and strong isolation. The cost is blunt, and you will feel it — you run the platform twice, you staff the team twice, and product velocity takes the hit.

So clean separation is the right call when separation has to be absolute, or the data is so sensitive that any coupling is a risk. For a lot of organizations, though, paying for everything twice is a non-starter — which is exactly what pushes you toward the second pattern.

Clean separation usually means duplicating Atlas, Ranger, Keycloak, Airflow and the whole Spark stack on both sides, plus two on-call rotations. When we costed it, the duplicated platform-engineering effort — not the compute — was the dominant cost.

Pattern 2: hybrid federation

Pattern two is hybrid federation, and this is the one we actually chose, so I will slow down here. The idea: raw data and all the high-granularity processing stay inside the sovereign boundary. Only approved features and aggregates pass through an export gate into the global product layer.

Walk it left to right — raw, local, clean; then features and aggregates; then the export gate; then the global product. Notice that the gate is the only thing that crosses the line, and it is the most important control in the whole design.

What you get is three things at once:

• local autonomy, because the sensitive processing never leaves;
• global continuity, because curated data joins your existing global products at the semantic layer; and
• AI that still works, because features and model outputs harmonize without ever pooling raw training data.

When do you use this? When most of your global products run on aggregates or features rather than raw events. In our experience, that is most of the time. Very few global dashboards or models actually need raw device telemetry — they need the features you derive from it.

In our build the gate exports Iceberg/Parquet incrementally by watermark; the global Databricks side ingests it as just another regional input and unions it at Layer 2, so downstream Qlik apps didn’t change. The main export-safe products are FCR (feature & condition reduction) outputs and FFT spectral features.

Decision matrix

How do you choose? This is the bridge from architecture theory into implementation, and the honest answer is there is no default winner.

Clean separation gives you the simplest compliance story. Hybrid gives you more product continuity and more AI reuse, but at higher operational complexity. Run your own situation down these rows — compliance narrative, global product continuity, AI reuse, operational complexity, business agility — and score them honestly.

The decision really comes down to three questions: how sensitive is the data, what do your products actually depend on, and how much do you reuse models and features across regions. If you reuse a lot, hybrid usually wins. If you cannot tolerate any coupling, separation wins.

A heuristic we actually used: if more than ~70% of global products depend on aggregates or features, hybrid pays off; if regulators or customers demand provable isolation — defense, some health data — clean separation wins despite the cost.

What crosses the boundary?

Now let us make the boundary precise, in language data scientists live in. The export boundary is about features and labels just as much as it is about tables.

Think in four layers. L0 is raw — archives, telemetry, high-granularity events. L1 is clean — normalized event logs, diagnostics. L2 is features — aggregates, condition features, the FFT and FCR outputs we compute. L3 is products — global KPIs, AI-ready tables, BI marts.

Here is the rule: L0 and L1 stay local. Raw payloads, precise local identifiers, full reprocessing history — none of that crosses. L2 and L3 can move — approved features, aggregated KPIs, model outputs. The sovereign boundary sits right between L1 and L2. That line is your data contract.

And the design implication for you is direct: when you engineer a feature, you should already know which side of that line it lives on, because that determines whether it can ever be part of a global model.

Make the contract enforceable in the catalog.

Atlas tags below the line:

cn_raw_only, cn_processed.

Above it:

export_candidate → export_allowed, aggregated, fcr_features, fft_features.

Ranger then denies SELECT on raw.* for analysts and only allows export.* where tag = export_allowed.

Rebuilding the local plane with open tools

So how do you rebuild the local side, given that Databricks cannot run there? The goal is not to clone the Databricks UI. The goal is to keep the things that actually matter: Spark, SQL, open table formats, and reproducible pipelines.

Here is the open stack we used.

Storage: an S3-compatible object store, with raw, silver, and export zones.

Tables: Iceberg or Delta, for portable table metadata.

Processing: Spark on Kubernetes, batch and streaming.

SQL: Trino, for interactive, BI-ready access.

Orchestration: Airflow, for schedules, retries, and dependencies.

Governance: Atlas plus Ranger, for lineage and authorization.

Every piece of that is open and portable, it runs inside the sovereign environment, and — this is the key — the data contracts stay compatible with the global Databricks side. A feature computed locally means the same thing globally. You are not rebuilding the brand; you are preserving the capabilities.

Concrete versions from the POC: MinIO, Spark 3.5, Hive Metastore 3.1.3, Trino 444, Iceberg, Atlas 2.3 (HBase + Solr + ZooKeeper backend), Airflow 2.9. The classic gotcha: Atlas shows Spark lineage but not Hive tables unless the Hive Metastore hook JARs are installed and the metastore is restarted.

Governance runs at runtime, not on paper

Governance has to run at runtime, not live in a policy document nobody opens. The mental model I hand teams is two letters apart: RBAC protects the platform — who is allowed to run jobs and operate things — and DAC protects the data — who is allowed to read which rows and columns.

In our stack: identity through Keycloak or LDAP, RBAC on Kubernetes and the UIs, DAC enforced by Ranger, lineage captured in Atlas.

And it maps cleanly onto people.

Data scientists query governed tables and reach features and training sets through SQL and approved notebooks.

Platform teams run the service accounts, the namespaces, the policies, and the export pipeline. Data owners sign off the tags — things like cn_raw_only, export_allowed, ai_trainable.

The group taxonomy we used: platform-admins, data-engineers, data-scientists, analysts (Trino-only), governance-admins, export-operators. Keycloak federates with LDAP, Ranger audits every SELECT, and analysts never touch object storage directly — they go through Trino.

Sovereign platforms must support ML/AI at scale

Now the turn toward the data science part. Watch how the demands escalate. BI usually gets by on aggregates — a dashboard rarely needs raw events. ML needs more — training data, labels, drift monitoring, and the ability to replay history. And generative AI needs even more — retrieval, embeddings, evaluation, observability.

So here is what sovereignty does to your job: feature engineering quietly becomes platform engineering. The hard part is no longer moving data from A to B. The hard part is keeping the signal statistically useful when the data has to stay where it is. That is a different skill — and it is the one that is becoming valuable.

Example contrast: a failure-prediction model needs labelled raw events and replay; a fleet-KPI dashboard needs only hourly aggregates. Same platform, very different data contracts — and only the dashboard can run purely on exported data.

How the ML lifecycle maps to the boundary

The trick is to stop treating the ML lifecycle as one environment and start splitting it deliberately across planes.

Walk the lifecycle: ingest, feature engineering, training or fine-tuning, evaluation, inference, monitoring. Now assign each step a home. The local plane does the things that need raw data — raw replay, labels, feature generation, local model validation. The shared contracts — and this is the connective tissue — carry feature schemas, metrics, evaluation protocols, and model cards across the boundary. The global plane does global reporting, a mirrored model registry, and aggregate monitoring.

The whole art is deciding which steps genuinely require raw data and must stay local, and which can be represented by a contract and safely shared. Get that split right and you can train and serve globally-relevant models without a single raw record leaving the country.

Concretely: raw replay and label generation run on the local Spark cluster; feature schemas and model cards are the shared contract; a registry holds mirrored models and metrics globally. Inference can be local — exporting only decisions and metrics — or global, running on exported features.

Three ML/AI patterns for sovereign data

That gives you three patterns — and I am deliberately careful not to oversell federated learning here. Think of them as a family, and pick per model.

Pattern one, feature export: generate features locally, export only approved feature vectors and aggregate labels. For most enterprises this is the practical starting point, and it covers a huge amount — forecasting, anomaly detection, fleet KPIs. Pattern two, local training: train regional models locally, and publish metrics, model cards, or parameters if you are allowed to. That is your answer for local behavior drift and for the most sensitive raw data. Pattern three, federated outputs: the global layer consumes predictions, embeddings, or evaluations — never raw training rows.

The mistake is picking one pattern for everything. A dashboard aggregate and a failure-prediction model do not need the same data contract, so do not force them into one.

Mapping to our fleet: fleet KPIs and anomaly detection → feature export; device-specific degradation models → local training with exported metrics; a global fleet-health model → federated outputs such as embeddings or predictions. We started with feature export and added the others only as needed.

What you have to keep maintaining

A word for the tech leads and managers, because this is where these platforms quietly fail. Operations is the product, not an afterthought.

Six things you maintain forever:

• schemas and feature definitions;
• policies and the approval workflow;
• pipelines, with their retries and service accounts;
• lineage from raw to features to exports;
• runtime versions across Spark, Iceberg, and Trino; and
• model quality — drift, evaluation windows, bias, monitoring.

And the failure mode that actually hurts is the slow one: governance drift. Schemas, policies, and model behavior gradually diverging between regions until the two sides no longer agree with each other. Nobody notices on day one. Everybody notices in month nine. Budget for that maintenance up front.

Make governance code: data contracts in version control, Ranger and Atlas policies as code, lineage re-validated after every migration, and Spark/Iceberg/Trino versions pinned and tested together. We treat the export pipeline as a product with its own SLAs, not an ad-hoc job.

Pros, cons, and trade-offs

What improves with this approach: you hold up to regulation, you get clear audit evidence, far less raw data is moving around, you get global products you can actually reuse, and your AI still scales — through features and outputs.

What gets harder, honestly: more moving parts to operate, runtime versions drifting apart, feature contracts that demand real discipline, some raw-level analysis that just stays regional and cannot be centralized, and more latency plus harder debugging across boundaries.

The one-line summary I would leave you with: a regulated platform costs you complexity; ignoring the rules can cost you the market. Pick your cost.

Honest limits worth naming: no Photon and no Unity Catalog outside Databricks; some streaming patterns need careful checkpointing on object storage; and exported aggregates can still need legal review if they reveal sensitive operational patterns. None are blockers, but budget for them.

Five takeaways for data scientists

So, five things to take away.

1. Think in data contracts — your features, labels, and evaluation sets are products, so treat them that way.
2. Design for locality — local raw data can still power global insight.
3. Track lineage early — a model with no provenance is a liability the moment you are in a regulated market.
4. Separate your access layers — RBAC guards compute, DAC guards data.
5. Plan for AI scale — sovereign BI is the easy part; sovereign ML is the next bottleneck, and it is coming for all of us.

If you do one thing after reading this article: pick a single model or dashboard, classify its inputs, draw the L1/L2 line, and stand up one export-gated feature flow end to end — archive → Spark → Iceberg → Trino → Atlas lineage. That one vertical slice teaches more than any architecture diagram.

• #BI & Big Data
• #Information Technology