Run AI workloads on any cloud, store on Hugging Face: zero-egress storage with SkyPilot

Why this matters now

The way teams train AI has shifted: they no longer stick to a single cloud provider’s GPUs but hunt for available compute across multiple clouds and even on-prem clusters. Yet their data often remains trapped in one region’s object storage. When you finally snag scarce GPUs on AWS but your dataset sits in a GCP bucket, reading it costs nearly $0.09 per GB in egress fees. This “cross-cloud data tax” is a hidden cost many fail to account for.

Meanwhile, Hugging Face has evolved from a model-sharing community into a full-fledged AI storage platform and announced zero egress charges. SkyPilot is an open-source universal compute scheduler that launches jobs on 20+ clouds, Kubernetes, and Slurm. Their partnership bridges the gap between compute and storage, seamlessly.

How it works

In short, SkyPilot now treats Hugging Face Storage (Buckets, model repos, dataset repos) as a first-class storage backend. In a SkyPilot task YAML, you mount a hf:// path with just a few lines:

• hf://buckets/<org>/<name>: read-write Buckets for checkpoints and logs;
• hf://<author>/<model> or hf://datasets/<org>/<dataset>: read-only mounts for models and datasets.

Once SkyPilot finds GPUs on any cloud, the task container mounts this remote data via FUSE without pre-downloading. Because Hugging Face charges no egress, reading data onto those GPUs is free, regardless of cloud.

An underrated technical highlight: Hugging Face Buckets are backed by Xet, which offers content deduplication. This means incremental checkpoint saving transfers only the chunks that changed — a big saving for multi-GB checkpoints.

The bigger trend

This integration signals three shifts:

1. Extreme decoupling of compute and storage. Cloud providers used data gravity as a lock-in tactic; now data can be fluid. You’re no longer tied to the cloud where your data lives just to avoid egress fees.

2. AI platforms are becoming neutral infrastructure layers. Hugging Face is stepping beyond code hosting to become a storage layer that rivals cloud vendors — and zero egress makes it even more compelling. It could become the “data embassy” for AI.

3. Multi-cloud goes from a big-company luxury to a norm for small teams. Anyone can now arbitrage across clouds, using the cheapest spot instances to train large models, practicing true “compute nomadism.”

Practical value for developers and teams

If you run training or inference pipelines, you can adopt this paradigm today: store all models, datasets, and experiment checkpoints on Hugging Face. Write a single SkyPilot task; it will find the best GPU spot across your reserved instances, on-demand, or spot markets. No per-cloud data syncing scripts, no cross-region surprise bills.

To get started, all you need is a Hugging Face token. In your SkyPilot YAML, set file_mounts with store: hf, and your training script reads and writes to those mount paths as if they’re local. Just remember that Buckets support read-write; model and dataset repos are read-only. And while Hugging Face’s zero-egress policy is long-term, it’s wise to keep an eye on service terms.

A perspective most people miss

Many assume cross-cloud training inevitably involves costly data transfers or complex caching. But this partnership shows a neutral storage platform can act as a “data demilitarized zone.” Just as a standardized power socket lets you plug in appliances anywhere, your AI project can plug into any compute socket while your data stays in one reliable, free-to-access home. This quietly shifts the infrastructure priority from “where is my data?” to “where is the best compute deal?”