Managing virtual machines, network load balancers, and GPU drivers on cloud providers like AWS or GCP requires dedicated engineering overhead. Specialized AI platform-as-a-service (PaaS) and serverless providers like Hugging Face Spaces and Replicatestreamline deployment, offering automatic HTTPS, scaling to zero, and cold-start optimization.
Hugging Face Spaces allows you to host interactive ML applications (using frameworks like Gradio, Streamlit, or custom Docker) directly inside the Hugging Face ecosystem. Spaces operate via a Git-based workflow: you commit your code, aDockerfile, or configuration files, and Hugging Face automatically compiles, exposes, and hosts the application.
To configure a custom Docker application in a Hugging Face Space, you specify application metadata using a YAML header (front-matter) inside the root README.md file of your Space's repository:
--- title: Stable Diffusion Inference emoji: 🎨 colorFrom: indigo colorTo: purple sdk: docker app_port: 8000 pinned: false --- # custom-docker-app Exposing custom FastAPI backend on port 8000.
Replicate is a serverless platform designed specifically to run large deep learning models at scale. Replicate handles dynamic autoscaling, scaling down to 0 instances when idle (eliminating constant GPU costs), and rising instantly to meet incoming load.
To deploy to Replicate, you use Cog, their open-source tool that wraps ML models in standardized containers. Cog requires two main components:
# Example cog.yaml configuration
build:
gpu: true
cuda: "11.8"
python_version: "3.10"
python_packages:
- "torch==2.0.1"
- "transformers==4.30.2"
predict: "predict.py:Predictor"
The predict.py model script exposes a unified interface. The setup method loads weights once into GPU RAM, and predict processes concurrent execution streams:
from cog import BasePredictor, Input, Path
import torch
class Predictor(BasePredictor):
def setup(self):
"""Load model weights and move to GPU RAM"""
self.device = "cuda" if torch.cuda.is_available() else "cpu"
self.model = torch.load("weights.pt", map_location=self.device)
self.model.eval()
def predict(
self,
image: Path = Input(description="Input image file"),
scale: float = Input(description="Upscale multiplier", default=2.0)
) -> Path:
"""Run a single model inference process"""
# Inference computations occur here...
return output_path
| Platform Feature | Hugging Face Spaces | Replicate Serverless |
|---|---|---|
| Execution Type | Persistent instances (Always-On/Sleep schedules) | Serverless Functions (Cold Starts vs Scale-to-0) |
| Scale Dynamics | Manual vertical or horizontal scaling | Elastic auto-scaling from 0 to hundreds of GPUs |
| Packaging Format | Standard Docker files or custom SDKs (Gradio) | Cog Container YAML specifications |
| Primary Use Case | Interactive UI demos, portals, lightweight APIs | Production pipelines, large batch jobs, high scale APIs |
Scale-to-zero means no instance is running, and so nothing is billed, while nobody is calling your model. The price is the cold start: the first request after an idle period must wait for a machine to be allocated, the container image to be pulled, and your setup() to load the weights into memory. For a large model this can take from seconds to minutes, which is why the weights belong in setup() (run once per instance) and never insidepredict() (run once per request). Whether cold starts are acceptable depends on the product: a batch job that runs overnight does not care, while an interactive chat feature might. Three common mitigations are keeping a minimum number of warm instances (which costs money), shrinking the image and the model (for example with a smaller or quantized model), and showing the user a loading state instead of a spinner that looks frozen.
For a quick demo you often do not need Docker at all. With the Gradio SDK a Space is two files: app.py and a requirements.txt. In the README front-matter set sdk: gradio instead of sdk: docker.
# app.py
import gradio as gr
def shout(text: str) -> str:
return text.upper()
demo = gr.Interface(fn=shout, inputs="text", outputs="text")
if __name__ == "__main__":
demo.launch()Push these files to the Space's Git repository and it builds and serves automatically. With the Docker SDK, remember that the app_port in the front-matter must match the port your server actually listens on, and the server must bind to 0.0.0.0. API keys and tokens belong in the Space's Secrets settings, which are exposed to your app as environment variables, never in the repository, because Space repositories are often public.
Cog lets you test the container locally before publishing, which catches missing dependencies early. Input names map to the arguments of predict(), and @ tells Cog to read a file:
# build and run one prediction locally (requires Docker)
cog predict -i image=@photo.png -i scale=2.0
# after creating a model on Replicate and logging in
cog push r8.im/<your-username>/<your-model>Once published, any client can call it over HTTP. The official Python client reads the token from the REPLICATE_API_TOKEN environment variable:
import replicate
output = replicate.run(
"<your-username>/<your-model>",
input={"image": open("photo.png", "rb"), "scale": 2.0},
)
print(output)Check the platform's current documentation for the exact client behavior and return types, as these details have changed between versions.
Estimate before choosing: compare (requests per month × seconds of GPU per request × price per second) with the monthly price of an always-on instance. The answer flips as traffic grows. Prices change, so use current numbers from each provider rather than ones from a tutorial.
cog.yaml example, and make sure the CUDA version matches your PyTorch build.app_port, the Dockerfile and the server command, which shows up as a Space stuck on "Starting".Managed platforms trade control for speed. Spaces give you a Git-pushed, shareable demo with minimal setup; Replicate with Cog gives you a packaged, autoscaling model API that costs nothing while idle. In both cases the same engineering rules apply: load the model once, pin your dependencies, keep secrets out of the repo, plan for cold starts, and compare real traffic-based costs before committing to one approach.
Prepare the structural blueprints to support multi-platform deployments for a serverless diffusion model:
docker and port to 8000.cog.yaml file detailing GPU settings, CUDA version 11.8, and package dependencies: torch, cog.predict.py script by inheriting from BasePredictor.setup() routine to create a mock dictionary containing weights: { 'gamma': 0.82, 'beta': 0.15 }.predict() function parameterizing input arguments (e.g. integer seed and float prompt_weight) and returning the calculated inference result.