Enterprise Architecture

Scaling Video APIs

Q: How many GPUs do I need?

Calculate: (jobs per day) / (jobs per GPU per day at target utilization). For 10K jobs/day at 5 min each and 70% utilization, you need ~50 GPUs.

Q: How do I handle burst traffic?

Over-size your queue (cheap), use auto-scaling for workers (slightly delayed), and maintain a minimum pool for baseline responsiveness.

Q: What metrics should I monitor?

Queue depth, queue latency, processing time percentiles, success rate, and GPU utilization. Alert on queue > 10 min, success < 99%.

Q: Serverless or self-managed scaling?

Serverless (Modal, Lambda) for variable workloads and simplicity. Self-managed for predictable high-volume or strict control requirements.

Scaling video processing is different from scaling web applications. Jobs take minutes, not milliseconds. GPUs are expensive. This guide covers how to scale from 100 to 1M jobs per month.

The Async-First Architecture

Video APIs must be asynchronous:

Synchronous Problems

HTTP timeouts (30-60 seconds typical)
Client connections held for minutes
Load balancer issues
No retry on failure

Async Pattern

1. POST /v1/jobs → Returns job_id (100ms)
2. Worker processes (5-30 minutes)
3. Webhook fires on completion
4. GET /v1/jobs/{id}/output → Signed URL

This decouples API response time from processing time.

Horizontal Scaling

API Layer

Scale horizontally behind a load balancer:

Stateless services
Session affinity not required
Scale based on request rate

Worker Layer

Scale based on queue depth:

Add workers when queue grows
Remove workers when idle
Minimum pool for cold start mitigation

Storage Layer

Object storage scales automatically:

S3, GCS, Azure Blob: unlimited capacity
Distribute across regions if needed
Use CDN for download distribution

Database Layer

Job metadata database:

Read replicas for status queries
Partition by time or customer
Consider managed services (RDS, Cloud SQL)

Capacity Planning

Work Backwards

Target: 10,000 jobs/day
Average duration: 2 minutes source video
Processing time: ~5 minutes per job
Jobs per GPU per hour: 12
Jobs per GPU per day: 288
GPUs needed at 100%: 10,000 / 288 = 35
GPUs at 70% utilization: 50

Burst Capacity

Plan for 3-5x burst:

Marketing campaigns
Enterprise onboarding
Seasonal patterns

Queue Sizing

Size queue for burst absorption:

Burst rate: 5x normal = 50,000 jobs/day
Processing rate: 35 GPUs = 10,000 jobs/day
Queue must hold: 40,000 jobs
At 1KB/job: 40 MB queue (trivial)

Queue is cheap; GPUs are expensive. Over-size the queue.

Auto-Scaling Strategies

Queue-Based Scaling

Scale workers based on queue metrics:

if queue_depth > workers * 2:
    add_workers(queue_depth / 2 - workers)
if queue_depth < workers / 2 and workers > min_workers:
    remove_workers(workers - queue_depth)

Predictive Scaling

Scale based on expected demand:

Historical patterns (weekday vs weekend)
Marketing calendar (campaigns)
Customer growth trajectory

Serverless Scaling

Let the platform handle scaling:

Modal: Automatic container scaling
AWS Lambda + Step Functions
GCP Cloud Run + Workflows

Cost-Aware Scaling

Balance speed and cost:

Use spot instances when queue is long
Fall back to on-demand for SLA
Right-size instance type to job type

Monitoring at Scale

Key Metrics

Queue depth: Jobs waiting
Queue latency: Time from enqueue to start
Processing time: P50, P95, P99
Success rate: Jobs completed / jobs submitted
GPU utilization: Actual vs. available compute

Alerting

Queue depth > 10 minutes worth of work
Processing time P95 > 2x normal
Success rate < 99%
GPU utilization < 50% (over-provisioned)

Dashboards

Real-time queue and worker status
Historical throughput trends
Per-customer usage breakdown
Cost per job over time

Frequently Asked Questions