Video Processing Reliability

Checkpointing

Save progress at intervals to minimize wasted work:

What to Checkpoint

• Frame number processed
• Intermediate outputs (every N frames)
• Model state (if stateful)
• Timestamp and duration processed

Checkpoint Strategy

Every 100 frames:
1. Write intermediate output
2. Update checkpoint record
3. Continue processing

On failure:
1. Read last checkpoint
2. Seek to checkpoint frame
3. Resume processing

Storage

• Checkpoints to object storage (S3)
• Checkpoint metadata to database
• Clean up checkpoints on job completion

Health Checks

Liveness

Is the worker alive?

GET /health/live
200 OK

Check: process running, not hung.

Readiness

Is the worker ready for jobs?

GET /health/ready
200 OK (GPU loaded, models ready)
503 Service Unavailable (still warming up)

Deep Health

Detailed health for debugging:

GET /health/deep
{
  "status": "healthy",
  "gpu_memory_used": "18GB / 24GB",
  "models_loaded": ["real-esrgan", "gfpgan"],
  "jobs_processed": 147,
  "last_job_at": "2026-07-11T14:30:00Z",
  "uptime_seconds": 86400
}

Graceful Degradation

Queue Overflow

When queue exceeds capacity:

1. Accept jobs with longer estimated time
2. Prioritize paid tiers
3. Eventually reject with 503 + Retry-After

GPU Shortage

When GPUs are constrained:

• Lower quality processing (faster)
• Smaller output resolution
• Skip optional stages (face restoration)

Storage Full

When storage approaches limits:

• Accelerate cleanup (shorter retention)
• Reject new jobs temporarily
• Alert operations team

Retry Strategies

Automatic Retry

Retry policy:
- Max attempts: 3
- Backoff: exponential (1m, 5m, 15m)
- Retry on: GPU OOM, timeout, transient storage error
- Don't retry on: invalid input, quota exceeded

Dead Letter Queue

After max retries:

1. Move to DLQ
2. Alert operations
3. Notify customer (optional)

Manual Intervention

DLQ inspection workflow:

1. Identify failure reason
2. Fix underlying issue
3. Replay job from DLQ
4. Or fail permanently with refund

Disaster Recovery

Multi-Region

Deploy in multiple regions:

• Primary: US-East
• Secondary: US-West
• Failover time: <5 minutes

Failover Triggers

• Region-wide outage
• GPU availability drops below threshold
• Error rate exceeds threshold

Data Replication

• Job metadata: replicated database
• Video files: customer-provided URLs (stateless)
• Outputs: replicated storage or regenerate

Recovery Procedures

1. Detect outage (automated or manual)
2. Update DNS to secondary region
3. Resume in-flight jobs from checkpoints
4. Notify customers of any impact