Video Preprocessing for AI Agents

Why Preprocessing Matters

The quality of input directly affects agent performance:

• Garbage in, garbage out: Low-quality video produces low-quality understanding
• Token efficiency: Cleaner images compress better, using fewer tokens
• Accuracy: Face recognition fails on blurry faces; OCR fails on low-res text
• Consistency: Preprocessing normalizes quality across different sources

A small investment in preprocessing pays dividends in agent performance.

Enhancement Pipeline

The standard preprocessing flow:

Raw Video
    ↓
┌─────────────────────────┐
│  BetterVideo API        │
│  - AI upscaling         │
│  - Face restoration     │
│  - Contrast enhancement │
│  - Noise reduction      │
└─────────────────────────┘
    ↓
Enhanced Video
    ↓
Frame Extraction
    ↓
Agent Processing

Integration Code

import bettervideo

def preprocess_for_agent(video_url: str) -> str:
    """Enhance video before agent processing."""
    job = bettervideo.submit(
        video_url=video_url,
        resolution="1080p",  # Upscale for detail
    )
    result = job.wait()
    return result.download_url

# Use in your agent
enhanced = preprocess_for_agent(original_url)
frames = extract_frames(enhanced)
result = agent.analyze(frames)

What Enhancement Fixes

Input Problem	Enhancement	Agent Benefit
Low resolution (480p, 720p)	AI upscaling to 1080p/4K	More detail for extraction
Dark/underexposed	Adaptive contrast (CLAHE)	Visible detail in shadows
Blurry faces	GFPGAN face restoration	Face recognition works
Compression artifacts	AI denoising	Cleaner signal, fewer false positives
Soft focus	Sharpening	Readable text, clear edges

Frame Extraction Strategies

Don't process every frame — it's expensive and unnecessary.

Strategy 1: Uniform Sampling

Extract every Nth frame (e.g., 1 frame per second).

# 1 FPS sampling for a 60-second video = 60 frames
frames = extract_uniform(video, fps=1)

Good for: General understanding, when nothing specific matters.

Strategy 2: Keyframe Extraction

Extract only I-frames (keyframes) from the video codec.

# Keyframes only - typically much fewer frames
frames = extract_keyframes(video)

Good for: Quick overview, storage efficiency.

Strategy 3: Scene Detection

Extract frames when the scene changes.

# One frame per scene
scenes = detect_scenes(video)
frames = [get_frame(s.start) for s in scenes]

Good for: Narrative understanding, edited content.

Strategy 4: Motion-Based

Extract frames when significant motion occurs.

# Frames with motion above threshold
frames = extract_on_motion(video, threshold=0.3)

Good for: Surveillance, activity detection.

Strategy 5: Content-Aware (Agent-Directed)

Let the agent decide what frames it needs.

# Agent requests specific frames
while not agent.satisfied:
    frame = agent.request_frame(timestamp)
    agent.process(frame)

Good for: Complex reasoning, when the agent knows best.

When to Skip Preprocessing

Enhancement costs time and money. Skip it when:

• Source is already high quality: 4K, good lighting, modern camera
• Task is coarse-grained: "Is there a person in frame?" doesn't need 4K
• Latency is critical: Real-time applications may need to skip
• Volume is extreme: At very high volume, sample rather than enhance everything

Decision Logic

def should_enhance(video_metadata, task):
    # Always enhance for these tasks
    if task in ["face_recognition", "ocr", "license_plate"]:
        return True

    # Enhance low-quality source
    if video_metadata.resolution < 720:
        return True
    if video_metadata.bitrate < 2_000_000:  # 2 Mbps
        return True
    if video_metadata.brightness < 0.3:
        return True

    # Skip for high-quality source with coarse tasks
    return False