Video Generation & Image-to-Video: Sora 2, Veo 3.1, Runway, Open-Source

Native Synchronized Audio

First major model with integrated dialog, foley, ambient sound, and music in a single pass. Lip-sync to generated speech, footsteps that match terrain, and Doppler-correct passing vehicles. No separate TTS or sound-design step.

Up to 1080p, 20s

Sora 2 standard outputs 720p/1080p at up to 20 seconds. Sora 2 Pro pushes higher fidelity, sharper detail and 30-60s segments suitable for short-form ads, film previs, and stylized music video sequences.

~$0.10–$0.50 / second

API pricing: Sora 2 ~$0.10/s at 720p, ~$0.15/s at 1080p. Sora 2 Pro ~$0.30–$0.50/s at 1080p. ChatGPT Pro includes a monthly generation quota; pay-per-use beyond that. Caching does not apply to video generation.

Improved World Simulation

Better handling of rigid bodies, gravity, collisions, and fluid dynamics. Multi-shot continuity (cuts and camera changes) is more reliable. Still struggles with very fine-grained hand-object interaction and consistent text rendering.

OpenAI Videos API

REST endpoint POST /v1/videos accepts prompt, reference image (input_image), duration, resolution, aspect ratio. Returns a job ID; poll /v1/videos/{id} until status=completed, then download the MP4 + audio. Async only.

C2PA Watermarking

All outputs carry C2PA cryptographic provenance metadata identifying them as AI-generated. Visible watermark on free tier; metadata-only on paid tiers. Likeness protection blocks named public figures unless explicitly licensed.

Native Audio Generation

Synchronized dialog, foley, music, and ambience generated jointly with the visuals. Dialog can be specified verbatim in the prompt with quoted speaker tags. Multi-language speech supported across 30+ languages.

Multi-Reference "Ingredients"

Pass up to 3 reference images (character, product, environment) and Veo blends them into a single coherent shot. Critical for brand-consistent advertising where the actor, product, and location must all match supplied references.

Camera Controls

Explicit camera directives in the prompt: pan-left, dolly-in, orbit-right, crane-up, handheld. Interprets cinematic language ("Dutch angle", "rack focus") more reliably than competitors. 16:9, 9:16, and 1:1 aspect ratios.

Frames-to-Video Interpolation

Provide a start frame and an end frame; Veo generates the motion between them. Enables storyboard-driven production: artists draw key beats, Veo fills the in-betweens. Reduces the unpredictability of pure text-to-video.

Vertex AI API

Long-running operation pattern: POST to publishers/google/models/veo-3.1:predictLongRunning, poll the operation, fetch GCS-hosted MP4. IAM-controlled, VPC-SC compatible, and auditable — suitable for regulated enterprises.

SynthID Watermarking

Invisible watermark embedded in every video frame and audio sample. Survives compression, cropping, and re-encoding. Detectable via Google's SynthID Detector. Mandatory on all Veo outputs across consumer and API surfaces.

Runway Gen-4 + Gen-4 Turbo

Best-in-class for character and world consistency across shots. "References" feature locks an actor or location to a single image set across an entire production. 1080p, up to 10s. ~$0.05/credit, 5s clip ~$0.25-0.50. Native API + SDK.

Kling 2.5 (Kuaishou)

Strongest cinematic motion and physics among Chinese-origin models. 1080p up to 10s, 4K upscale available. Excellent dance/sports/action footage. Pricing ~$0.10/s standard, ~$0.20/s Master mode. API via Kling Cloud and fal.ai.

Hailuo 02 (MiniMax)

Aggressive pricing: ~$0.03–$0.05/s at 720p, ~$0.08/s at 1080p. Strong prompt adherence and stable motion for the price. 6–10s clips. Ideal for high-volume b-roll generation, A/B testing, and stock-style content.

Pika 2.0 + Pikaffects

Stylized output with signature "Pikaffects" (squish, crush, melt, inflate, explode). Strong for social-first, meme-style, and music-video creative. Up to 5s, 1080p. Subscription tiers from $10/mo with credit-based usage.

Aggregators: fal.ai, Replicate

fal.ai and Replicate expose Runway, Kling, Hailuo, Pika, plus open-source models behind a single API and key. Useful for multi-model A/B routing without separate billing. Slight markup vs. direct provider pricing.

When to Pick Which

Sora 2 / Veo 3.1: best overall + audio. Runway Gen-4: character continuity across shots. Kling 2.5: action and cinematic motion. Hailuo 02: cheap bulk generation. Pika 2.0: social/stylized creative with effects.

Wan 2.5 (Alibaba)

14B-parameter DiT (Diffusion Transformer). 1080p up to 10s. Wan 2.5 adds native audio generation, narrowing the gap with Sora 2 and Veo 3.1. Apache 2.0. Runs on 24GB VRAM with offloading; 48GB recommended for 1080p. ComfyUI + Diffusers support.

HunyuanVideo (Tencent)

13B params, dual text encoders (LLM + CLIP). 720p, 5–15s clips. Best-in-class open-source quality on VBench. Strong I2V variant. Apache 2.0. Needs 60GB VRAM at full precision; FP8 + offloading runs on 24GB at 720p with longer gen times.

LTX-Video (Lightricks)

2B-param DiT optimized for speed. Real-time 5s @ 768x512 on a single RTX 4090 (~5s gen for 5s output). Open RAIL-M license. Best choice for interactive UIs, batch pipelines, and rapid prototyping. T2V + I2V + V2V.

Mochi 1 (Genmo)

10B-param AsymmDiT. Strong motion fidelity and prompt adherence. 480p base, with HD upsampler. Apache 2.0. ~24GB VRAM with sequential offloading. Active community fine-tunes for film/anime/photorealism on Hugging Face.

CogVideoX-5B (Zhipu)

5B params, 720x480, 6s clips. Most mature ecosystem for LoRA fine-tuning — trainable on 1x A100 40GB. Apache 2.0. CogVideoX-5B-I2V variant for image-to-video. Excellent baseline for character/style customization.

ComfyUI & Diffusers

All five models ship with ComfyUI custom nodes (visual graph editor, fastest iteration) and Hugging Face Diffusers pipelines (Pythonic API, batch workflows). xformers, SageAttention, and TeaCache accelerate inference 2–3x.

Motion Brush

Paint a mask on the source image and assign a direction vector + intensity. The brushed region animates along that vector while the rest stays static. Pioneered by Runway Gen-2/3, now in Kling, Pika, and ComfyUI nodes for HunyuanVideo I2V.

Character Consistency

Runway "References", Kling "Custom Character", Veo "Ingredients" all let you upload 1-5 images of a face/actor/product and lock identity across an entire production. For open-source, train a CogVideoX or HunyuanVideo LoRA (~30 images, 1-2 hours on A100).

Start + End Keyframes

Provide a first frame and a last frame; the model interpolates motion between them. Veo "Frames-to-Video", Kling "Start & End Frame", LTX-Video "Image Conditioning". Eliminates motion ambiguity — the model only invents the in-betweens.

Camera Path Control

Specify camera motion (pan, dolly, zoom, orbit) independently of subject motion. Runway "Camera Controls" and Kling "Cinematic Lens Movement" expose explicit sliders. ComfyUI exposes per-frame camera trajectory tensors for fine control.

Video Extension & Looping

Feed the last frame of one clip as the start frame of the next to chain shots into 30-60s sequences. "Loop" mode forces the last frame to match the first — ideal for reels, ads, and animated backgrounds.

Upscaling & Frame Interpolation

Generate at 720p/24fps, then upscale with Topaz Video AI / SeedVR2 to 4K, and interpolate to 60fps with RIFE / Practical-RIFE. Common pipeline: cheap base generation → high-quality post-processing for delivery.

RTX 5090 (32GB) — Sweet Spot

32GB GDDR7 + Blackwell tensor cores. Runs HunyuanVideo and Wan 2.5 at 720p natively without offloading. Wan 2.5 5s @ 720p in ~90s. LTX-Video real-time. ~$2,000 retail when in stock. Best single-card option for serious creators in 2026.

RTX 4090 (24GB) — Mainstream

24GB GDDR6X. Mochi 1 and CogVideoX-5B run natively. HunyuanVideo and Wan 2.5 require FP8 + offloading. Wan 2.5 5s @ 720p in ~3-5 min with offloading. LTX-Video 5s in ~5-10s. Excellent value at ~$1,600 used.

RTX 3090 (24GB) — Budget

Same 24GB VRAM as 4090 at half the price (~$700 used). 30-50% slower than 4090. Still capable of running every open-source model with offloading. The most cost-effective entry point for VRAM-bound workflows.

RTX 6000 Ada / Pro 6000 (48GB-96GB)

48GB (6000 Ada) or 96GB (Pro 6000 Blackwell) VRAM eliminates all offloading. HunyuanVideo and Wan 2.5 at 1080p without compromise. 4-7x more expensive than 4090/5090 but the only single-card path to high-end production at 1080p+.

Cloud Alternative: Runpod / Vast.ai

A100 80GB ~$1.20-1.80/hr, H100 80GB ~$2-3/hr. Spin up a ComfyUI instance on demand, generate, shut down. Cheaper than a 5090 for sub-50 hours/month of usage. Pre-built ComfyUI templates available.

VRAM Optimization Stack

FP8 quantization (Q8_0 GGUF for video), SageAttention, TeaCache, VAE tiling, sequential CPU offload, torch.compile. Combined: HunyuanVideo on 12GB VRAM (RTX 4070), Wan 2.5 on 16GB. 2-3x slower but accessible.

Audio Capability Matters

Only Sora 2, Veo 3.1, and Wan 2.5 generate native audio. Everyone else needs a separate TTS + foley + music pass. Native audio cuts post-production from hours to minutes for short-form content.

Top-Tier Quality (VBench)

As of early 2026: Sora 2 Pro and Veo 3.1 lead, followed closely by Kling 2.5 and Runway Gen-4. Wan 2.5 and HunyuanVideo are the best open-source models, within 5-10% of Kling on most VBench metrics.

High-Volume Pipelines

Generating 1000+ clips/day? Hailuo 02 ($0.03-0.08/s) or self-hosted LTX-Video on 4-8 GPUs. Closed APIs hit rate limits and budgets fast. Open-source wins on $/clip at scale.

Enterprise Compliance

Vertex AI Veo 3.1 is the only option with VPC-SC, IAM, audit logs, and data residency guarantees out of the box. Open-source on private VPC closes the rest. Sora 2 has enterprise tier with stricter data handling.

Watermarking & Provenance

Sora (C2PA), Veo (SynthID) embed cryptographic provenance. Most open-source models don't — you must add C2PA tags downstream if your platform requires "AI-generated" disclosure (EU AI Act, YouTube, Meta).

Quick Decision Tree

Need audio + best quality? Sora 2 / Veo 3.1. Need character consistency? Runway Gen-4. Cinematic action? Kling 2.5. Bulk b-roll cheap? Hailuo 02. Privacy + customization? Self-host Wan 2.5 or HunyuanVideo on 5090.

Runway Gen-4 Image-to-Video (Python)

Submit an image + prompt, poll for completion, download the MP4. Async-friendly with SSE events. Install: pip install runwayml.

from runwayml import RunwayML
import time, requests, base64

client = RunwayML()  # uses RUNWAYML_API_SECRET

# Encode source image as data URI
with open("hero.jpg", "rb") as f:
    img_b64 = base64.b64encode(f.read()).decode()
prompt_image = f"data:image/jpeg;base64,{img_b64}"

# Submit Gen-4 image-to-video job
task = client.image_to_video.create(
    model="gen4_turbo",
    prompt_image=prompt_image,
    prompt_text="Slow camera dolly-in, warm sunset light, "
                "subject smiles, hair moves in the wind",
    duration=5,             # 5 or 10 seconds
    ratio="1280:720",
)

# Poll until complete
while True:
    task = client.tasks.retrieve(task.id)
    if task.status in ("SUCCEEDED", "FAILED"):
        break
    time.sleep(5)

if task.status == "SUCCEEDED":
    url = task.output[0]
    open("out.mp4", "wb").write(requests.get(url).content)
    print("Saved out.mp4")

OpenAI Sora 2 Generate (Python)

OpenAI Videos API. Submit prompt, poll job, fetch video. Native audio is on by default in Sora 2.

from openai import OpenAI
import time, requests

client = OpenAI()

job = client.videos.create(
    model="sora-2",
    prompt="A golden retriever surfs a small wave at sunset, "
           "GoPro POV, water splashes, dog barks happily",
    size="1280x720",
    duration_seconds=8,
    # input_image="data:image/png;base64,..."  # optional I2V
)

while True:
    job = client.videos.retrieve(job.id)
    if job.status in ("completed", "failed"):
        break
    time.sleep(5)

if job.status == "completed":
    url = job.output[0].url
    open("sora.mp4", "wb").write(requests.get(url).content)

Google Veo 3.1 (Vertex AI)

Long-running operation pattern. Install: pip install google-genai. Auth via gcloud auth application-default login.

from google import genai
from google.genai import types
import time

client = genai.Client(
    vertexai=True, project="my-proj", location="us-central1"
)

op = client.models.generate_videos(
    model="veo-3.1-generate-preview",
    prompt='A barista pulls an espresso shot, steam rises, '
           'narrator says "Single origin, slow extracted."',
    config=types.GenerateVideosConfig(
        aspect_ratio="16:9",
        duration_seconds=8,
        number_of_videos=1,
        generate_audio=True,        # native audio
    ),
)

while not op.done:
    time.sleep(10)
    op = client.operations.get(op)

video = op.response.generated_videos[0].video
client.files.download(file=video)
video.save("veo.mp4")

ComfyUI: Wan 2.5 / HunyuanVideo

ComfyUI is the standard graph-based UI for open-source video. Install custom nodes for the model, drop the workflow JSON, hit Queue.

# Install ComfyUI on a Linux box with RTX 4090/5090
git clone https://github.com/comfyanonymous/ComfyUI
cd ComfyUI && pip install -r requirements.txt

# Install model-specific nodes via ComfyUI Manager
# - ComfyUI-WanVideoWrapper      (Wan 2.5)
# - ComfyUI-HunyuanVideoWrapper  (HunyuanVideo)
# - ComfyUI-LTXVideo             (LTX-Video)
# - ComfyUI-MochiWrapper         (Mochi 1)
# - ComfyUI-CogVideoXWrapper     (CogVideoX-5B)

# Download weights to models/diffusion_models/
huggingface-cli download Wan-AI/Wan2.5-T2V-14B \
  --local-dir models/diffusion_models/Wan2.5

# Launch
python main.py --listen 0.0.0.0 --port 8188

# Drag a workflow .json onto the UI:
# https://github.com/kijai/ComfyUI-WanVideoWrapper/tree/main/example_workflows

Diffusers: HunyuanVideo (Python)

Pure-Python pipeline for batch jobs and CI/CD. Memory optimizations let HunyuanVideo run on a single 24GB card.

import torch
from diffusers import HunyuanVideoPipeline
from diffusers.utils import export_to_video

pipe = HunyuanVideoPipeline.from_pretrained(
    "tencent/HunyuanVideo",
    torch_dtype=torch.bfloat16,
)
pipe.vae.enable_tiling()
pipe.enable_sequential_cpu_offload()  # fits on 24GB VRAM

video = pipe(
    prompt="A cat dressed as a wizard casts a sparkling spell, "
           "soft volumetric lighting, cinematic 35mm",
    height=720, width=1280,
    num_frames=85,        # ~3.5s at 24fps
    num_inference_steps=30,
    guidance_scale=6.0,
).frames[0]

export_to_video(video, "wizard_cat.mp4", fps=24)

ComfyUI Headless API (Python)

Drive ComfyUI from production code via its websocket+HTTP API. Submit a workflow JSON, get the output filename, fetch the MP4.

import json, requests, websocket, uuid

CUI = "http://localhost:8188"
client_id = str(uuid.uuid4())

# Load a workflow exported via "Save (API Format)" in ComfyUI
workflow = json.load(open("ltx_video_i2v.json"))

# Override the prompt and source image
workflow["6"]["inputs"]["text"] = "Subject smiles, sunset light"
workflow["10"]["inputs"]["image"] = "hero.jpg"

# Queue the prompt
r = requests.post(f"{CUI}/prompt",
    json={"prompt": workflow, "client_id": client_id})
prompt_id = r.json()["prompt_id"]

# Listen for completion via websocket
ws = websocket.create_connection(
    f"ws://localhost:8188/ws?clientId={client_id}")
while True:
    msg = json.loads(ws.recv())
    if msg.get("type") == "executed" and \
       msg["data"].get("prompt_id") == prompt_id:
        break

# Fetch the resulting video from /history
hist = requests.get(f"{CUI}/history/{prompt_id}").json()
video = hist[prompt_id]["outputs"]["12"]["gifs"][0]
url = f"{CUI}/view?filename={video['filename']}" \
      f"&subfolder={video['subfolder']}&type=output"
open("out.mp4", "wb").write(requests.get(url).content)