Build · AI product · Generative video

Shipping generative video from model experiment to reliable product

TL;DR

I turned toy-character motion ideas into 5+ production workflows by sequencing image-prep and video models, defining anatomy-aware evaluations, and adding fallback and monitoring. The workflows reached 10K+ users per day with ~60% generation-to-publish conversion.

Skills & stack

  1. 01Product strategy
  2. 02Model evaluation & selection
  3. 03Prompt engineering
  4. 04Latency, cost, quality tradeoff
  5. 05Reliability & fallback design

Impact

Shipped 5+ reusable workflows across motion, storytelling, and campaigns. One motion-control workflow reached ~70% publish conversion versus a 56% average, and the product served 10K+ users per day.

5+
production workflows shipped
10K+
users reached per day
~60%
generation-to-publish conversion
+14 pp
Knee Slide vs average

Motion control

  • Hover to play

    Knee Slide ~70% publish conversion vs 56% average

  • Hover to play

    Hadouken

Campaign

Hover to play
FIFA World Cups Reusable jersey-personalization infrastructure
Hover to play
May the 4th

Background

The problem

Toy images vary in quality and anatomy, while video models introduce fidelity, motion, moderation, and reliability failures. I needed to turn a promising demo into a repeatable production workflow.

What I did

Prioritized ideas from community and cultural signals, then sequenced image-prep and video models around controllable inputs. Defined anatomy-specific fidelity gates and shipped with cross-family fallbacks, monitoring, and a release checklist.

How I built it

From demand signal to monitored production workflow

Toy-character video looked like a model problem. In production, it was a product-system problem: messy uploads, unfamiliar anatomy, motion fidelity, moderation limits, latency, and fallback behavior.

I owned the workflow from idea selection through model orchestration, evaluation, deployment, and monitoring. The result was 5+ reusable workflows reaching 10K+ users per day at ~60% generation-to-publish conversion.

Route Demand signal -> Product hypothesis -> Input architecture -> Model sequence -> Evaluation -> Release -> Monitoring

01 / Research

Research here did not mean a survey. It meant deciding what was worth building at all — a clip climbing a subreddit, a scene people quote from a movie, a joke only insiders get, sometimes a hallucinated video that came straight out of prompting.

I used community traction and cultural relevance to prioritize ideas before spending time on model testing.

Research funnel: four inspiration sources narrow through human judgment Reddit finds, scenes from movies, insider jokes and model hallucinations converge on a judgment gate, with an arrow leading onward to the next step. MANY CANDIDATES Reddit finds Scenes from movies Insider jokes Hallucinations HUMAN JUDGMENT

The funnel

Many candidates, one filter. Reddit finds, scenes from movies, insider jokes and hallucinations all pass through human judgment before anything gets built.

“Kaiju Fight” — reference → shipped

Hover to play
Hover to play
Found on Reddit
The reference
Shipped
Kaiju Fight: user's own toy is substituted in.

02 / Prototype

Building the model system

The first question was not simply, “Which video model is best?”
It was, “What sequence of models gives the video model a solvable input?”

Generative media platforms

Three vendors rather than one: the same model is priced differently across them, and some make a workflow faster to assemble than others.

  • Fal.ai fal.ai
  • Replicate replicate.com
  • Runware runware.ai

Workflow · Kaiju Fight

Reference → video Delivery
User photouploaded figure
Seedance 2.0 MiniReference-to-video · 3:4 · 720p · 5s
Fetch to CDNStable URL, not the vendor's
Notify backendCallback with the result
Final videodelivered
on failure
rejoins
Fallback route
FLUX 2 Klein 9B EditNormalize the figure to an upright solo portrait
Kling O3 ProVideo-to-video against a pinned reference clip
One model carries the primary route. When it fails, the fallback switches model family rather than model tier — normalizing the figure first, then re-running the shot on Kling — and rejoins before delivery. Failures past that point are reported back to the product.
Model and fallback decisionsGrok, Flux Klein, and when to switch model family instead of model tier4 decisions
  • Grok image edit was versatile, fast, and generally reliable, but could reject some protected-character inputs.
  • Flux 9B Klein provided a safer alternate image-preparation route.
  • A standard-to-pro fallback helped when the issue was performance.
  • A different model family (for example, Kling instead of Seedance) was more useful when the failure came from moderation or IP restrictions.

03 / Implement & iterate

Diagnosing the real failure

Observe the failure -> Identify the constraint -> Change input, prompt, route, or infrastructure -> Test again

The long-leg and takeoff cases initially looked like general hallucinations. Looking across repeated failures revealed a consistent spatial problem: head-position mapping. The problem was the model’s anchor, not the visual style.

The user's plush toy: a round white body with a blue striped cap, black eyes and yellow side tufts, with no legs.

The input

A round plush with no legs — the anatomy that head-position mapping cannot survive.

Head-anchored · failure ✕

heads aligned reference plush (stretched)

Head mapped to the fighter's head, so the plush's legs stretch to reach the ground.

Ground-anchored · fix ✓

40–60% reference plush (grounded)

Feet anchored to the ground, body scaled to 40–60% of the fighter.

The reference-model head-anchoring fixThe prompt that moved the character anchor from the head to the groundPrompt

I moved the character anchor to the ground, specified the intended scale, and placed those constraints inside the prompt section responsible for character positioning.

Anchor the character to the ground using its feet, bottom edge, or lowest point. Scale it proportionally from the ground up. Do not align its head with the reference fighter’s head. Element 1 and Element 2 are small toy-sized characters; their total height should be approximately 40–60% of the reference video fighters’ height. Scale down proportionally and keep them grounded.

Changing the spatial anchor fixed the underlying constraint; more surface-level prompt tuning would only have masked it.

The complete evaluation framework

Fidelity gates for humanoid, semi-humanoid, and limbless toy anatomies.

5 × 3 matrix
Metric A Humanoid B Semi-humanoid C Limbless
Character fidelity High High — shape + color + markings, not face proportions High — silhouette + color is the whole identity
Skeleton / rig sanity High Gating — hallucinated limbs = instant fail (most fragile metric for type B) Replace with: no phantom limbs added
Motion quality High Mid — creative locomotion is ok if coherent Mid — sliding or floating should feel intentional
Prompt adherence High Mid — credit partial adherence if limbs cannot physically do the move Low — score intent, not literal execution
BG / style consistency Mid Mid Mid — scene-level metric, same weight across all

04 / Deploy

The release checklistFrom hosted reference assets to production configuration8 steps
  1. Assets

    1. Host the reference assets.
    2. Prepare the demo video and a correctly cropped cover.
  2. Pricing

    1. Convert API cost into the product’s pricing unit.
  3. Engine

    1. Configure the engine, provider, target, and preprocessing policy.
    2. Bind the user image to the workflow input.
    3. Connect the engine to the template.
  4. Release

    1. Test the complete experience in beta.
    2. Configure the production version.

Connecting model quality to product behaviour

I used the product dashboard to connect model testing with workflow behaviour — generation success, fallback behaviour, and whether users moved from generating to publishing. A model could look strong in selected demonstrations and still expose a different pattern at real usage levels.

Redacted production health dashboard with volume, failure, success, conversion, recovery, user, and alert metrics
Production health view, redacted for confidentiality. See how I defined and built the analytics layer →

Reflection

The decisions happen around the model, not inside it

What I learned

  • The strongest AI product decisions often happen around the model rather than inside it.
  • A cleaner input can matter more than another round of prompt tuning.
  • A fallback from another model family can be more useful than a more expensive version of the same model.
  • An evaluation system designed around real toy anatomy reveals failures that a generic quality score would miss.

What failed

  • The workflow depended fully on vendor availability. When a vendor hit very high latency or CDN errors, we needed to switch vendors flexibly — a lesson that led to switching some of our workflows onto AWS Step Functions.
  • Surface-level prompt tuning delayed the real fix on head-anchoring by several rounds.