PRODUCT ANALYTICS · AI RELIABILITY
Rebuilding AI video metrics around user outcomes
TL;DR
I found that endpoint error counts were misclassifying fallback-rescued generations as failures. I redefined success at the generation level and built four role-specific views across 40+ templates, now used daily by Content Ops and weekly by product leadership.
SKILLS & STACK
- 01Product analytics
- 02Metric definition & governance
- 03Data modeling
- 04Dashboard & decision systems
- 05Reliability monitoring
BACKGROUND
THE PROBLEM
All 40+ video templates appeared as one aggregated line. Worse, generations rescued by fallback were still counted as failures, so teams could not tell what to promote, fix, or retire.
WHAT I DID
Rebuilt measurement around one row per generation, defined failure by the user’s final outcome, added seven outcome modes, and tied each failure to its template, model, vendor, and step.
IMPACT
Outcome-based analytics that Content Ops monitors daily and product reviews weekly.
HOW I BUILT IT
The decisions behind the layer
The user outcome had to be defined before the grain, attribution, or migration constraints could be made legible.
ROUTE
Measure the request, not the deployment path
A · MEASURE FAILURE AT THE LEVEL OF THE USER OUTCOME
Roughly half the templates route to a second engine when the first fails. Fan’s Golden Goal made the consequence clear: endpoint counting reported 39.1% failure; the user-level outcome was 7.4%.
Seven outcome modesA decision tree replaces one ambiguous boolean with outcomes that have different owners and next actions. 7 MODES
in_progress Primary succeedednormal No fallback · by designprimary_failed_
no_fallback Fallback skipped · routing bugprimary_failed_fallback_skipped Fallback rescuedfallback_rescued Fallback timed outfallback_timed_out Fallback also failedfallback_also_failed NO YES YES NO YES NO YES SUCCESS TIMEOUT FAILED B · ONE REQUEST, COUNTED TWO WAYS
The old dashboard measured deployment paths rather than complete generation requests. A primary failure and the fallback that rescued it appeared as separate series, so the system could describe an endpoint accurately while misrepresenting what happened to the person waiting for a video.
C · THE RESULT
ENDPOINT GRAIN
GENERATION GRAIN
PRIMARY + FALLBACK + FINAL OUTCOMESUCCEEDED
Daily generation failure rate
One template appears as two separate lines—one for the primary path and one for fallback—so a single generation looks like two unrelated events.
Failed generation count + failed rate
With failure defined by the user’s final outcome, each generation now contributes one result—even when fallback uses multiple endpoints.
FROM VISIBLE TO ACTIONABLE
Engineering already had backend observability, but a spike could not be traced to a template, model, vendor, or failed step. The generation record added that attribution, plus HTTP status and duration.
How is_config_anomaly worksFallback configured + never fired + generation failed 244 CASES
ONE GENERATION RECORD
- TEMPLATE
- Which experience failed
- MODEL
- What served the request
- VENDOR
- Who owned availability
- FAILED STEP
- Where the pipeline broke
- HTTP STATUS
- How it failed
- DURATION
- How long it took
DERIVED SIGNAL · is_config_anomaly
THE CONTRACT HELD; THE DETAIL DID NOT
When part of the service moved to AWS Step Functions, the meaning of a row survived because it was defined around generations. Where fallback fields had no equivalent, I returned “unknown” and added a step-level heatmap as an interim diagnostic.
The outcome contract that survivedGeneration-level meaning stayed stable across the backend change UNKNOWN > GUESS
ORIGINAL PIPELINE30+ TEMPLATES
Full seven-mode detail Fallback fields available · definition intactAWS STEP FUNCTIONS10 TEMPLATES
Fallback fields null Some modes collapse to unknownREFLECTION
A metric definition is a product decision
WHAT I LEARNED
There was no external ground truth for whether a fallback-rescued generation “really” failed. Trust came from an inspectable definition, and a stable unit of analysis mattered more than a stable implementation.
WHAT’S STILL OPEN
Success rate does not capture whether an output was worth keeping. The implementation still needs manual validation, and fallback semantics remain less detailed on migrated workflows.