PRODUCT ANALYTICS · AI RELIABILITY

Rebuilding AI video metrics around user outcomes

TL;DR

I found that endpoint error counts were misclassifying fallback-rescued generations as failures. I redefined success at the generation level and built four role-specific views across 40+ templates, now used daily by Content Ops and weekly by product leadership.

SKILLS & STACK

  1. 01Product analytics
  2. 02Metric definition & governance
  3. 03Data modeling
  4. 04Dashboard & decision systems
  5. 05Reliability monitoring

BACKGROUND

THE PROBLEM

All 40+ video templates appeared as one aggregated line. Worse, generations rescued by fallback were still counted as failures, so teams could not tell what to promote, fix, or retire.

WHAT I DID

Rebuilt measurement around one row per generation, defined failure by the user’s final outcome, added seven outcome modes, and tied each failure to its template, model, vendor, and step.

IMPACT

Outcome-based analytics that Content Ops monitors daily and product reviews weekly.

5×failure overstatement corrected
40+templates measured individually
244fallback anomalies surfaced
PRODUCTION HEALTH VIEW
REDACTED FOR CONFIDENTIALITYJULY 2026
Redacted production health dashboard with volume, failure, success, conversion, recovery, user, and alert metrics
Surfaces configuration anomalies quickly and shows how each template behaves today and across the week.

HOW I BUILT IT

The decisions behind the layer

The user outcome had to be defined before the grain, attribution, or migration constraints could be made legible.

ROUTE

USER OUTCOME REQUEST GRAIN ATTRIBUTION MIGRATION
01 /Redefinition

Measure the request, not the deployment path

A · MEASURE FAILURE AT THE LEVEL OF THE USER OUTCOME

Roughly half the templates route to a second engine when the first fails. Fan’s Golden Goal made the consequence clear: endpoint counting reported 39.1% failure; the user-level outcome was 7.4%.

THE 5× OVERSTATEMENT

ENDPOINT-BASED REPORTED FAILURE39.1%

USER OUTCOME7.4%

5×Endpoint counting reported 39.1%; the user-level outcome was 7.4%.
Seven outcome modesA decision tree replaces one ambiguous boolean with outcomes that have different owners and next actions. 7 MODES

B · ONE REQUEST, COUNTED TWO WAYS

The old dashboard measured deployment paths rather than complete generation requests. A primary failure and the fallback that rescued it appeared as separate series, so the system could describe an endpoint accurately while misrepresenting what happened to the person waiting for a video.

C · THE RESULT

ENDPOINT GRAIN VS. GENERATION GRAIN
ONE REQUEST PRIMARY FAILED→ FALLBACK SUCCEEDED→ DELIVERED · USER HAS IT

ENDPOINT GRAIN

Primary attemptFAILED
Fallback attemptSUCCEEDED
Two unlinked rows. The primary reads as failed.

GENERATION GRAIN

One generation
PRIMARY + FALLBACK + FINAL OUTCOME
SUCCEEDED
One row. The user’s final outcome survives.

Daily generation failure rate

Before: endpoint-grain failure rate split across deployment paths, with one generation appearing as separate lines

One template appears as two separate lines—one for the primary path and one for fallback—so a single generation looks like two unrelated events.

02Attribution

FROM VISIBLE TO ACTIONABLE

OBSERVE SPIKE→JOIN TEMPLATE→NAME MODEL, VENDOR, AND FAILED STEP→ROUTE TO OWNER

Engineering already had backend observability, but a spike could not be traced to a template, model, vendor, or failed step. The generation record added that attribution, plus HTTP status and duration.

PER-TEMPLATE TABLE
REDACTED
TEMPLATESUCCESSFALLBACKPUBLISHANOMALIES
Template — 01————
Template — 02————
Template — 03————
How is_config_anomaly worksFallback configured + never fired + generation failed 244 CASES

ONE GENERATION RECORD

TEMPLATE
Which experience failed
MODEL
What served the request
VENDOR
Who owned availability
FAILED STEP
Where the pipeline broke
HTTP STATUS
How it failed
DURATION
How long it took

DERIVED SIGNAL · is_config_anomaly

Fallback configuredTRUE
Fallback firedFALSE
Generation succeededFALSE
A routing problem rather than a model problem.244
03Migration

THE CONTRACT HELD; THE DETAIL DID NOT

When part of the service moved to AWS Step Functions, the meaning of a row survived because it was defined around generations. Where fallback fields had no equivalent, I returned “unknown” and added a step-level heatmap as an interim diagnostic.

The outcome contract that survivedGeneration-level meaning stayed stable across the backend change UNKNOWN > GUESS
PARTIAL-MIGRATION ASYMMETRY
THE CONTRACT HELD · THE DETAIL DID NOT
Generation-level outcome contractONE FINAL OUTCOME

ORIGINAL PIPELINE30+ TEMPLATES

Full seven-mode detail Fallback fields available · definition intact

AWS STEP FUNCTIONS10 TEMPLATES

Fallback fields null Some modes collapse to unknown
INTERIM DIAGNOSTICStep-level heatmap recovers missing detail

REFLECTION

A metric definition is a product decision

WHAT I LEARNED

There was no external ground truth for whether a fallback-rescued generation “really” failed. Trust came from an inspectable definition, and a stable unit of analysis mattered more than a stable implementation.

WHAT’S STILL OPEN

Success rate does not capture whether an output was worth keeping. The implementation still needs manual validation, and fallback semantics remain less detailed on migrated workflows.