← Back to the portfolio

Technical deep dive · AI video reliability

No video may fail.

Generative video breaks in more ways than normal software: a model refuses a prompt, a provider times out, the product changes shape between clips, a voice line runs long. This is how I built the Creative Studio so a customer still gets a finished ad, and what happens on the rare day nothing can save it.

The rule
Only a provider's own failure may end a job
Retries
Bounded, and never charged twice
Quality checks
Repair or retake, never fail the video
Last resort
A plain cause, and the credits back

What was breaking

In the pre-launch test round, people met raw failure cards: a reference ID, a Retry button, "contact support". Some renders failed after ten minutes with no reason at all. Long sources of 45 to 120 seconds were lost inside the engine before they ever reached the video model.

Every one of those is a person who paid for an ad and got an error. So I set one rule for every pipeline: each step that can fail has to name the fallback that still makes the video.

  • Providerrefuses the prompt, times out, or is down
  • Modelchanges the product between clips
  • Speecha line runs long or is cut mid-word
  • Captionsno room on screen, or text already in the video
  • Inputa reference ad with nobody on camera, no speech
  • Enginea long source runs out of memory or time

One render, every way it can go

Watch a single clip: the first provider refuses, the fallback films it, the product drifts and the clip is filmed again, then it ships with one charge.

  1. Gate. Inputs that cannot work are refused before any money is spent: a reference ad with nobody on camera, or no speech to remake.
  2. Provider client. One client for every lane. Seedance runs on BytePlus Ark first and falls back to KIE; Veo and Gemini Omni lanes have their own fallback.
  3. QA gates. Word accuracy, product drift and caption room are checked. A check may retake or repair a clip; its verdict, or its own crash, never ends the job.
  4. Last resort. Only when the provider itself is down does a job end, with a plain cause and the credits back.

Five mechanisms that make it hold

01

One provider client, a fallback per lane

Every video lane calls one client instead of a provider directly. Seedance goes to BytePlus Ark first and KIE second, so an outage or refusal on one becomes a re-route, not an error.

The trade-off I kept: Clone edits a reference video, and Ark refuses a person in a reference video, so that one lane stays on KIE by rule.

02

Retries that are bounded and never charge twice

A refusal re-queues a bounded number of times. A video that failed on our side can be sent again twice within two hours, at the price that was quoted, with one charge.

In a multi-variant build, a failed variant retries on its own, so the set is never left short.

03

Re-film on drift

The product in every clip must be the one in the customer's photo. A clip where the model changed the product is filmed again before the cut, so a drifted bottle never reaches the ad.

04

Quality checks repair, they never kill

Hook variations must match the script at 75% word accuracy; captions are placed where there is room and skipped where the video already has text. If a check fails or crashes, the best take ships.

A missing value or a failed new check falls back to the old working behaviour, never to an error.

05

Failures that explain themselves

Every failure carries an origin label: our code, provider, input, infra or deploy. The customer sees a short, plain cause and the credits back; never a reference ID or a dead-end Retry.

How I prove it, not just claim it

  • Test first. Every fix ships as two commits: a test that injects the failure and fails, then the change that makes the video still come out.
  • Break it on purpose. Fault injection across the engine and the backend, with long 45 to 120-second sources tested first, since they fail first.
  • One scoreboard. Per pipeline: delivered, failed from our own code, failed from anything else. The bar I set is at most 5% of runs failing from our own code.
  • Real users. In the pre-launch round I fixed 20 customer-reported issues across 3 services in 3 days, test-first with pytest, Playwright and CI.

The overnight switch

The night our video provider stopped serving videos, I called the BytePlus team, walked them through the whole pipeline, and moved our video pipeline to Seedance on Ark directly, with the old provider kept as the fallback.

Built with

Python · FastAPI · Redis · PostgreSQL · ffmpeg · Seedance on BytePlus Ark · KIE · Veo · Gemini · OpenAI · pytest · Playwright

Want this kind of reliability on your AI product?