Video as code: Remotion, and where an LLM actually helps

Remotion lets you describe a video as React components and render it frame by frame to a file. You write a composition, every component is told which frame it is on, and a headless browser draws each one in turn.

It’s a genuinely different way to think about video, and the difference that matters isn’t that it’s programmable. It’s that video becomes text in a repository — diffable, reviewable, and, which is the part that changes what’s possible, generatable.

Why code beats a timeline for anything repeated

If you’re making one video, an editor is faster, and nothing about writing React competes with dragging a clip.

The calculus inverts the moment you’re making the same shape of video repeatedly, because a composition is a template with real logic in it. Change the brand colour once and every video ever made from that template changes. Add a caption style and it applies retroactively. Fix a spacing bug, re-render, done.

Two properties follow that are hard to get any other way. A change to a video becomes a diff, so you can see that somebody altered the title timing because it says so in the patch, and you can revert it the way you revert anything else. And it’s deterministic, so a rebuilt machine produces an identical video and a regression is a real regression rather than a vague sense that it used to look better.

The division of labour that works

The obvious idea is to ask a language model to write the composition. I tried it, and I think it’s the wrong shape.

Models will happily emit React that renders. What they won’t reliably emit is motion that feels right — the timing, easing and overlap that separate something watchable from a slide deck with transitions — and every mistake in that category is invisible in the code and obvious in the output, so you only find it by rendering, which is slow.

What works far better is that the model writes the words and the code owns the timing. The narration script, the caption text, the on-screen copy: those are language problems, and language models are extremely good at them. Get the script first, generate the narration audio, measure it, and derive every timing in the composition from the real measured duration.

That ordering matters more than anything else here. Write the animation first and generate the voiceover second and you’re forever nudging keyframes to fit audio that doesn’t quite match; let the audio come first and the composition read its actual length and the two can’t drift apart, because a scene becomes “as long as this sentence takes”, which is both correct and self-maintaining.

Structured output is what makes it reliable

The other place a model earns its keep is deciding the shape of a video — which scenes, in what order, carrying what content.

What turns that from a novelty into something dependable is asking for structured output, a schema the response must conform to, rather than prose you then parse. You get an object with the fields you asked for, validated, or you get an error. No regex over prose, no “sometimes it wraps the JSON in a code fence”.

Two rules I wouldn’t now break. The first is only offering choices that are actually available: I had a bug where the model picked a scene type the renderer had no data to fill, and it rendered as a heading over an empty screen for five seconds while the narration described content that wasn’t there. It looked like the video was broken, because it was. The fix wasn’t “populate the data” — it was computing the list of scene types that have data first, interpolating that list into the prompt, and then filtering the response against it anyway. Both layers matter, because the prompt keeps the words honest and the filter keeps the picture honest.

The second is enforcing limits in code rather than in the prompt. I asked for scripts under a certain length and got, on average, scripts under that length, with a long tail that overran badly. The prompt was advice. Nothing checked it, and the composition took its duration from the measured audio, so an overlong script silently produced an overlong video. Any constraint that actually matters needs a deterministic check after generation; if it’s only in the prompt, it’s a preference.

Budget for the gaps you forgot

A small thing that cost me an afternoon. Each scene had a short pause after its narration, a fraction of a second so lines don’t run together, plus a slightly longer one at the end.

Individually trivial. Across seven scenes it added several seconds, which meant a script sized to exactly fill the target duration produced a video that overran it, because the budget was computed against the total length rather than the length minus the padding the renderer would add.

The general form is that when you derive a budget from a duration, you subtract the overhead your own code introduces. Obvious in hindsight, invisible until you measure the output rather than trusting the plan.

Render times and the feedback loop

Remotion renders in a headless browser and parallelises across cores well, so on a machine with enough of them a short vertical video is a couple of minutes. That’s fast enough to be practical and slow enough that you don’t want it in a tight loop.

Two things helped. Extracting stills rather than videos when checking layout takes seconds and answers most “does this look right” questions, which leaves full renders for checking motion. And typechecking before rendering, always — the composition is TypeScript, and a type error caught in a second is a type error you didn’t find four minutes into a render.

Where it doesn’t fit

Remotion isn’t an editor and it’s bad at being one. Anything driven by footage — cutting to a performance, reacting to what’s on screen, matching a feel — wants a timeline and a human, and expressing that as code is a lot of work to arrive somewhere worse.

The sweet spot is designed video: motion graphics, explainers, data-driven pieces, anything templated, where the structure is known in advance and the content varies. For those I’d take it over a timeline, and the fact that a language model can fill in the content makes the whole thing move at a speed that surprised me.