Running image and speech generation at home

I generate images, short video clips and narration at home, on one graphics card. Most of what I’ve learned isn’t about model quality — the models are all astonishing and they all get better every few weeks. It’s about the boring constraints around them: how long things take, how much memory they need, and whether you’re allowed to use what comes out.

The GPU decides everything, and it decides it by VRAM

The number that determines what you can run is video memory. Not the core count, not the generation. A card with 24GB will run things a faster card with 16GB simply cannot load.

That’s why an RTX 3090 is still one of the best value cards for this work despite being several generations old. It has 24GB, it’s widely supported, and the second-hand price reflects gamers wanting frame rates rather than capacity.

Quantisation moves the line without erasing it. Running weights at fp8 rather than fp16 roughly halves what a model needs and, in my experience, costs very little visible quality for this kind of work. That’s the difference between a model running and not running, which isn’t a subtle trade.

Expect minutes, not seconds

The expectation set by hosted services is that you type something and an image appears. Locally a still image is a handful of seconds, so that holds. Video doesn’t.

Generating a short video segment on the 3090 takes me somewhere between four and eight minutes depending on resolution and how many steps the sampler is doing, so chaining eight of those to make something a minute long is the better part of an hour of continuous GPU time for one clip.

That has a consequence I took a while to accept, which is that local video generation is a batch workload rather than an interactive one. Once you stop trying to iterate on it live and start treating it like a build job — submit, go away, come back to a finished artefact and review it — the whole thing becomes pleasant. Fought against, it’s agony.

The other one would be scheduling. If one job wants four hours of GPU and another wants twenty minutes, they need to know about each other. I ended up deliberately placing the long weekly job in the idle window between the short daily ones, which sounds obvious written down and wasn’t obvious at the time.

Two stages beat one

The approach that finally worked for video wasn’t “describe a clip and generate it”. It was generating keyframes as still images first, iterating there where a frame costs seconds rather than minutes, then feeding the chosen frames to a first-frame/last-frame model and letting it interpolate the motion between them.

The reason that works isn’t technical, it’s economic. Iteration belongs in the cheap stage. If you refine a prompt by regenerating a five-minute video you get four attempts an hour and you settle for the least bad one; if you refine the still and then generate once, you get what you actually wanted.

Two post-processing passes are worth having as standard — frame interpolation to raise the effective frame rate, and an upscale to reach delivery resolution — and both are fast relative to generation, so generating at a lower resolution and upscaling afterwards is dramatically cheaper than generating at full size.

The licence is a real constraint

This is the part I’d most want somebody to take away.

Model quality varies less than you’d think between the top open-weight options. Licences vary enormously. Some are Apache-2.0 and you can do what you like, commercially, without asking. Others are explicitly non-commercial, or research-only, or free until some revenue threshold.

I’ve twice picked the second-best model on quality because the best one had a licence that was incompatible with what the output was for, and both times it was the right call and mildly annoying. If anything you generate might ever sit next to advertising, be sold, or be used to promote something, read the licence before you download forty gigabytes of weights and build a workflow around them.

The same applies with more teeth to voices, because text-to-speech has a second layer of rights on top of the model licence: whose voice is it. A synthetic voice cloned from a real person without consent is a legal and an ethical problem regardless of what the model licence says. Stock or fully synthetic voices from a provider that grants commercial use are the boring, correct answer.

Where I didn’t self-host

I generate images and video locally. I don’t generate speech locally, and that was a decision rather than a technical limit.

Local text-to-speech has become genuinely good. But the gap between “good” and “doesn’t sound like a robot by the fourth sentence” is still real, and narration is the one component where a listener notices immediately. A hosted voice API costs a trivial amount per video and removes a whole class of quality problem, and the GPU is better spent on the thing that actually needs 24GB.

The principle I’d take from that is to self-host what’s expensive to buy and cheap to run badly, and to buy what’s cheap to buy and expensive to run badly.

Sound effects you can solve yourself

A small thing that saved a disproportionate amount of hassle. Rather than sourcing sound effects from “free” packs, which routinely carry attribution requirements or no-redistribution clauses that become a problem at any scale, I synthesise the simple ones — clicks, tones, transitions — in a few dozen lines of code producing waveforms directly.

It’s deterministic, so a rebuilt machine produces byte-identical audio, and it’s owned outright, so the licensing question disappears. Clean synthetic tones also suit an instrumentation aesthetic better than cinematic stock effects. That won’t work for anything organic, but for interface sounds it’s strictly better than the alternative.

What it costs

Roughly one graphics card, some electricity, and considerably more of your attention than you expect during setup. Running costs after that are close to zero, which is the whole point.

Whether that beats paying per generation depends entirely on volume. If you make a handful of things a month, hosted services win and it isn’t close. If you’re producing something every day the card pays for itself surprisingly quickly, and the marginal cost of trying something drops to nothing.