Building an AI Shorts Generator: From Prompt to Pipeline

Creating short‑form clips from longer videos eats up hours if you do it by hand. An AI shorts generator can automate the cut, caption, and export steps so you spend minutes instead of mornings. By the end of this guide you’ll have a working pipeline you can run locally or in the cloud.

What most guides get wrong

Most tutorials treat the AI shorts generator as a single magic button. They show you a demo where a ten‑minute video becomes a perfect fifteen‑second clip with captions, then they stop. In reality the pipeline has at least four moving parts: transcription, scene detection, clip assembly, and export. If any one of those pieces drifts, the whole thing falls apart.

Another common mistake is ignoring aspect‑ratio safety. Many guides assume the source video is already 16:9 and that cropping to 9:16 will keep the subject centered. That’s not true for talking‑head footage where the person drifts left or right. You need a smart reframing step or you’ll end up with half a face.

Finally, people forget about cost controls. Running a transcription model on every uploaded video can burn through credits fast if you don’t cache results or set a concurrency limit. I’ve seen a solo creator blow $80 in a single night because the script launched a new WhisperX job for each retry.

Why does the AI shorts generator lose sync at scale?

When you start processing dozens of videos per day, tiny timing errors add up. The transcription step returns word‑level timestamps with a few milliseconds of jitter. The scene‑detector then picks a start frame based on those timestamps, but the video codec may drop or repeat frames during encoding. The result is a clip where the audio is ahead of the lips by a noticeable amount.

Another sync killer is variable frame rate. Sources from phone recordings or webcams often switch between 30 fps and 60 fps mid‑file. If you assume a constant frame rate when converting timestamps to frame numbers, the drift grows linearly with clip length.

Finally, batch processing on shared hardware can cause GPU memory fragmentation. When the model runs out of VRAM it falls back to CPU, which is slower and introduces additional latency that wasn’t present in single‑run tests.

A concrete example: using WhisperX and Pika Labs

Let’s walk through a real‑world prompt and the tools I actually use. First, I feed a YouTube URL to a small Python script that downloads the MP4 with yt-dlp. Next, I run the audio through WhisperX to get word‑level timestamps. The prompt I give WhisperX is simple: “transcribe with word timestamps and language detection.”

Here’s the exact command I run in a terminal:

whisperx audio.mp3 --model large-v2 --language en --output_format json

The output is a JSON file where each word has a “start” and “end” field in seconds. I then load that JSON into a Python function that looks for the highest‑energy three‑second window (based on RMS volume) and cuts a 9:16 clip using FFmpeg. The FFmpeg filter chain looks like this:

ffmpeg -i input.mp4 -vf "crop=in_w:in_h*0.5625:(in_w/2)-(out_w/2):(in_h/2)-(out_h/2),scale=1080:1920" -c:a copy -t 9 output.mp4

Notice the crop math: I assume the source is landscape and I want a vertical slice that keeps the center. This works well for centered speakers but fails when they drift.

For the voice‑over I sometimes replace the original audio with a synthetic version from ElevenLabs. The prompt there is: “read this script in a friendly, upbeat tone, pace 1.1x.” The cost for ElevenLabs is $5 per month for 30 000 characters, which is enough for about twenty shorts.

Putting it together, the total per‑video cost looks like this:

• WhisperX API (self‑hosted on a GPU instance): $0.0002 per second of audio → ~$0.02 for a 2‑minute source
• FFmpeg compute: negligible on a modest VPS
• ElevenLabs (optional): $0.00016 per character → ~$0.08 per short

Even with the optional voice‑over, each short stays under $0.15 in compute, which feels fair for the time saved.

How to debug when this breaks

First, check the transcription timestamps. Open the WhisperX JSON and look for any word where “end” is less than “start” – that indicates a failed alignment. If you see that, reduce the model size or increase the CPU timeout.

Second, verify the frame‑rate assumption. Run ffmpeg -i input.mp4 and note the reported fps. If it varies, replace the fixed‑fps conversion with a timestamp‑based seek: use -ss before the -i flag for accurate seeking.

Third, inspect the crop coordinates. If the subject’s eyes move outside the cropped region, the clip will look awkward. I add a simple heuristic: compute the average x‑position of the face landmarks (using mediapipe) and shift the crop window to keep that average centered.

Finally, watch the export logs for GPU OOM errors. If you see “CUDA out of memory”, lower the batch size in WhisperX or switch to the “base” model instead of “large‑v2”.

Putting it all together: a numbered setup

1. Install dependencies: pip install yt-dlp whisperx mediapipe and ensure FFmpeg is on your PATH.
2. Create a folder shorts_pipeline and add a file run.py with the following steps:
1. Download video with yt-dlp.
2. Extract audio to audio.wav.
3. Run WhisperX on the audio and save transcript.json.
4. Load the JSON, find the loudest three‑second window using RMS.
5. Calculate crop coordinates based on mediapipe face detection (if available) or fallback to center.
6. Run FFmpeg with the crop, scale to 1080×1920, and copy audio.
7. (Optional) Send the transcript to ElevenLabs for a synthetic voice‑over and mux it back.
3. Make the script executable: chmod +x run.py.
4. Test with a short clip under two minutes to verify timing.
5. Scale up by wrapping the script in a simple bash loop or a cron job that watches a Dropbox folder for new MP4s.

One‑sentence paragraph: I’ve seen this fail more times than I’d like to admit.

Price thoughts and final recommendation

I think paying $49/mo for a basic captioning API is overpriced when you can run WhisperX yourself for a fraction of that cost. The free tier of Kapwing adds a watermark you can’t remove without upgrading — which, yes, is annoying. I love how WhisperX gives you word‑level timestamps that line up perfectly with speech, making the sync problem tractable.

Adjacent reading: AI meeting tools coverage.

If you’d rather skip the build and deploy a working version in an afternoon, we’ve packaged this workflow as a blueprint at deepusecase.com/vault/ai-ad-renderer-blueprint.