How to Build an AI Shorts Generator That Doesn't Suck

The Core Stack: What You Actually Need

Forget the all-in-one, click-a-button web apps. They produce generic, soulless content that gets zero engagement. If you want to build a real ai shorts generator for your business, you need to assemble it from specialist components. This is the only way to get the control you need for quality output. It’s an AI automation blueprint, not a toy.

Here’s the stack I use:

• The Brain (LLM): This is your scriptwriter. I use OpenAI‘s GPT-4o via their API for this. It’s fast, relatively cheap, and its instruction-following for structured output (like JSON) is reliable enough for production. You could use Claude 3 Opus, but I find it gets a bit too verbose for short-form video scripts.
• The Voice (TTS): You need a text-to-speech API that doesn’t sound like a robot from 2008. ElevenLabs voice is the clear winner here. The quality is exceptional, and their API is straightforward. The Starter plan at $5/mo is a no-brainer for high-quality voiceovers. Honestly, the free tier is a joke for any real work due to the limitations and lack of commercial licensing.
• The Assembler (Video API): This is the magic piece that stitches everything together. You send it a timeline with video clips, audio files, text overlays, and transitions, and it renders a final MP4 for you. I use Shotstack for this. It’s a true API-first video editor, which is exactly what an operator needs. It’s not a drag-and-drop tool with an API tacked on as an afterthought.

That’s it. Three core components. You orchestrate these with a simple workflow tool. You can use the Make platform or Pipedream for this. I prefer Pipedream for its code-level control, but Make is a solid no-code AI option if you’re less technical.

What Most Guides Get Wrong About AI Shorts

Most tutorials on this topic are fundamentally flawed. They show you how to create a glorified PowerPoint presentation with a robot voiceover. They miss the entire point of what makes a Short or Reel engaging.

Here’s what they get wrong:

1. They treat it as a single prompt problem. They’ll show you a massive, complicated prompt that tries to generate a script, scene descriptions, and keywords all at once. This never works reliably. The LLM gets confused, and the output quality plummets. Real automation uses a chain of smaller, focused prompts.
2. They rely on static images. A video made of DALL-E images panning slowly is not a video. It’s boring, and platforms are starting to de-prioritize this kind of low-effort content. You need actual video clips to hold attention.
3. They ignore pacing. Shorts need to be fast. The visuals must change every 1-3 seconds. The script needs to be punchy. Most AI-generated scripts are rambling and academic. You have to explicitly prompt for a fast-paced, direct style suitable for the medium.

The goal isn’t just to automate video creation. It’s to automate the creation of *good* video. Big difference.

Step-by-Step: From Topic to Rendered Short

Let’s build a workflow for a short video explaining the concept of ‘API’.

Step 1: Generate the Script with a Master Prompt

First, we hit the GPT-4o API. The key is to ask for the script *and* a list of visual concepts for each sentence. We also demand the output in a clean JSON format.

{ "role": "system", "content": "You are a scriptwriter for viral tech-explainer TikToks. Your style is extremely direct, fast-paced, and uses simple analogies. You will write a script that is under 60 seconds when read aloud. For each sentence, provide a corresponding visual concept. The final output must be a JSON object with a single key 'scenes', which is an array of objects, each with 'text' and 'visual' keys." }

{ "role": "user", "content": "Write a script explaining what an API is." }

The LLM will return a structured JSON that your automation workflow can easily parse. This is far more reliable than trying to regex a block of text.

Step 2: Generate the Voiceover

Take the ‘text’ value from each scene and concatenate them into a single string. Send this to the ElevenLabs API. It will return an MP3 file. Store this file somewhere accessible, like an S3 bucket, and save its URL. The duration of this audio file is now your master timing for the entire video.

Step 3: Source Visual Assets

Now, iterate through the ‘visual’ value for each scene from Step 1. These are your search queries for a stock video provider. Use the Pexels API or Storyblocks API. For a visual like “waiter taking order from a customer,” you’ll search for that. My biggest gripe with the Pexels API is its search relevance; you ask for ‘server rack blinking lights’ and get a stock photo of a dinner server. It’s infuriating and requires either very clever keyword mapping in your prompt or a manual review step.

For each scene, grab the URL of the best video clip you find.

Step 4: Assemble and Render the Video

This is where Shotstack comes in. You’ll construct a JSON payload that defines the video edit. You’ll have one audio track (the ElevenLabs MP3) and multiple video clips on the timeline. You calculate the duration for each clip to match the pacing of the voiceover. You can also add text overlays, captions, and transitions here.

A simplified Shotstack request might look something like this:

{ "timeline": { "tracks": [ { "clips": [ { "asset": { "type": "video", "src": "PEXELS_VIDEO_URL_1" }, "start": 0, "length": 3.5 }, { "asset": { "type": "video", "src": "PEXELS_VIDEO_URL_2" }, "start": 3.5, "length": 4.1 } ] }, { "clips": [ { "asset": { "type": "audio", "src": "ELEVENLABS_MP3_URL" }, "start": 0 } ] } ] }, "output": { "format": "mp4", "resolution": "1080p", "aspectRatio": "9:16" } }

You send this to their API. It queues the render job. A few minutes later, you get a webhook ping with the URL to your final, rendered MP4 file. Done.

Why Does My AI Shorts Generator Produce Such Generic Garbage?

This is the most common question, and the answer isn’t about finding a better prompt. It’s about your process. If your generator is spitting out boring videos, it’s almost certainly because you’re trying to do too much in one step.

A single-prompt approach fails because the AI has to compromise. It tries to write a good script, imagine good visuals, and format it all correctly at the same time. It does all three poorly.

The operator playbook is to break the problem down. The chain of specialized prompts is the solution:

• Prompt 1: Core Script. Focus only on the narrative, the hook, and the call to action. Get the words right first.
• Prompt 2: Visual Brainstorm. Feed the approved script back into the LLM and ask it to *only* generate vivid, interesting, and concrete visual ideas for each line. Tell it to avoid abstract concepts.
• Prompt 3: Asset Keyword Extraction. Feed the visual descriptions from Prompt 2 back into the LLM and ask it to extract 2-3 perfect search keywords for a stock video API. This step alone dramatically improves the quality of the sourced footage.

This multi-step process, this AI agent framework, feels slower to design but produces exponentially better results. It treats the LLM like a specialized worker on an assembly line, not a magical black box.

This is the difference between building a demo and building a system that actually produces assets you’d be willing to put your brand on.

How to Debug When This Breaks

And it will break. Building real automation means building for failure. Here are the three most common failure points I’ve hit in production and how to fix them.

1. API Timeouts and Rate Limits: You’ll be making a lot of API calls in quick succession. Pexels might rate limit you, or the ElevenLabs render might take longer than your workflow orchestrator’s default timeout. The fix is to build in pauses and ‘wait’ steps. In Pipedream, I can add a `$.flow.delay(5000)` to wait 5 seconds between calls. Also, implement exponential backoff for retries. If an API call fails, wait 2 seconds, retry. If it fails again, wait 4, then 8. Don’t just hammer the API.

2. Audio/Video Mismatch: The most common bug is the video ending before the audio, or vice-versa. This happens when you miscalculate the total duration. The fix is to treat the generated audio file as the source of truth. After you generate the voiceover with ElevenLabs, get its exact duration (they provide it in the API response). This number becomes the target length for your entire video timeline in Shotstack. All your video clip durations must add up to this number. No guesswork.

3. Bad Visuals Sourced: As mentioned, stock video APIs can be hit-or-miss. The debug process here is manual at first. Log every search term you send to the Pexels API and the top 3 video results it returned. When you see a bad match, you can fine-tune your keyword extraction prompt (Prompt 3 from the previous section). For example, you might add a rule: “Never use the words ‘business’ or ‘corporate’ in the keywords.” This iterative refinement is the core work of an AI operator.

The best part of this component-based system is its testability. I can run just the scripting part 100 times to see how often the JSON format breaks. I can’t do that with a closed, all-in-one tool. That control is everything.

For more on this exact angle, deeper coverage of AI agent platforms.

If you’d rather skip the build and deploy a working version in an afternoon, we’ve packaged this entire workflow as a deployable system. It’s the AI Ad Renderer Blueprint, and it’s available at deepusecase.com/vault/ai-ad-renderer-blueprint.