How to Build an AI Question Generator That Doesn't Suck

The Core Problem: Getting Good Questions from Raw Text

You have a document. It might be a 3,000-word blog post, a transcript from a webinar, or a dense PDF manual. You need to pull questions out of it—for a quiz, an FAQ section, or a study guide. Doing it by hand is a soul-crushing, multi-hour task. The obvious solution is to use an AI question generator. The problem is that most simple approaches produce generic, useless questions and break the moment you try to automate them. This guide shows you how to build one that actually works.

We’re not talking about a playground toy. We’re talking about a repeatable system you can plug into a real workflow, using an API and structured data. After reading this, you’ll be able to build an automation that takes raw text and outputs clean, usable questions in a format a computer can understand.

The Engine: Your Prompt Is 90% of the Battle

Everything starts with the prompt. If you get this wrong, nothing else matters. The common mistake is a lazy, one-line request like “Generate questions from this text.” That gives the AI far too much creative freedom, resulting in inconsistent, unparsable garbage.

You need to be brutally specific. Your goal is to force the Large Language Model (LLM) into a tight corner where the only possible output is the exact format you need. For this, we’ll use a structured prompt that specifies the output as JSON. Why JSON? Because it’s a universal data format that every automation tool, from Zapier automations to custom code, can read instantly.

Here’s a solid starting prompt for generating factual questions:

<strong>System Prompt:</strong><br>You are an expert instructional designer. Your task is to generate a specified number of factual questions based on the provided text. You will ONLY respond with a valid JSON object. Do not include any introductory text, explanations, or markdown formatting. The JSON object must contain a single key, "questions", which is an array of objects. Each object in the array must have two keys: "question_text" (a string) and "answer_text" (a string).<br><br><strong>User Prompt:</strong><br>Please generate 5 factual questions from the following text.<br><br>[PASTE YOUR TEXT HERE]

This prompt does several critical things:

• It assigns a role: “You are an expert instructional designer.” This primes the model to think about question quality, not just text extraction.
• It defines the output format: “ONLY respond with a valid JSON object.” This is the most important instruction. The capitalization is intentional; you’re yelling at the machine to ensure compliance.
• It specifies the schema: It details the exact keys (`questions`, `question_text`, `answer_text`) and data structures (an array of objects) to use. This eliminates ambiguity.

This level of instruction is the difference between a fun experiment and a production-ready component of an AI agent framework.

What Most Guides Get Wrong

Almost every tutorial I’ve seen on this topic demonstrates the process in the ChatGPT web interface. They paste in some text, get a nicely formatted list of questions, and declare victory. This is a lie.

It’s a lie because the web UI is designed to be helpful and forgiving. It formats output for human eyes. The API, which you need for any real automation, is not. The moment you try to pipe the output of a lazy prompt from an API call into another system—like a Google Sheet or a database—the whole thing collapses. You’ll get back a block of text with random markdown, conversational intros like “Sure, here are the questions you requested!”, and other junk that will choke any parser.

My biggest gripe is this happy-path demonstration. It completely ignores the single most difficult part of building LLM-powered workflows: output reliability. Getting a good result once is easy. Getting a perfectly structured, machine-readable result 1,000 times in a row is hard. That’s the real job. Focusing on structured data (like JSON) from the very beginning is the only way to build something that doesn’t require constant manual intervention.

Choosing Your AI Model and Wrapper

You have options for the brain of your AI question generator. The big three are OpenAI’s GPT models, Anthropic’s Claude models, and open-source options like Llama 3. For this specific task, I’ve found a clear winner.

My pick is Claude 3 Sonnet. Here’s why. It has an enormous context window (200K tokens), meaning you can feed it very long documents without needing to chunk them up first (which is a huge workflow simplification). More importantly, it is exceptionally good at following complex formatting instructions, especially for JSON output. It’s also significantly cheaper than its more powerful sibling, Opus, or OpenAI’s GPT-4o. The price, around $3 per million input tokens, is a bargain for this level of performance. It’s so cheap that generating questions for a hundred blog posts might cost you a dollar.

Honestly, the free plans for these models via their APIs are mostly for testing. You’ll hit rate limits immediately if you try to do any real work.

To use the model, you need an API wrapper. You aren’t going to be building raw cURL requests. A no-code AI platform like Make or Pipedream is perfect for this. They provide pre-built blocks for calling the Anthropic or OpenAI APIs, handling all the authentication and request formatting for you. My personal preference is Pipedream. I love that if a pre-built action isn’t quite right, you can drop into a Node.js code step and write a few lines of JavaScript to process the data exactly how you need. It’s the ideal blend of no-code speed and pro-code flexibility.

How Do You Handle Different Question Types?

A truly useful AI question generator needs to be more than a one-trick pony. You won’t always want simple factual questions. Sometimes you’ll need multiple-choice questions for a quiz, or open-ended conceptual questions for a study guide. The good news is, you don’t need a separate system for each. You just need a more advanced prompt.

You can modify your prompt to accept a variable that dictates the question type. Think of it like a function in your automation.

Here’s how you could adapt the prompt for multiple-choice questions:

<strong>System Prompt:</strong><br>... You will ONLY respond with a valid JSON object. ... Each object in the array must have four keys: "question_text" (a string), "options" (an array of 4 strings), "correct_answer" (a string that exactly matches one of the options), and "explanation" (a brief string explaining why the answer is correct).

By changing the required JSON schema, you force the model to generate not just the question, but also the distractors and the correct answer in a structured way. In your automation workflow, you can have a simple dropdown or a variable that says `question_type = ‘multiple_choice’` and then use logic to select the correct prompt template. This turns your single-purpose tool into a flexible part of your operator playbook.

This is a small change, but it’s what separates a basic script from a reusable asset.

Building the Automation: A No-Code Workflow

Let’s walk through a concrete example using Make. This workflow will watch a Google Drive folder, and any time a new text file is added, it will generate questions and save them to a Google Sheet.

1. Trigger: Google Drive – Watch Files. Configure this module to watch a specific folder for new files.
2. Module: Google Drive – Download a File. This step takes the file ID from the trigger and downloads its content.
3. Module: Anthropic – Create a Completion. This is the core step. You’ll connect your Anthropic account. In the prompt section, you’ll insert the system prompt from above and map the downloaded text content from Step 2 into the user prompt. You’ll also set the model to Claude 3 Sonnet.
4. Module: JSON – Parse JSON. The output from the Anthropic module is a string of JSON text. This module turns that string into a data structure that subsequent modules can use. You’ll map the `choices[0].message.content` from the Anthropic output into this module.
5. Module: Iterator. The output of the parser is an array of questions. The iterator takes this array and processes each question one by one. You’ll map the `questions` array from the JSON module here.
6. Module: Google Sheets – Add a Row. For each question the iterator processes, this module adds a new row to your specified sheet. You’ll map the `question_text` and `answer_text` from the iterator into the appropriate columns.

This entire process can be built in under an hour without writing a single line of code. It’s a perfect example of a practical AI automation blueprint.

How to Debug When This Breaks

It will break. Every automation does. The key is knowing what to look for.

First, the most common failure is the LLM ignoring your JSON instructions. It happens. Sometimes it will wrap the JSON in markdown backticks ( … ) or add a friendly sentence before it. This will cause your “Parse JSON” step to fail. The fix is to make your prompt even more aggressive: “Your response MUST begin with `{` and end with `}`. Do not add any other text or formatting.” Sometimes, you may also need to add a text parsing step before the JSON parser to strip out any garbage—which, yes, is annoying but sometimes necessary.

Second, you might get low-quality or irrelevant questions. This usually happens when the source text is either too long or too vague. The model loses focus. The solution is to pre-process the text. Before sending a 10,000-word document to the LLM, break it into 1,500-word chunks. Run the question generator on each chunk individually. The results will be far more specific and useful.

Third, you’ll eventually hit API errors. Rate limits, server timeouts, temporary outages—they happen. A professional-grade workflow needs error handling. Make and other platforms have built-in error handling directives. You can configure a step to automatically retry a few times if it fails. For a production system, this isn’t optional. An automation without error handling is just a fragile prototype waiting to die.

Building a reliable AI question generator is entirely achievable. It just requires more discipline than most tutorials let on. You have to think like a systems builder, focusing on structure, reliability, and error handling, not just on a single, perfect-looking output.

For more on this exact angle, deeper coverage of AI agent platforms.

If you’d rather skip the build and deploy a working version in an afternoon, we’ve packaged this entire AI question generator workflow as an ai automation blueprint at deepusecase.com/vault.