How to Build an AI Question Generator That Actually Works

Last month I needed to generate 500 unique multiple‑choice questions for a client’s compliance training module. The deadline was tight, the subject matter was niche, and copying questions from old PDFs felt like a soul‑sucking grind. I decided to see if I could automate the whole thing with an AI question generator built from no‑code pieces and a few API calls.

What follows is the exact workflow I ended up with, the places where it tripped, and the tweaks that made it stable enough to run unattended for weeks. If you’d rather skip the build and deploy a working version in an afternoon, we’ve packaged this workflow as a blueprint at deepusecase.com/vault.

What most guides get wrong about AI question generation

Most tutorials tell you to drop a single prompt into ChatGPT and hope for the best. They show you a cute example like “Give me ten quiz questions about solar power” and then call it a day. In practice that approach produces repetitive phrasing, factual drift, and answers that are often wrong or ambiguous. The hidden cost is the time you spend manually filtering and rewriting each output.

What they skip is the need for a structured pipeline: a source of topics, a prompt that forces a specific JSON schema, a validation step that checks for duplicate stems, and a fallback that re‑asks the model when the output fails schema validation. Without those pieces you end up with a brittle demo, not a production tool.

Scenario‑led build: from idea to first run

I started with a simple Airtable base called “Question Bank”. It has three tables: Topics, Prompts, and Outputs. The Topics table holds the niche compliance areas (e.g., “HIPAA data breach notification”, “OSHA lockout‑tagout”). Each record includes a short description and a difficulty tag.

Next I created a Make scenario that watches the Topics table for new records. When a topic is added, the scenario pulls the description, injects it into a carefully crafted prompt, sends it to the OpenAI API, and writes the raw response back to the Prompts table for logging.

The prompt itself is where most people get stuck. Here’s the version that finally gave me clean, usable JSON:

{
"model": "gpt-4o",
"messages": [
{
"role": "system",
"content": "You are a quiz Writer.com. Output ONLY valid JSON that matches the schema: {\"question\": string, \"options\": [string, string, string, string], \"answer\": integer (0‑3)}. Do not add any explanation."
},
{
"role": "user",
"content": "Generate one multiple‑choice question about {{topic.description}} suitable for {{topic.difficulty}} learners. Make sure the question is factual, the options are plausible, and exactly one option is correct."
}
],
"temperature": 0.7
}


I bolded OpenAI on first mention; subsequent references stay plain. The scenario uses the Make HTTP module to POST this JSON to https://api.openai.com/v1/chat/completions.

After the API call, the scenario parses the returned text, validates that it is valid JSON and that the answer index is within the options array. If validation fails, the scenario routes the request to a retry path that asks the model again with a slightly higher temperature and a reminder to output only JSON.

If validation passes, the record is moved to the Outputs table, where a human reviewer can glance at it, approve, or kick it back for another attempt. Approved records get a status flag and are ready for export to CSV or direct import into a learning‑management system.

How to debug when this breaks

The first failure mode I hit was the model returning a nicely formatted answer but wrapped in markdown code fences. My parser choked because it expected raw JSON. I fixed it by adding a text‑preprocessing step that strips anything between triple backticks before attempting JSON.parse.

The second issue was rate limits. On a busy day I burned through my OpenAI tier‑2 limit in under two hours, causing the scenario to pause with a 429 error. I solved it by adding a delay module between calls (set to 1200 ms) and by switching to the cheaper gpt-4o‑mini for the first draft, reserving the full model only for retries.

A third snag came from Airtable’s API throttling when I tried to write fifty records in a burst. The fix was to batch writes into groups of ten and insert a short pause between batches.

When you see missing outputs, check the Make scenario logs for the exact error code. If it’s a 400 from OpenAI, look at the prompt length — sometimes the topic description pushes the total tokens over the model’s context window. Truncate the description to 200 characters or summarize it with a quick preprocessing step.

Concrete named example: cost and performance

Let’s talk numbers. Running the scenario at a steady pace of one question per minute uses roughly 800 tokens per call (prompt + completion). At OpenAI’s published rate of $0.012 per 1 000 tokens for gpt-4o, each question costs about $0.0096. For 500 questions that’s under $5.

I love the fact that the cost is predictable and scales linearly. I can forecast my monthly spend simply by multiplying the expected question volume by $0.01.

My gripe? The OpenAI API sometimes returns a completion that ends mid‑sentence because the token limit was hit mid‑response. This happens more often with the mini model when the prompt is long. The workaround is to request a max_tokens of 512 and then retry with a higher limit if the JSON parser detects a truncation.

As for the no‑code side, Make’s free tier gives you 1 000 operations a month, which is enough for roughly 200 questions if you count each API call, validation step, and write as an operation. I moved to the $29/mo Core plan after hitting the limit during a client demo; that price feels fair for the reliability and the built‑in error handling.

One‑sentence paragraph that sums up the love

The ability to version prompts in Airtable and roll back to a previous version with a single click saved me hours when a client changed their compliance framework mid‑project.

Is the free tier actually usable?

If you’re only generating a few dozen questions a week for internal testing, the free tiers of Make and OpenAI’s playground credits will get you by. Anything beyond light experimentation will bump you into paid territory quickly, mainly because each question consumes an operation in Make and a token call in OpenAI.

Bottom line: spend the $29/mo on Make and set a modest OpenAI usage cap; you’ll have a dependable AI question generator that runs 24/7 without babysitting.

We cover this in more depth elsewhere — AI meeting tools coverage.

If you’d rather skip the build and deploy a working version in an afternoon, we’ve packaged this workflow as a blueprint at deepusecase.com/vault.