How to Build an AI Chart Generator That Actually Works

The Core Stack: LLM Brains and a Rendering Engine

At its heart, an AI chart generator isn’t magic. It’s just two components working together: a Large Language Model (LLM) to act as the data analyst, and a charting library to draw the picture. The goal is to feed it raw data—a messy CSV, a pasted table, a block of text—and get back a clean, useful visualization.

For the LLM, you need something with a strong reasoning engine and a large context window. I use Claude 3 Opus for this kind of task. Its ability to handle large data dumps and follow complex, multi-step instructions is critical. You could use a GPT-4 model, but I’ve found Claude is slightly more reliable for generating structured JSON, which is exactly what we need here.

For the rendering engine, you need a library that accepts a declarative specification. This means you describe the chart you want in a format like JSON, and the library handles the rendering. Vega-Lite is the gold standard for this. It’s expressive, well-documented, and the JSON format is something an LLM can learn to generate perfectly. You could use something like Chart.js, but it often requires writing JavaScript code, which is a much harder and more error-prone target for an LLM to generate directly.

The whole point of building this yourself is control. You aren’t stuck with a pre-canned tool’s limited chart types or weird formatting. You define the logic, you control the prompts, and you can integrate it directly into your existing workflows, like a Slack bot that charts sales data on command or a system that auto-generates visuals for client reports.

What Most Guides Get Wrong About Prompting for Charts

If you search for this topic, you’ll find a ton of simple tutorials. They all make the same mistake. They use a single, monolithic prompt that looks something like this: “Here is some data. Make me a bar chart.”.

This approach is fragile and will fail constantly in production.

A single prompt asks the LLM to do too many things at once: analyze the data, decide on the best chart type, extract the correct columns, format the data, and generate a perfect chart specification. It’s a recipe for inconsistent, unpredictable output. My biggest gripe with this method is how it fails silently—you just get a broken chart or a nonsensical visualization with no clue which of the five steps went wrong.

The correct way to build this is with a two-step prompt chain. You break the problem down into two distinct tasks, handled by two separate LLM calls.

• Step 1: Analysis & Chart Selection. The first prompt’s only job is to understand the data. You feed it the raw data and ask it to act as a data analyst. Its output should be a simple JSON object describing the data’s structure (columns, data types) and recommending the most appropriate chart type with a brief justification. For example: {"analysis": "The data represents monthly user growth over time. A line chart is the best way to visualize this trend.", "chart_type": "line"}.
• Step 2: Specification Generation. The second prompt takes the original data AND the JSON output from Step 1. Its only job is to generate the final, valid Vega-Lite JSON specification. The prompt is much simpler now: “Using the provided data and the analysis that recommends a line chart, generate a complete Vega-Lite JSON specification.”.

This separation of concerns is the key to reliability. If the final chart is wrong, you can inspect the output of Step 1 to see if the analysis was flawed. If the chart spec is invalid, you know the problem is in Step 2. It makes the system transparent and much, much easier to debug.

A Concrete Example: From Raw Text to a Vega-Lite JSON Spec

Let’s make this real. Imagine you have this raw data about website traffic for the first quarter of 2026:

Month, Visitors, Signups
January, 15000, 450
February, 18000, 600
March, 22000, 710

You’d use an automation tool like Pipedream or Make.com to orchestrate the two LLM calls. Pipedream’s free tier is generous enough for this kind of internal tool, but for anything client-facing, the $29/mo plan is a no-brainer for the increased execution limits and higher performance.

Step 1: The Analysis Prompt

Your first call to the Claude 3 Opus API would use a prompt like this:

You are a data analyst. Your task is to analyze the provided dataset and recommend the best chart type for visualization. Analyze the columns, data types, and relationships. Output your findings as a single JSON object with two keys: 'analysis' (a brief explanation of your reasoning) and 'chart_type' (one of: 'bar', 'line', 'pie', 'scatter').

Data:
---
Month, Visitors, Signups
January, 15000, 450
February, 18000, 600
March, 22000, 710

The LLM would return something like:

{"analysis": "The data shows two quantitative metrics (Visitors, Signups) over a temporal x-axis (Month). A bar chart is ideal for comparing the discrete monthly totals.", "chart_type": "bar"}

Step 2: The Generation Prompt

Now, the second API call takes that output and combines it with the original data:

You are a Vega-Lite expert. Based on the provided data and the analyst's recommendation, generate a complete and valid Vega-Lite JSON specification for a bar chart. The x-axis should be the 'Month' and there should be two bars per month for 'Visitors' and 'Signups'.

Data:
---
Month, Visitors, Signups
January, 15000, 450
February, 18000, 600
March, 22000, 710

The final output from this step is the clean JSON spec you can feed directly to a renderer:

{
"$schema": "https://vega.github.io/schema/vega-lite/v5.json",
"data": {
"values": [
{"Month": "January", "Metric": "Visitors", "Value": 15000},
{"Month": "January", "Metric": "Signups", "Value": 450},
{"Month": "February", "Metric": "Visitors", "Value": 18000},
{"Month": "February", "Metric": "Signups", "Value": 600},
{"Month": "March", "Metric": "Visitors", "Value": 22000},
{"Month": "March", "Metric": "Signups", "Value": 710}
]
},
"transform": [{
"fold": ["Visitors", "Signups"],
"as": ["Metric", "Value"]
}],
"mark": "bar",
"encoding": {
"x": {"field": "Month", "type": "ordinal"},
"y": {"field": "Value", "type": "quantitative"},
"xOffset": {"field": "Metric"},
"color": {"field": "Metric"}
}
}

This two-step process turns a messy, unreliable art into a predictable engineering problem.

Is the AI’s Chart Choice Always Correct?

No. And you shouldn’t expect it to be. The AI is a powerful analyst, but it lacks context. It might see ten data points and suggest a pie chart, which is almost always a terrible idea. It might fail to recognize that a dataset needs to be normalized before being plotted on a dual-axis chart.

This is where you, the operator, come in. For an internal tool, a bad chart choice is a minor annoyance; you just tweak the prompt and rerun it. But for a production system, you need guardrails. One effective pattern is to add a human-in-the-loop step. Instead of asking the AI for one chart type, ask it for its top three recommendations with justifications. Then, present those three options to a user to make the final choice.

What I really love about this approach is how auditable it is. Because the Vega-Lite spec is just structured JSON, you can programmatically inspect it before rendering. For example, I have a simple Python script that checks the generated spec. If it sees "mark": "pie" and the data has more than six categories, it flags the chart for manual review. You can’t do that when you’re just asking an AI to “draw a picture.” This is the difference between a demo and a real AI automation blueprint.

How to Debug When This Breaks

Even with a solid two-step process, things will go wrong. Debugging an LLM-based workflow means understanding the distinct points of failure. It’s almost never the code; it’s the data or the prompt.

First, check the input. Is the CSV malformed? Is the text data garbled? An LLM can sometimes fix messy data, but it’s not a data-cleaning tool. Validate your inputs before you ever send them to the API. Garbage in, garbage out.

Second, look at the output of the analysis step (Step 1). If it’s suggesting a scatter plot for time-series data, your analysis prompt isn’t specific enough. You can improve it with few-shot prompting. Give it a few examples of good and bad analysis directly in the prompt to guide its reasoning. For example: `Here is an example of good analysis: [example]. Here is an example of bad analysis: [example]. Now, analyze the following data.`

Third, and most commonly, the LLM will generate invalid JSON in the generation step (Step 2). It might miss a comma, forget a closing bracket, or hallucinate a field. The fix here is a self-correction loop. After the LLM generates the Vega-Lite spec, your code should immediately try to parse it. If it throws a JSON parsing error, you catch that error and automatically call the LLM again. The prompt for this second call is simple: `The following JSON is invalid and produced this error: [insert parsing error here]. Fix the JSON. Only output the corrected JSON. Do not add any commentary.`

Honestly, any production LLM workflow that generates structured data without a self-correction loop is just a toy. It will break, and you’ll be left wondering why.

Building this system requires thinking like an operator, not just a prompter. You have to anticipate failure modes and build small, automated loops to handle them. The goal isn’t a perfect prompt; it’s a resilient system.

If you want the deep cut on this, deeper coverage of AI agent platforms.

You can build this yourself following the steps above. It’s a solid weekend project that teaches you the fundamentals of creating reliable, multi-step AI agents. If you’d rather skip the build and deploy a working version in an afternoon, we’ve packaged this entire workflow as a deployable AI automation blueprint. You can find it at deepusecase.com/vault.