Building an open ai agent builder from scratch sounds like a lot of work, but you can get a functional system running in a day if you know where to focus. I’ve put together a workflow that uses the OpenAI Assistant API, a simple Airtable base for storage, and a Make.com scenario to glue everything together. By the end of this guide you’ll have a working agent that can qualify leads, answer FAQs, and push data to your CRM.
Why does the agent forget context after three turns?
The Assistant API keeps the whole conversation in its context window, which is limited to 8 k tokens by default. Each user message and each model reply adds tokens, and the system also stores any function‑call results. After a few exchanges the window fills and the oldest messages are dropped, making the agent seem to lose memory.
You can see this happen when you ask the agent to recall a detail you shared earlier in the chat. If the detail was in the first user message and you have already sent six or seven replies, the model may no longer see it and will answer “I don’t have that information.”
(which, yes, is annoying) but there is a simple workaround: store the important facts yourself in Airtable or another database and retrieve them with a tool call when needed. That way the agent never relies on the built‑in memory for anything critical.
It’s that simple.
What most guides get wrong about tool integration
Many tutorials treat the Assistant API’s tool calls as a plug‑and‑play feature: you define a function, expose an endpoint, and the model magically uses it. In reality you have to handle three distinct steps correctly.
First, you must return a valid JSON object that matches the schema you declared. If the model expects a string called “status” and you send a boolean, it will ignore the result and ask for clarification.
Second, you need to surface any errors from your endpoint back to the model in the same format it expects. A plain HTTP 500 will not be understood; you must wrap the error in the tool‑response shape.
Third, you must keep the conversation ID stable across turns. If you accidentally create a new thread for each tool call, the model loses the ability to chain multiple steps.
I love how the built‑in retrieval tool lets me upload a PDF and have the agent answer questions without extra code. It saves me from writing a custom embed‑and‑search pipeline for every knowledge base I want to expose.
How to debug when the agent returns empty responses
Empty replies usually mean the model decided not to generate text, often because it hit a tool‑call loop or received a malformed response.
My concrete gripe: the Assistant API returns very little diagnostic information when something goes wrong. You get a generic “max_tokens” or “stop” finish reason, with no hint about which tool call failed or why the model chose to stop.
To troubleshoot, start by logging every request and response from your middleware (Make.com or whatever you use). Check the tool_calls array in the assistant’s response—if it is empty, the model is not trying to use any tool. If it contains entries, verify that each one includes a valid function name and that the arguments parse.
Next, look at the HTTP response from your endpoint. Did it return 200 with a JSON body? Did the body contain the exact keys you declared in the schema? A missing key or a type mismatch will cause the assistant to discard the result and fall back to a plain text answer, which may be empty if you also gave it no instructions for that case.
Finally, reduce the complexity. Strip out all but one tool call and see if the model returns a sensible answer. Add tools back one by one until the problem reappears; that isolates the offending integration.