Open Source AI Agents

Last month I needed to pull a list of LinkedIn profiles, enrich each with company size and tech stack, then fire off personalized cold emails—all without hiring a virtual assistant. I tried a few no‑code wrappers, but they kept breaking when the API returned unexpected JSON. After a day of frustration I turned to an open source AI agent framework and wired together a scraper, a data enricher, and an email composer in under four hours.

What you’ll walk away with is a working agent that can run on a cheap VPS, repeat the workflow on a schedule, and cost you only the price of the API calls you actually make. No fluff, just the pieces that matter.

It worked on the first try.

What most guides get wrong about agent memory

Most tutorials treat memory as an afterthought. They show you how to call a model once, then pretend the conversation never happened. In reality, an agent that forgets the previous step will keep re‑asking the same question or lose the context needed to format an email correctly.

What you actually need is a short‑term store that lives for the lifetime of a run and a long‑term store that persists across runs. The long‑term store can be as simple as a JSON file on disk; the short‑term store is just a list of messages you pass to each model call.

If you skip this, you’ll see the agent loop: it scrapes a profile, tries to enrich, fails to find the company size, then starts over because the prompt still says “find the company size”.

How do you keep the agent from looping forever?

Loops happen when the agent has no clear exit condition. You must give it a way to detect success and a way to detect failure after a limited number of retries.

One pattern that works well is to add a “step counter” to the state. Each time the agent finishes a subtask, increment the counter. If the counter exceeds a preset limit—say five steps for a single lead—force the agent to output a failure flag and move on to the next lead.

Another guardrail is to ask the model to explicitly state whether it succeeded. Prompt it with: “Reply with JSON that contains {“done”: true, “data”: …} or {“done”: false, “reason”: …}”. Then your code can check the “done” field before proceeding.

These two tricks cut the runaway loops I saw in early prototypes from dozens of iterations to a clean finish.

A concrete named example: scraping LinkedIn with crewAI

Here’s the exact flow I used, with the tools and costs I actually paid.

1. Install crewAI (first mention, so bold) and its dependencies: pip install crewai.
2. Create a folder called lead_agent and add a file config.yaml that defines three roles: scraper, enricher, Writer.com.
3. Scraper role prompt: “You are a LinkedIn scraper. Given a search URL, return a JSON array of profile objects with fields: name, headline, profile_url.”
4. Enricher role prompt: “You are a data enricher. Take a profile object, call the Clearbit API (or any free alternative) to add company_size and tech_stack. Return the enriched object.”
5. Writer role prompt: “You are a cold‑email writer. Using the enriched profile, produce a short, personalized email that mentions one detail from the tech stack.”
6. Define a workflow that runs the scraper, then for each profile runs the enricher, then the writer, and finally writes the output to a CSV.
7. Set the step counter to 5 and the “done” check as described above.
8. Run the agent with: python run_agent.py --search-url "https://www.linkedin.com/search/results/people/?keywords=founder".

Cost breakdown: The LinkedIn scrape uses the free public search (no API key). Clearbit’s free tier gives 100 requests per month; I stayed under that. The only paid piece was the LLM calls. Using GPT‑4o via the OpenAI API at $0.06 per 1k tokens, each lead cost about $0.008 (roughly 130 tokens for the three calls). For a batch of 500 leads the total came to $4.00.

I think $0.008 per lead is a fair price for the automation you get—especially when a virtual assistant would charge $3‑$5 per lead for the same work.

How to debug when this breaks

First, check the logs. crewAI prints each role’s input and output to stdout. If you see a blank array from the scraper, the LinkedIn page likely changed its HTML structure; you’ll need to update the selector or switch to a different data source.

Second, watch for token‑limit errors. The model will return a 429 or a message about max tokens if your prompt plus the retrieved data exceeds the context window. Trim the data you pass—send only the fields you need, not the whole raw JSON.

Third, verify the JSON parsing step. A missing comma or trailing comma in the model’s output will crash your parser. I wrapped the parsing in a try/except and logged the raw response whenever it failed; that saved me hours of guesswork.

Finally, if the agent seems to hang, look at the step counter. A bug that never increments the counter will cause an infinite loop. Adding a simple print statement after each role helped me spot where the counter got stuck.

(Which, yes, is annoying—but it’s the kind of detail you only learn after you’ve seen it break a few times.)

Price check: is the free tier enough?

If you’re just testing, the free tiers of OpenAI (limited credits) and Clearbit (100 requests) will let you run a few dozen leads without spending a cent. For regular use, you’ll need to pay for the LLM API; the cost scales linearly with volume.

I’ve found that $19/mo for a modest usage of GPT‑4o (around 300k tokens) covers a few thousand leads per month, which is more than enough for a solo operator or a small agency. Anything beyond that and you start paying per‑token, which still feels reasonable compared to hiring a human.

Honestly, the free plan is a joke if you expect to run more than a handful of leads per day—you’ll hit the rate limits fast and waste time troubleshooting.

We cover this in more depth elsewhere — deeper coverage of AI agent platforms.

If you’d rather skip the build and deploy a working version in an afternoon, we’ve packaged this workflow as a blueprint at deepusecase.com/vault/ai-agent-builder-kit.