Building a Real Relevance AI System That Doesn't Break

What Is Relevance AI, Really?

You have data flooding in. Leads from a contact form, support tickets, social media mentions, scraped content. You need to know what’s important now and what’s just noise. A relevance ai system is supposed to solve this by automatically sorting, tagging, and routing that incoming stream. By the end of this article, you’ll know how to build a real one—not a toy wrapper around a single API call that falls over with the first weird input.

This isn’t a theoretical exercise. I run multiple businesses on top of these kinds of automations. They handle thousands of events a day. What I’m showing you is the architecture that actually works in production, warts and all.

What Most Guides Get Wrong

Almost every tutorial I’ve seen on this topic is dangerously simplistic. They tell you to pipe your incoming data directly into a large language model like GPT-4 with a prompt that says something like, “Is this lead a good fit for my business? Answer yes or no.”

This is a terrible idea for three reasons.

First, it’s incredibly slow. Sending every single inbound item to a flagship model creates a huge bottleneck. You’ll be waiting seconds for a response when you should be getting one in milliseconds. Second, it’s expensive. GPT-4o, while cheaper than its predecessor, still costs money for every single token. Processing 10,000 low-value support tickets a month could run you a noticeable bill for no real gain. Third, it’s brittle. A single, complex prompt is a nightmare to debug. When it gives a weird answer, you have no idea which part of its reasoning failed. It’s a black box, and you can’t build a reliable business process on a black box.

The correct approach is a multi-stage pipeline. It’s more work to set up, but it’s cheaper, faster, and infinitely more reliable. It’s the difference between a hobby project and a real piece of infrastructure.

The Core Stack: Vectors, Agents, and Rules

A production-ready relevance system has three main components that work in sequence. Think of it as a series of filters, each one smarter and more expensive than the last.

• 1. Vector Search (The Fast, Cheap Filter): This is your first line of defense. Instead of asking an LLM about every single item, you first check for semantic similarity. You create vector embeddings—numerical representations of your ideal data points (e.g., descriptions of a perfect customer). When new data arrives, you embed it and use a vector database to find the closest matches. This is blazing fast and costs fractions of a penny per query. I use this to instantly discard 80% of irrelevant inputs.
• 2. LLM Agent (The Smart, Expensive Filter): Only the data that passes the vector search filter gets sent to an LLM. And you don’t send it with a generic question. You send it with the original data *plus* the context from the vector search result. The LLM’s only job is to make a final, nuanced judgment based on this pre-filtered, high-context information. This is where you use a model like Claude 3 Haiku or GPT-4o.
• 3. Rule-Based Fallbacks (The Safety Net): What happens if an API is down or the LLM returns gibberish? You need a dead-simple set of `if/then` rules. For example: `if message contains ‘urgent’ or ‘down’, tag as P1 incident`. This safety net catches failures and handles obvious cases without needing any AI at all. Never trust an AI system without a non-AI backup.

This tiered structure ensures you’re only using your most powerful (and expensive) tools on the data that actually matters.

A Concrete Example: Building a Lead Qualification Router

Let’s make this real. Imagine you get 100 leads a day from your website. You need to separate the high-intent demo requests from students doing research. Here’s how you build a router using the stack above.

Step 1: Define and Vectorize Your Ideal Customer Profile (ICP)

First, forget AI. Write 3-5 paragraph-long descriptions of your absolute best customers. Be specific.

Example ICP: “A B2B SaaS company in the marketing automation space with 50-250 employees. They just raised a Series A or B round. Their Head of Growth is feeling pain around lead attribution and probably uses HubSpot. They explicitly mention wanting to ‘improve ROI’ or ‘get better analytics’.”

You’ll use an embedding API, like OpenAI’s `text-embedding-3-small`, to turn each of these descriptions into a vector. Then you store these vectors in a database. For starting out, a service like Pinecone is great. Its free tier is enough to get started, though I find the developer experience a bit clunky. Honestly, for solo work, the $25/month starter plan is more than enough and saves you a lot of headaches.

Step 2: The Inbound Workflow

Now, set up a workflow that triggers every time a new lead comes in. I use the Make platform for this, but Zapier automations or Pipedream works too.

1. New Lead Received: The webhook fires.
2. Embed the Message: Take the text from the lead’s “message” field and send it to the same embedding API you used for your ICPs.
3. Query Pinecone: Search your ICP vector index with the new lead’s vector. Ask for the top 1 match.
4. Check the Score: Pinecone will return the closest ICP and a similarity score (from 0 to 1). If the score is below a threshold you set, say 0.8, you know it’s not a close match. You can automatically tag it as “Low-Intent” and stop. The process for that lead ends here. Fast and cheap.

Step 3: The LLM Agent Prompt

If the similarity score is *above* 0.8, it’s promising. Now, and only now, do you spend money on a proper LLM. You’ll make an API call to a model like Claude 3 Haiku (my preference for these tasks because it’s fast and cheap) with a carefully structured prompt. It’s crucial to give it both the lead’s data and the ICP it matched with.

This prompt is specific. It tells the AI its role, the exact format for the output, and gives it all the necessary context to make an intelligent decision.

Step 4: Route Based on a Clean Output

The AI will return a clean JSON object like `{“is_hot_lead”: true, “reasoning”: “Lead is from a 100-person SaaS company and mentioned improving ROI, which directly matches the ICP.”, “suggested_next_step”: “Send to senior account executive.”}`. Your automation platform can now easily parse this JSON and route the lead to the right person in Slack or your CRM.

What’s the Real Cost to Run This Thing?

This is way cheaper than you think.

• Embeddings: OpenAI’s `text-embedding-3-small` model costs $0.02 per million tokens. That’s effectively free for this use case. You could process 10,000 leads and it would cost you pennies.
• Vector Database: Pinecone’s starter plan is about $25/month. You can find other options, but this is a fair price for a managed service that you don’t have to worry about.
• LLM Agent: This is the main cost. But because we’re pre-filtering, we’re not running it on every lead. Let’s say only 20% of your leads are promising enough to reach this step. If you use Anthropic’s Claude 3 Haiku, it costs $0.25 per million input tokens. For 20 leads a day (600 a month), each with a 1,000 token prompt, you’re looking at a monthly cost of… maybe $0.15. It’s ridiculously cheap.

Your total cost for an automated, intelligent lead routing system is less than $30 a month. That’s a no-brainer.

How Do You Debug This When It Misfires?

It will absolutely misfire. Plan for it.

Your first and most important tool is logging. You must log every step: the incoming data, the vector search score, the exact prompt sent to the LLM, and the raw JSON response from the LLM. Without logs, you’re flying blind.

When a lead is misclassified, look at the logs. The problem is usually in one of two places.

First, the vector search. Is the similarity score really low? Maybe your ICP descriptions are too vague or too different from the language your actual leads use. My biggest gripe here is that fine-tuning vector search feels like a dark art. My concrete love, however, is that once you get it dialed in, it’s incredibly satisfying to see 90% of junk leads get filtered out before they ever hit your inbox.

Second, the LLM prompt. If the vector search found a good match but the LLM made the wrong call, copy the exact prompt from your logs and paste it into the Anthropic or OpenAI playground. Tweak the wording. Add an example. Clarify an instruction. When you find a prompt that works for that specific edge case, update it in your automation. This is the feedback loop that makes your system smarter over time.

This is the work. It’s not a one-and-done setup. It’s a system you have to monitor and tune, especially in the first few weeks.

Building this multi-stage relevance AI system is a serious step up from basic automation. It’s how you build a process that is efficient, scalable, and doesn’t cost a fortune to run. You can apply this same pattern to support tickets, content moderation, or any high-volume data stream.

For more on this exact angle, deeper coverage of AI agent platforms.

If you’d rather skip the build and deploy a working version in an afternoon, we’ve packaged this entire workflow as a deployable AI automation blueprint. You can find it at deepusecase.com/vault.