It’s 10 AM on Monday. Michelle (I hope you remember her from here) has a weekly sync-up with her CMO. She needs to come up with an answer for:
“What changes did we make on Meta last week and what was its effect?”
Sounds easy, right?
When she asked the same question on GetCrux 6 months ago, the system went haywire.
What looked like a simple data fetch, brought out an entirely new aspect of AI agents.
What happens behind the scenes of AI agents?
Here’s what a typical process would look like:
Step 1: Decide what data to fetch based on the question + relevant context + what is available
Step 2: Use RAG to gather relevant data and make one or multiple LLM calls to answer the question
Step 3: Synthesize and display the output to the user
But what happens with the data to be fetched is too large? I’m talking GBs.
The system falls apart!
Either it starts:
1. Missing out relevant data and/or
2. Picking up irrelevant data
After breaking our heads at it for days and trying countless things, we realised:
The answer was surprisingly simple: Only fetch the data you REALLY need. But how do you do that?
The Multi-Agent Framework:
The task:
Give our RAG a layer of intelligence with which it can understand and retrieve the relevant information in context of the question asked in real time.
We went ahead and designed a simultaneous multi-agentic structure of AI agents which would run in the background breaking down the process step by step where each agent handled the end-to-end completion of the task in the process.
Below are the agents that handled each task of the process at once to deliver the context of data retrieval in real time.
1. Context Agent: The only work of this agent was to understand the question completely without further input from the user.some text.
- Intent classification - First step is to understand the question and classify intent based on the team’s objectives & constraints.
- Data requirement identification - Next step was to make an exhausted list of data requirements for catering the intent and answer the question.
- Priority assignment - Amongst those listed, the next step for the agent was to prioritise the requirements for the next agent to do its job.
2. Performance Metrics Agent: This agent takes the requirements from the Context Agent and finds the relevant data from the database.
- Source mapping - The agent starts with finding the location of data in the database and maps to the list. Consider getting a list of people and writing their address in front of them from the address book.
- Access optimization - The agent then finds the best way to access the data effectively. Similar to making a route so that all addresses come on that route saving time for revisiting the already visited colony.
- Real-time validation - Validation for correct data is done as the last process.
3. Action Classification Agent: The agent that actually executes everything.
- Fetch orchestration - All data from the mapped source is then fetched.
- Data validation
- Response compilation - And then compile all the data after validation and get it ready to send it to LLM along with the question and context for output.
How this works in real-time:
Going back to the question Michelle asked GetCrux:
“What changes did we make on Meta last week and what was its effect?”,
Let’s break down what actually happened behind the scenes.
When the question got asked, the agents started working simultaneously -
Context Agent- figures out the right context required to come up with a response based on the team’s objectives & constraints.
For example: To know the effects of change in the campaigns, following are the relevant data points -
1. Last week’s action history
2. List of ads launched
3. Performance metrics
4. Revenue data from Shopify/Salesforce
Performance Metrics Agent - figures out which metrics are needed to answer the question, the period in the discussion, and the level i.e. campaign, Ad set, & Ad.
For example: Following metrics might be relevant to the query -
Performance Metrics:
- Click-through rate (CTR)
- Conversion rate
- Cost per click (CPC)
- Cost per acquisition (CPA)
- Impression share
- Engagement rate
Revenue Metrics:
- Revenue per ad
- Return on ad spend (ROAS)
- Average order value
- Total revenue attributed to campaigns
Action Classification Agent - Classifies all actions taken into various types (ads introduced, budget changed, campaigns created, etc.) and selects the one we are interested in as per the query deciphered by the previous agents.
How the future of AI Agents is evolving:
We still have a long way to go in terms of making the AI agents reliable and robust.
Here are some of the challenges we are working on:
1. More Complicated Data Sources: Videos are already the most consumed format of content and with upcoming text-to-video models, this trend will only continue to grow. The question for us is - how do we keep up with this speed and make it easier to analyse videos on-the-go.
2. Smarter Decision Frameworks: Real-life decisions can hardly be captured in rule-based mechanisms. We experimented with Keyword Analysis, where a lot of human-biased decisions are made due to lack of context despite great tools being out there, and we were able to identify $19.8M+ worth of potential revenue through the exercise.
But the question is still unanswered - Can the AI agents work with ambiguity and new situations with the same level of reasoning and rigour?
While we keep experimenting with a lot of ideas, if you think a lot about AI’s application in your organisation, we’d like to hear from you.
