A supplier agent that more suppliers answer
We rebuilt the agent that asks suppliers for quotes at Sourcy, a B2B sourcing marketplace. It's in production and still running.
R&D Lead at Sourcy. Agent design, implementation and evaluation.
- 73%
- of suppliers replied, across 169 live conversations
- 10–20%
- of chats ran past 40 messages
- ~5x
- lower monthly model cost than the previous bot
Suppliers who replied
How we measured it: counted from live production conversations and compared with the previous bot's own record, where about half of suppliers went silent. It's a before and after comparison, not a controlled test. What the numbers don't say covers what that means.
The problem
A buyer's sourcing request ends in a quotation. Before that, someone has to ask suppliers about price, lead time and material, and get real answers back. An agent does the asking, one conversation per supplier. When those conversations stall, the quotation waits.
The previous bot stalled a lot. About half of the suppliers it wrote to went silent. When a chat went wrong you couldn't really see why. What was it even trying to get? Every fix was a guess, and fixing one conversation could quietly break another.
Around then a buyer asked for two products. Somewhere along the way they got merged into one, and the bot asked a hiking-shoe maker about backpacks. It was a game of telephone. I'd already decided to rebuild by then, and this one made it easy to explain why.
What human buyers do differently
Before writing any code, we read. We pulled about 4,000 supplier conversations, around 1,000 of them written by human buyers. The bot's logs showed where it failed. The human threads showed what a good buyer does at the same moment.
Agents grouped them into patterns. I didn't trust that much on its own, because a group only tells you something keeps happening. So we read the conversations in each group ourselves to work out why.
Buyers write "how much". The bot wrote paragraphs. A long, careful message from a stranger doesn't read like someone about to place an order.
Price and quantity were another one. A supplier said "at 2,000 I can do 65." The bot came back with "can you do 500?" Any human would get it. 65 was the price at 2,000, so there's no way it holds at 500. You ask about the two together.
Then there's what the supplier makes of you. A bot that asks vague, open questions reads like someone who's just looking, and suppliers don't spend much time on that. I think the agent needs a sense of what it wants. Otherwise why would they bother?
We wrote each repeated failure down as a mental model: the situation, what goes wrong, and what a good buyer does instead. We ended up with about 50 of them.
Evals over rules
Before the rebuild we tried putting the product team's whole rulebook into the prompt. The agent got worse. It followed some rules and dropped others, and you couldn't tell which rule caused what.
So most of the playbook went into the evals instead. Each mental model became a case we replay against the agent. The prompt keeps the few rules that matter most, and the evals check the rest still holds after every change.
The evals also have to agree with a human about what good looks like. Early on, a supplier told the first bot "ready stock, no minimum order". That's the best answer a buyer can get. The bot scored it as a failure, because it never got the minimum order number it was asking for. A human buyer would have been pleased. So the problem wasn't the chat. It was how we were scoring it. That's when I knew we needed proper evals.
Traces go with the evals. Every turn records why the agent sent what it sent, so when a case fails I can open the trace and see what it was thinking.
What we rebuilt
The new agent works from explicit goals, so each conversation knows which facts it still needs and has a reason for its next message. It writes every reply with the whole thread in view. When it hits something it shouldn't decide alone, it hands the conversation to a person.
The old bot only moved when the supplier wrote something. I asked what happens if they never write back at all. A human buyer would nudge them after a while, so now the agent does too.
Simulation
Turn 0 of 5. 0 of 4 conversations finished.
Supplier A
Not started- Price: ?
- Lead time: ?
- Material: ?
Supplier B
Not started- Price: ?
- Lead time: ?
- Material: ?
Supplier C
Not started- Price: ?
- Lead time: ?
- Material: ?
Supplier D
Not started- Price: ?
- Lead time: ?
- Material: ?
Going live
Close to launch, the agent handled the easy path and not much else. Each feature we added made it a little worse. "My bot just does the happy flow" was how I put it at the time. So we cut scope to what it could do reliably and shipped that.
We also skipped the 50/50 split with the old bot and switched fully. My view was that you can't find these edge cases in simulation. Suppliers are really unfiltered, they'll just say things. So I wanted all the traffic on the new one.
It's in production and still running. The numbers are at the top. Long chats are a good sign here, because suppliers don't stay that long once they've lost interest.
Longer chats brought new problems. Suppliers who keep talking ask harder questions and send things nobody planned for. Most new failures became eval cases, so fixing one didn't quietly break another.
Some of those failures were rough. One supplier sent a price split across three messages, and the agent read the pieces wrong. The supplier wrote back: "are you stupid or are you a robot?" Around then someone called the bot close to done. My answer was that at very best it's like a one-week intern. So the plan was to spot the questions a one-week intern couldn't answer and hand those to a person.
One supplier sent a file the agent couldn't open. Nobody had told it what to do in that case. It asked for photos instead, which is what a human buyer would do. I didn't expect that.
Longer conversations also made an old risk bigger. One flagged sending account can freeze every conversation on it, and the deeper the chats, the more there is to lose.
When a supplier says no
A supplier would say "we can't serve you, find another supplier," and the old bot kept asking about fabric. A human buyer would just say thanks and move on. Now the agent does that.
What the numbers don't say
- This was not a controlled A/B test. The old and new agents ran at different times with different suppliers, so some of the change could come from that.
- A reply is not a good quote. The response rate counts suppliers who answered, not whether the answer led to a usable quotation.
- Some conversations still go to a person. That's on purpose. Some questions need a person.
What they do show: more suppliers answer, and the model bill is about 5x smaller.
Related writing
Need something like this built?
Pick a time and tell me what you're building.