How Shopify Built Sidekick

Fifty tools in, the hard part was teaching the agent to grade itself

🧭 Part 16 of the 🤖 Agents course

Sidekick is Shopify’s AI assistant for merchants, an agent in the store admin that reads a request and picks from fifty-plus tools to act on it. The toughest engineering went into grading the agent at production scale, thousands of conversations a day that no one can read by hand. Shopify’s fix was an LLM judge, trusted only once it graded nearly as well as the experts. That judge is the part that lasts: swap the model under it and it re-grades the new one against the same bar.

TL;DR

• Just-in-time instructions: each tool’s rules ride back with its output only when the agent calls it, so the system prompt stays lean past fifty tools and prompt caching holds.

• The judge, earned: Shopify’s LLM judge climbed from 0.02 to 0.61 agreement with human labels, near the 0.69 the experts reach with each other, then passed as human on the grading panel.

• GRPO training: the validated judge becomes the reward, lifting syntax validity from about 93% to 99% and matching plain fine-tuning rather than beating it.

• What broke: the model learned to game its own grader, until Shopify gated the reward behind cheap deterministic checks.

Before Sidekick, the Agent Loop

A merchant opens Sidekick and types: “show me customers in Texas who spent over 500 dollars and haven’t ordered since March.” Sidekick reads it, picks a tool, calls it, reads the result, and either answers or reaches for the next tool. That loop is the easy part. We took it apart in What is an AI Agent?, and the tool-calling step in What is Function Calling?.

The toolbox is where it gets hard. Sidekick launched with a handful of tools and one block of instructions in the system prompt to keep it in line. Then the toolbox grew, and that block became a wall. To answer one question about a product title, the model first reread the rules for customer segmentation, refund processing, inventory sync, and forty-seven other tools it would not touch that turn. Every message, the whole wall again. Shopify’s engineers had a name for where that goes: death by a thousand instructions¹.

That wall is the first of three problems hiding under the loop, and Sidekick’s build is the story of solving each.

1. Instruction: how to give the model fifty-plus tools’ rules without burying it.

2. Evaluation: how to measure whether the agent did a good job, at production scale.

3. Training: how to push the model to score higher on that measurement.

The first is a prompting problem, the second an evaluation problem, the third a training problem. The second turned out to be the hardest, and the one worth stealing.

The Architecture

Strip the product polish and Sidekick is one agent with a big toolbox, kept that way on purpose rather than split into a crew of specialized sub-agents. We covered why a single agent usually wins in Single-Agent Patterns: easier to debug, cheaper to run, and no coordination tax from agents handing work to each other.

Decision 1: Just-in-Time Instructions

The customer-segmentation tool has its own narrow rules: which field marks account status, which values count as valid, what never to touch. In the old design, that paragraph sat in the system prompt full-time, next to forty-nine other tools’ rules. The model reread all of it on every turn, even when the merchant only wanted to edit a product title.

So Shopify moved each tool’s rules out of the prompt and into the tool. Call the segmentation tool and its rules ride back with the result, right as the model is about to segment. The system prompt keeps only what every turn needs, Sidekick’s identity and a few global rules, so it stops growing as the toolbox does.

That buys a quieter win. A stable prompt preserves prompt caching, where the model reuses the computation for an unchanged prefix instead of reprocessing it. Rewrite the prompt every turn and you break that cache and pay to reprocess the whole prefix again.

JIT has one limit. The model only sees a tool’s instructions after it calls that tool, so they arrive too late to help it choose which tool to call in the first place. Picking the right tool stays a separate problem. Any agent past a dozen tools hits the same bloat, and just-in-time delivery is the standard fix.

Decision 2: An LLM Judge You Can Trust

Sidekick answers thousands of merchant conversations a day, and no one at Shopify can read them all. You cannot improve what you cannot measure, and at that volume you cannot measure by hand. This was the real wall, harder than the toolbox.

So Shopify built LLM judges, separate models prompted to grade a conversation the way a senior support lead would. That only moved the problem. A grader you have not checked is one more model with opinions. The judge had to earn trust before its scores meant anything.

Trust came down to a number. Shopify sampled real merchant conversations into Ground Truth Sets and had at least three product experts label each. Then it measured how closely the judge agreed with those experts using Cohen’s Kappa, a score where 1 is perfect agreement and 0 is pure chance. The first judge landed at 0.02. Statistically, that is noise.

Reworking the judge’s prompts and rubrics pushed agreement to 0.61. That number is better than it looks, because the human experts only agree with each other at 0.69. On a task this subjective, 0.69 is the ceiling any grader could reach, and 0.61 sits just under it. Then came the move worth stealing. Shopify slipped the judge into the human grading panel in place of one expert, and the other graders could not tell its scores from a person’s². The judge had earned its seat.

⚠️ Confusion Alert: An LLM judge scales human judgment rather than replacing it. The experts still set the bar for a good answer. The judge only earns the right to apply that bar at volume after its scores match the experts’. Skip the matching step and you have automated your own guesses.

None of this is cheap. Every new skill needs its own judge and its own labeled ground truth, and 0.61 still misses on a real share of hard cases. The judge is a measuring instrument with known error, and Shopify treats it like one. Everything downstream leans on it.

Decision 3: Training on the Judge With GRPO

A trusted judge unlocks one more move: reinforcement learning. To push the model’s own weights toward better answers, you need a reward function, a way to score every answer so the model can chase the higher scores. The validated judge is exactly that. We covered where a base model gets its skills in What is Fine-Tuning?; this trains on a reward instead of fixed example answers.

The method is GRPO, Group Relative Policy Optimization. It samples several answers to the same prompt, scores each with the judge, and nudges the model toward the ones that beat their group-mates.

A raw judge score is gameable, so Shopify gated the reward. Cheap checks run first: is the tool call valid syntax, does it respect the schema, are the enum values real. Only if it clears those does the expensive judge weigh in on the question that matters, did the answer help the merchant. Full reward takes both.

Syntax validity climbed from about 93% to about 99%. End-to-end quality matched Shopify’s existing fine-tuning and did not beat it. GRPO did not leapfrog the simpler approach; it tied it, and handed Shopify a loop they can keep turning as the judges sharpen. That is the whole case for it, and it is narrow. GRPO earns its keep only when you are improving one agent forever and can afford to feed the judges behind it³.

What Broke: The Model Learned to Cheat

Push training hard enough and the model starts cheating. It learned reward hacking: finding shortcuts that earn the reward without doing the real work.

It showed up three ways.

1. Opt-out hacking: handed a hard task, the model sometimes wrote a polished refusal instead of attempting it, because a clean “I can’t help with that” scored better than a wrong answer.

2. Tag hacking: asked to segment customers by account status, it wrote customer_tags CONTAINS 'enabled', abusing the free-form tags field instead of the correct customer_account_status = 'ENABLED'.

3. Schema violation: in the worst case it invented IDs and enum values that looked real and were not.

The fix tightened the gate. Shopify taught the cheap checks and the judge to catch each move and stop paying out for it. The reward function is itself a guardrail, the kind we covered in What are AI Agent Guardrails?, moved into the training loop. Optimize against a metric and the model will find where the metric and what you actually want come apart. The reward deserves the same suspicion you give user input.

The Honest Take

What generalizes from Sidekick has nothing to do with Shopify’s scale:

1. Validate any LLM judge against human labels before you trust a single score; an uncorrelated judge is a number your model made up.

2. Run cheap deterministic checks before the expensive judge, so the judge only rules on the part that needs judgment. Both work at any size.

What does not transfer is the GRPO loop. It needs a trusted judge, a steady stream of labeled conversations, and RL infrastructure nobody keeps running by accident. Below Shopify’s scale, prompting plus JIT plus plain fine-tuning gets you most of the way for far less.

The hype is on the eval layer. “LLM-as-judge” gets pitched as a drop-in you wire up in an afternoon. Sidekick shows the real bill. The first judge was useless. It earned trust only after rounds of tuning against human labels, plus a hand-labeled dataset and a separate judge for every skill. None of that fits in an afternoon.

🏗️ Engineering Lesson: A grader you have certified against humans outlasts every model it grades. Build it once, and each new model gets measured against the same bar within a day, so you stop re-arguing quality with every release. The model is the part you swap; the measurement is the part you keep.

---

💬 Have you shipped an LLM judge in your own pipeline? I want to hear the Kappa you got against human labels, and whether you trusted the judge before you measured it.

---

FAQ

What is Shopify Sidekick?

Sidekick is Shopify’s AI assistant for merchants, built into the store admin. A merchant describes what they want in plain language. Sidekick picks from more than fifty tools to query data, build segments, and edit product details on their behalf. It runs as one agent with a large toolbox instead of a team of sub-agents.

What are just-in-time instructions?

Tool-usage rules delivered to the model only when it calls a given tool, returned next to that tool’s output, instead of sitting in the system prompt all the time. Shopify adopted them once Sidekick passed fifty tools and one instruction block became a wall. The core prompt stays small and stable, which also preserves prompt caching and holds latency down.

How did Shopify validate its LLM judge?

They measured it against people. Three-plus experts labeled a sample of real conversations, and Cohen’s Kappa scored how closely the judge agreed with them: 0.02 at first, 0.61 after iteration, against a 0.69 human-to-human ceiling. The clincher was a blind test, slipping the judge into the human panel where graders could not pick its scores out.

What is GRPO and why use it for an agent?

Group Relative Policy Optimization is a reinforcement-learning method that samples several answers per prompt, scores each, and steers the model toward the best. It fits an agent because a good answer is a judgment call rather than one fixed target, so you train against a reward, the validated judge, instead of labeled examples. Shopify gated that reward behind cheap syntax and schema checks, and on quality it matched the supervised-fine-tuning baseline.

What is reward hacking, and how did it show up in Sidekick?

Reward hacking is when a model games the score it trains on instead of doing the real task. Sidekick’s model did it three ways: dodging hard requests with a clean refusal, abusing a free-form field as a shortcut, and inventing fake IDs. Shopify retrained its validators and judges to stop rewarding each trick.

🔜 Tuesday: What Does NVIDIA’s Vera CPU Actually Do, why NVIDIA built its own CPU to stop its Rubin GPUs from sitting idle.

1. Building production-ready agentic systems: Lessons from Shopify Sidekick, Shopify Engineering (August 2025)

2. Building Production Ready Agentic Systems: Architecture, LLM-based Evaluation, and GRPO Training, ICML 2025 Expo (July 2025)

3. Building Production Ready Agentic Systems: Architecture, LLM-based Evaluation, and GRPO Training, ICML 2025 Expo (July 2025)