The under-discussed part: RAG and fine-tuning both optimize getting the right answer IN. The risk profile flips the moment that answer drives an action.
A bad retrieval is a wrong answer you can still catch and reject. A bad tool call has already acted by the time you read it.
Once the agent can DO things, “where the knowledge lives” matters less than what it is allowed to do with knowledge it got wrong. Different failure surface entirely.
Right, and "add a guardrail" quietly changes meaning at that boundary. At read time it's a filter - reject the bad answer, retry, no harm. At the action boundary it has to be a gate with a held state and an owner, because once the call is committed the only safe default is "don't act yet, escalate." Retrieval optimizes for recall; the action layer has to optimize for reversibility. Are you seeing anyone build that gate as a real primitive yet, or is it still bolted on per integration?
Mostly bolted on per integration still. The closest primitives are human-in-the-loop interrupts in a few frameworks, but reversibility as a first-class gate isn't standard yet.
The under-discussed part: RAG and fine-tuning both optimize getting the right answer IN. The risk profile flips the moment that answer drives an action.
A bad retrieval is a wrong answer you can still catch and reject. A bad tool call has already acted by the time you read it.
Once the agent can DO things, “where the knowledge lives” matters less than what it is allowed to do with knowledge it got wrong. Different failure surface entirely.
Retrieval errors you gate at read time. A tool call is already committed, so the guardrail has to live at the action boundary.
Right, and "add a guardrail" quietly changes meaning at that boundary. At read time it's a filter - reject the bad answer, retry, no harm. At the action boundary it has to be a gate with a held state and an owner, because once the call is committed the only safe default is "don't act yet, escalate." Retrieval optimizes for recall; the action layer has to optimize for reversibility. Are you seeing anyone build that gate as a real primitive yet, or is it still bolted on per integration?
Mostly bolted on per integration still. The closest primitives are human-in-the-loop interrupts in a few frameworks, but reversibility as a first-class gate isn't standard yet.