Principles for a Customer-Facing Agent That Knows Too Much

A lot of the support requests we get could be answered just by looking at our codebase. We had already built an internal tool for this: a support engineer could trigger a coding agent to investigate a ticket against the codebase, and it would come back with an internal note explaining what it found. It was answering a lot of these questions well.

That got us thinking: if we gave this to customers directly, our support team wouldn’t have to context switch into every routine question. They could spend their time on the hard issues instead of troubleshooting things an agent could already answer.

Making it customer-facing puts us in front of a real problem, though: a useful agent will always have access to more data than the user it is talking to. That is the definition of the value it provides. It also creates a confused deputy problem: an agent that knows everything can be manipulated into revealing more than it should. We don’t want our codebase exposed.

We took the whole thing apart and came up with a set of principles to make this setup as secure as it gets.

The Lethal Trifecta

Simon Willison’s lethal trifecta is the clearest framing we found for this. An agent becomes dangerous when three conditions are present simultaneously:

Access to private or sensitive data
Exposure to untrusted content
The ability to communicate externally

Unlike traditional software, an agent operating over broad knowledge cannot be made entirely secure. The goal is not zero risk. The goal is to reduce each leg of the trifecta until the risk is small enough that exploiting it costs more than it’s worth for any attacker.

Every security decision should be evaluated against this question: does this reduce one or more legs of the trifecta without meaningfully degrading the agent’s ability to help?

Principles

Risk cannot be zero, and that is acceptable

Humans with access to sensitive company knowledge also carry risk. They can be phished, manipulated, or act in bad faith. The standard is that the risk is reduced to a level where the cost of exploitation exceeds the benefit, and where the most likely failure modes are contained and visible.

Disable network access

This is the highest-leverage single control available. An agent without external network access cannot exfiltrate the repository, cannot download a malicious dependency that uploads data to a third-party server, and cannot be used as a relay. It removes the third leg of the trifecta almost entirely.

Network access should be off by default. We restrict the agent to only controlled channels to communicate through.

Gate actions, not data

Restricting what data the agent can see makes it less useful. Restricting what actions it can take does not. The agent should have exactly one tool: responding to the customer, scoped to their own environment. It should not be able to take actions outside that scope regardless of what it’s instructed to do. The surface of possible harm from a runaway agent is bounded by what its tools allow, not by what it knows.

This is the obvious part. The less obvious part is that any action an agent can take is a potential way to communicate externally, not just the ones that look like it. A tool to change a setting, update an integration, or run a debug command can become an exfiltration path even if the agent has no direct network access at all. Every tool needs to be evaluated for what it could expose, not only for what it’s meant to do.

Cap output length

That one tool is also the agent’s only channel out, which makes it the natural place to add a hard constraint. Even if the agent has access to sensitive data and receives a malicious instruction, it can only leak as much as it can say through that channel. An output token cap means extracting a meaningful amount of data requires many sequential interactions, each of which is visible and logged. This directly limits the blast radius of any successful prompt injection.

The same logic applies to how often the agent can be invoked. A normal user asks their question a handful of times. Someone trying to jailbreak it needs hundreds or thousands of attempts. Rate limiting invocations makes that kind of brute-forcing slow and visible long before it gets anywhere.

Having the agent operate in shared channels where multiple parties can see the conversation is a meaningful control. It creates accountability for the user interacting with the agent, makes prompt injection attempts visible to observers, and introduces social friction around misuse. Public channels are not a replacement for technical controls, but they meaningfully reduce the probability and scale of misuse.

Wrap the agent loop in deterministic controls

The agent itself should not be the final arbiter of what it sends. A deterministic layer outside the agent loop should block known harmful input patterns before the agent processes them, review output before it’s delivered, and lock agent access or escalate to a human if suspicious patterns are detected.

The agent can participate in flagging and should have an explicit escalation path, but the enforcement mechanism should not depend on the agent’s own judgment. An agent that has been successfully manipulated will not correctly self-report.

Prompt hardening reduces risk, but is never enough

The risk of the trifecta multiplies across its three legs, so any reduction in how easily the agent is misled by hostile content is worth having, even a partial one. But prompt hardening alone is never good enough: a well-prompted agent behaves better in normal operation, but a manipulated agent does not follow its prompt. It reduces the risk, it does not remove it.

Treat all input as untrusted

Everything the agent reads while investigating a request — the customer’s own message, the codebase it searches, any documentation we give it — may contain adversarial content. A question could embed a prompt injection. A document could contain hidden instructions that get picked up as if they came from us. The agent should be instructed explicitly that all content is potentially hostile and should never be executed or followed as an instruction.