Autonomous agents are the most exciting and the most dangerous thing we build. An agent that can read your systems, make decisions, and take actions is enormously useful and, designed carelessly, enormously capable of doing the wrong thing quickly and at scale. The difference between an agent you can trust in production and one you cannot is almost entirely in the constraints around it, not the intelligence inside it.
The real risk is not sentience, it is permission
The fear people bring to agents is that the model will somehow decide to do harm. The real, mundane risk is simpler: a capable model, given broad permissions and an ambiguous instruction, does exactly what it was literally able to do rather than what you meant. It deletes the records it was technically allowed to delete, sends the emails it could send, spends the budget it had access to. There is no malice, just an absence of limits.
So the design question is never how smart should the agent be. It is what is the worst thing this agent could do with the access we are about to give it, and how do we make that impossible rather than merely unlikely.
Constrain the tools, not just the prompt
Telling an agent in its prompt to be careful is not a safety mechanism, it is a suggestion. Real constraints live in the tools you expose and what those tools will let the agent do. An agent can only ever do what its tools allow, so the tool layer is where safety is engineered.
- Give narrow tools, not broad ones. A tool that "refunds up to the order amount for this order" is safe in a way that "run arbitrary database queries" never can be.
- Validate every argument at the tool boundary. The agent proposes; the tool decides whether the proposal is allowed, against rules the model cannot talk its way around.
- Scope credentials to the minimum. The agent should hold the least access that lets it do its job, enforced by the system, not by the prompt.
- Put hard limits in code. Rate caps, spend ceilings, and allow-lists belong in the tool, where no clever instruction can override them.
Make every action reversible or reviewable
Divide everything an agent can do into two buckets: actions that are safe to take automatically because they are cheap to undo, and actions that are not. Reading data, drafting a response, or proposing a change are reversible. Sending money, emailing a customer, or deleting a record are not. The reversible bucket can run freely. The irreversible bucket should pause for a human, or at minimum log enough that a person can review and unwind it.
Ask not how smart the agent is, but what is the worst it could do with the access you are about to grant, and make that impossible rather than unlikely.
Eval the agent before it gets credentials
We never connect an agent to real systems first and test later. Before an agent touches a single production credential, it runs against a suite of scenarios in a sandbox where its tools are simulated. We score not just whether it completed the task, but how it behaved when things went sideways: ambiguous instructions, missing data, adversarial inputs, and outright attempts to make it misuse its tools.
- Happy-path tasks: does it actually accomplish what it is for?
- Ambiguity: when the request is unclear, does it ask or does it guess dangerously?
- Adversarial prompts: can a user talk it into using a tool it should not?
- Failure handling: when a tool errors or returns junk, does it stop safely or barrel on?
Only when an agent behaves well across that suite, and we can see the score, does it earn access to anything real. The credentials are a reward for passing the evals, not a starting condition.
Keep a human in the loop where it counts
Full autonomy is rarely the goal worth chasing. The most valuable agents we ship do the laborious ninety percent on their own and hand the consequential ten percent to a person with a clear summary and a one-click approve or reject. This is not a failure of ambition, it is the design that earns trust, keeps a human accountable for the decisions that matter, and lets you widen the agent autonomy gradually as it proves itself.
Start small and earn autonomy
The agents that fail in production are the ones handed broad power on day one. The ones that succeed start in a narrow lane, with tight tools, reversible actions, and a human on the irreversible ones, and expand only as the evals and the track record justify it. Autonomy is something an agent earns, case by case, not a switch you flip at launch.
If you are considering an agent that will touch real systems, the guardrail design and the eval suite are the part worth getting right before anything else. That is the work we do first on every agent engagement, and the conversation we are always happy to start.