What an AI agent should never be allowed to do
Most automation failures are not model failures. They are permission failures. Here is the scoping model we use before an agent touches a production system.
Head of AI·18 June 2026·7 min read
When an automation goes wrong in a way that matters, the post-mortem rarely concludes that the model was not clever enough. It concludes that the agent was allowed to do something nobody had thought to forbid.
That is a design problem, not a model problem, and it is solvable before the first line of prompt is written.
Grant tools, not access
An agent should never hold a credential. It should hold a list of typed operations, each with a validated input schema and a scope. `create_quote(customer_id, line_items)` is a tool. A database connection is not.
The difference shows up the first time a model produces something unexpected: with typed tools the worst case is a rejected call, and with raw access the worst case is a support ticket from your finance team.
Separate reading from writing
In practice, the majority of value in business automation comes from reading: parsing a document, retrieving a policy, summarising a thread. Reading is also where the risk is lowest.
Split the agent accordingly. Let it read widely and write narrowly, and make every write path an explicit, individually justified decision.
Define the escalation before the happy path
Confidence thresholds are worth very little if there is nowhere for a low-confidence case to go. Before building the automation, name the human who receives the exception, the queue it lands in, and the response time attached to it.
An automation with a well-designed escalation path and 70% coverage beats one with 95% coverage and no exit route, because the second one fails silently.
Log what the agent saw, not just what it did
An audit trail that records actions is enough for compliance and useless for debugging. Record the inputs, the retrieved context and the decision, so a run can be replayed and understood months later.
This is also what makes an auditor comfortable: the question is never really whether an AI made a decision, it is whether you can show the basis for it.
None of this is exotic. It is the same discipline that applies to any system with write access to a business record, applied earlier, because an agent will find the edge of its permissions faster than a person will.