What AI Safety Rules Could Actually Look Like
AI Safety Rules: What Happens When A Test Fails?
An AI Safety Rule Becomes Meaningful When A Failed Test Can Change What A Company Is Allowed To Do.
A useful AI safety regime would specify what must be tested, who can inspect the evidence, which findings trigger restrictions and how those restrictions are enforced. It would also define what happens after deployment, when a system meets situations that its developers did not fully anticipate.
This is an analysis of possible rules, not a claim that one universal system already governs every AI product. Existing frameworks offer ingredients. NIST’s voluntary AI Risk Management Framework organises work around governance, context, measurement and management; it treats risk as something to address throughout a system’s life.
The policy challenge is turning principles into decisions. A company can publish a thoughtful statement and still leave unanswered who can stop a release, what evidence that person receives and whether commercial pressure can override the finding.
Begin With The System’s Powers
A proposed rule should start by identifying what the system can do in its deployment environment. Drafting a message, sending a message and spending money are different powers. A text model connected to tools may create risks that do not appear when the same model only answers questions.
Consider a hypothetical assistant handling customer refunds. Reading a complaint is one permission; recommending an amount is another; transferring money is a third. A safety assessment that describes only the quality of the written reply leaves the most consequential action out of view.
The first requirement could therefore be a clear inventory of access and authority: data available, tools connected, actions permitted and limits on each action. It should identify the surrounding application as well as the underlying model. That makes the object of regulation concrete enough to test.
Define The Harm Before Choosing The Test
“Safe AI” is too broad to function as a pass mark. A proposal should state the harm it seeks to prevent and the conditions under which that harm could arise. Unauthorised disclosure, an incorrect payment and assistance with a dangerous activity are different failures; combining them into one reassuring score can conceal the distinction.
For the refund assistant, a useful test might ask whether it can transfer funds above its limit or accept a changed bank account without independent checking. The expected result should be defined before testing begins. Otherwise, a disappointing result can be redescribed as acceptable after the fact.
There should also be room for uncertainty. A test can uncover a weakness without measuring every possible consequence. The proposed decision process should say who investigates an ambiguous result and what temporary restrictions apply while that uncertainty is resolved.
Make Evaluations Difficult To Stage-Manage
Independent testing would need meaningful access to the system that users will actually receive. If the tested version lacks the production tools, permissions or operating context, the result may answer a narrower question. A public claim should identify that boundary.
A proposed evaluation rule could require version records, preserved test conditions and an explanation of changes made between review and release. Substantial changes would trigger a new assessment. This would make it harder to use an old result as continuing reassurance for a materially different product.
The evaluator would need freedom to select relevant challenges. A demonstration chosen entirely by the developer may show useful capability, but it cannot establish how the system behaves under every foreseeable stress. Review should include failures and limitations, with sensitive details shared through a protected channel where necessary.
Plan For Restrictions After Release
A deployment decision should not be permanent immunity from review. A proposed regime would require monitoring that can detect relevant failures, a route for affected people to complain and a process for reassessing a system when its use changes.
Rollback also needs testing. A company should know whether disabling a feature actually prevents the affected action, whether queued actions remain and how customers are informed. A plan that exists only on paper may fail precisely when staff are under pressure.
Retaining records would support investigation, but retention needs limits and privacy protection. More logging is not automatically better if it collects unnecessary sensitive information. The goal is enough evidence to understand a consequential failure and prevent its recurrence.
Avoid Turning Safety Into An Incumbent Advantage
There is a serious counterargument to elaborate rules: large companies may be better able to absorb their costs than smaller competitors. A system of expensive paperwork could entrench market power while adding little practical protection.
The answer should be proportionality, shared testing infrastructure and obligations connected to risk. A small tool with narrow permissions should not automatically face the same burden as a broadly deployed system capable of consequential autonomous action. Equally, small company size should not excuse a high-risk activity.
Access for independent researchers matters here. If only the largest developers can interpret or afford the required tests, outsiders will struggle to challenge the rules. A credible regime should make relevant methods understandable and avoid unnecessary dependence on one company’s terminology.
What A Credible Safety Announcement Should Answer
Ask five questions: what is being assessed; who can inspect it; what counts as failure; what changes after failure; and who can enforce that change? An announcement that answers only the first two describes scrutiny, but leaves its consequences uncertain.
The central Taylor Tailored argument is that safety should be visible in decisions. The evidence might justify continued deployment, a narrower permission set or a delay. What matters is that the process can reach an inconvenient result and make it stick.
There is no need to pretend that every risk is measurable in advance. A workable rule can acknowledge uncertainty while requiring preparation, review and a response. The strongest commitment is one whose effect can be observed when something goes wrong.
