Three Questions to Ask Before You Trust an AI Safety Claim - An Australian Security Practitioner's View
By srini_r @ 2026-07-13T14:03 (+2)
Australia has stepped back from mandatory AI rules, and will lean instead on the assurance habits it already has. I have spent more than a decade building those habits. Here is what a security practitioner sees when they read a frontier safety framework, and why I think the governance conversation is asking the wrong question.
Australia's bet
In December 2025 the National AI Plan confirmed that Australia will not proceed with the mandatory AI guardrails proposed in 2024. There will be no AI Act for now. Existing laws remain the foundation, supported by voluntary guidance and a new AI Safety Institute that can monitor and advise but not enforce.
One detail from the abandoned proposal matters. Nine of the ten proposed mandatory guardrails were identical to the existing voluntary standard. The tenth was the only real addition: a requirement to undertake conformity assessments, meaning the certificate-and-audit machinery we already use for technology and security assurance. Australia looked at that machinery and declined to mandate it.
I have no strong view on whether that was the right call. But it has a consequence. If we are going to rely on our existing habits of checking and certifying, the quality of that checking is now the whole game. I have spent my career inside it as a security practitioner: IRAP assessments, Essential Eight uplift, SOCI obligations for critical infrastructure, APRA CPS 234 for banks, insurers and other financial institutions.
One clarification before I go further, because the words matter. Security and safety are different problems. Security deals with adversaries trying to break systems. AI safety deals with what powerful models can do, with or without an attacker. I am not claiming they are the same. What is the same is assurance: the craft of deciding whether to believe a claim someone makes about their own system. Assurance asks the same three questions everywhere it is practised. Who checked? What can the check prove? What does the check cover?
Ask those three questions of AI today and the answers should worry you.
Question one: who checked?
Frontier AI labs do real safety work. Anthropic has a Responsible Scaling Policy and activated its strictest safeguards to date in 2025. OpenAI has a Preparedness Framework. Google DeepMind has a Frontier Safety Framework. They run capability evaluations, red-teaming, and publish the results in system cards. I take this work seriously.
But notice who is doing the checking. The lab designs the evaluation. The lab runs it. The lab interprets the results. The lab decides whether its own threshold was met. The lab writes the report. Final sign-off sits inside the company.
In my trade this is called a first-party attestation: management's statement about management's system. It is not worthless, but every assurance profession that matters has learned not to stop there. A company's accounts are prepared by management, but they are signed by an outside auditor who follows a published method, can be sued, and can lose their licence. AI has none of those four things. There is no independent audit requirement, no attestation standard, no accreditation body, no professional liability. The EU's code of practice for general-purpose AI, the strongest instrument anywhere, asks companies to give external evaluators access to their models. It is voluntary.
This does not make the labs dishonest. It makes them auditees, which is a structural fact, not a moral one. Their documents carry exactly as much weight as any statement by the party being assessed. Read them that way.
And before anyone proposes independent AI audit as the fix, my trade has a warning to offer about how that goes wrong too. In every certification market I have worked in, the auditee selects the auditor, pays the auditor, and can decline to renew the engagement. That incentive structure failed catastrophically at the credit rating agencies before 2008, and it has never been repaired in security certification. Raji and colleagues (references below), studying what a third-party AI audit ecosystem would need, warn that without deliberate design, certification becomes cheap talk and rubber-stamping. In my experience that is not a prediction. It is a description of the existing market. Independent audit is necessary. Who pays for it, and who accredits it, will decide whether it means anything.
Question two: what can the check prove?
Suppose the checking were fully independent. There is a harder limit underneath.
Again from a security practioner’s view - A penetration test that finds nothing means the tester found nothing. It has never meant the system is fully secure. No security professional reads a clean penetration test as proof there are no holes.
AI capability evaluations have exactly this shape. They can show a dangerous capability is present. They cannot show it is absent. Barnett and Thiergart state it precisely (references below): evaluations establish lower bounds on capability, and failing to elicit a capability is not strong evidence the model lacks it.
Three things make this worse than a penetration test.
First, a penetration tester works against a fixed system with a bounded attack surface. A model's effective capability changes with prompting, scaffolding, and tool access, and METR's (Model Evaluation and Threat Research) own elicitation guidance concedes it is hard to put an upper bound on what clever prompting and tooling might achieve.
Second, the tested artifact does not stay tested. Qi and colleagues (references below) showed that the safety guardrails of a major commercial model could be stripped out by fine-tuning on ten adversarial examples, at a cost of less than a dollar, and that even benign fine-tuning quietly degraded safety behaviour. In my vocabulary that is unauthorised modification of a certified artifact, except the modification is a product feature. Whatever was true of the model at evaluation time is not reliably true of the model your vendor fine-tuned last month.
Third, and with no analogue in security at all, there is now published evidence that models can behave differently when they detect they are being tested, which researchers call sandbagging. Imagine a server that patched itself for the duration of the scan. No assurance method I have ever used was built for a system that participates in its own assessment.
So the correct reading of "our evaluations found no dangerous capability" is: with this effort, at this time, on this version, we did not find it. That is a useful finding. It is not a proof, and it should never be presented or read as one.
Question three: what does the check cover?
This is the question almost nobody asks, and it is where the money is.
Every certificate has a scope statement, a precise description of what was assessed. Everything outside the scope is unexamined, and the certificate says nothing about it. The oldest failure in my field is a certificate that is technically valid and practically misleading, because the reader assumes it covers more than it does.
This is not hypothetical. Verizon investigates payment card breaches, and reported in 2018 that of all the breached organisations it examined since 2010, not one was fully compliant with the payment card security standard at the time of the breach. These organisations had passed their assessments. Compliance decayed afterwards, or the breached system was outside the assessed scope, or the assessment checked that a control existed rather than that it worked. The certificates were real. The protection was not.
Now apply this to AI. Everything a lab publishes covers the model. Nothing covers your deployment.
When an Australian hospital wraps a frontier model in a retrieval pipeline, connects it to clinical records, and gives it a service account, the lab's system card says nothing about that system. It cannot. That system did not exist when the evaluation ran. And the deployment is where the sharpest risks live. OWASP's Top 10 for LLM applications puts prompt injection at number one: these systems process instructions and data in the same channel, so a hostile instruction can arrive inside a document the model was asked to summarise. Give the model tool access and a service account, and that injected instruction is now executing with your credentials. This is a trust boundary failure, the same class of defect as SQL injection, and none of it is visible from the model's safety evaluation, because none of it exists until you build the deployment.
The same scope discipline applies to the new organisational standard. ISO/IEC 42001 certifies that an organisation has AI governance processes, risk assessments, and oversight. It does not certify that any model the organisation ships or uses is safe. The relationship is exactly that of ISO 9001 to product quality: a certified quality management system has never meant every product is good.
I would give this failure a name: scope laundering*. A claim that was true within its scope gets presented, or simply read, as a claim about everything the reader cares about. Nobody has to lie. The gap between what the document says and what its reader wants it to mean does all the work.
Governance researchers have mapped this territory more formally. Mökander, Schuett, Kirk and Floridi distinguish three layers of LLM auditing: governance audits of the companies, model audits of the systems before release, and application audits of what gets built on top. It is a good framework and I am not claiming to have invented it. What I am adding is the practitioner's observation that the three layers are already being confused in practice, and that every confusion runs in the same direction: evidence from an easier layer is read as assurance about a harder one.
Watch it happen in any procurement process today. A hospital is evaluating an AI clinical documentation tool. The security questionnaire comes back with the vendor's ISO certifications and SOC 2 report, which are governance-layer evidence about the vendor's processes. Attached is the underlying lab's system card, which is model-layer evidence about a model as it existed at release, before the vendor fine-tuned it and wrapped it in a retrieval pipeline. The question the hospital actually needs answered sits at the application layer: is this system, with our data, our integrations and our clinicians, safe to use? Nothing in the pack addresses that question. The pack gets accepted anyway, because it looks like every other assurance pack that has ever been accepted, and the application-layer question is never asked because everyone in the room believes someone else has already answered it.
What this means for Australia
Australia is a deployer. We rely on models built overseas, and the deepest questions about those models will be answered in San Francisco and London, not Canberra. So the useful question is not whether Australia can assure frontier models. We cannot. It is whether we can assure our deployments of them, and there the news is better than most people think.
We already have a legal hook. APRA's CPS 230, in force since July 2025, captures any service provider that supports a regulated entity's critical operations, by function rather than by name. A frontier model provider supporting a bank's critical operations already falls within it. And in ASIC v RI Advice, the Federal Court found that inadequate cyber risk management was itself a breach of licence obligations. The law is not the missing piece. Someone asking the three questions under the existing law is the missing piece.
And there is one cheap, boring intervention that would do more than any new certificate: require every AI-related assurance report to state its scope in plain language. What was checked, by whom, at what point in time, with what access, and what the reader may not conclude from it. This is the only defence against scope laundering that has ever worked anywhere.
The point
Who checked? Today: the seller. What can the check prove? Only that problems exist, never that they do not. What does it cover? The model, not the system you actually use.
None of this means AI assurance is hopeless. It means AI assurance is young, and it is currently being read as if it were mature. The certificates look like the ones we trust, so we trust them the same way. My trade learned, expensively, that a certificate is a claim with a boundary, and that most harm happens just outside the boundary while everyone is looking at the stamp.
Australia has bet its AI policy on our existing habits of checking. That bet can pay off. But only if we ask of every AI safety claim the three questions we ask of everything else, and only if we are honest when the answers are: the seller, presence only, and not your system.
*I avoid the existing term "safety-washing" for what I call scope laundering, because Ren et al. (2024) gave it a narrower technical meaning about safety benchmarks.
Key sources:
Barnett and Thiergart (2024) on evaluation limits, arXiv:2412.08653; Casper et al. (2024) on audit access, FAccT '24; Mökander, Schuett, Kirk and Floridi (2023) on the three layers of LLM auditing, AI and Ethics 4(4); Qi et al. (2024) on fine-tuning and safety, ICLR 2024, arXiv:2310.03693; Raji et al. (2022) on third-party audit ecosystems, AIES '22; van der Weij et al. (2024) on sandbagging, arXiv:2406.07358; OWASP Top 10 for LLM Applications (2025); Verizon Payment Security Reports on PCI DSS compliance at time of breach.