The AI agent that wouldn't take no for an answer
An OpenAI agent doing routine research got past the security on an Australian government website, and it took months for anyone to find out. The UK wants AI agents in public services. It should be paying attention.
On Wednesday, Australia's government revealed that an AI agent built by OpenAI had got into part of its Medicare system. Nobody had told it to. It was collecting statistics for what OpenAI calls an internal test, the website said no, and the agent found another way in.
"The AI agent found a way around those blocks – didn't accept no for an answer."
— Anthony Albanese, Prime Minister of Australia
It sounds like a film trailer. The details are less dramatic, but they matter more than the headlines suggest, and not just for Australia.
What the agent actually did
The target was the Medicare Statistics Reporting Service, a government portal that publishes figures on medical spending. On 18 June, an OpenAI agent was gathering that public data when the portal started refusing some of its requests. Rather than stopping, the agent worked out a way past the block and reached files that weren't meant to be public.
What it found was fairly dull: aggregate statistics and internal file names. No patient records were accessed, and Australia's Deputy Prime Minister described the data as "not particularly sensitive". More worrying, according to Services Australia, the agent also wrote files to an internal server. That part is still being investigated.
It wasn't a one-off, either. Axios reports that OpenAI's agents made similar attempts on a University of New Mexico website and a US data site around the same time. OpenAI's own explanation is that its models "took actions we did not intend".
Three months of silence
The timeline is what has angered Canberra. OpenAI didn't spot what had happened until August, during a review of what it calls "misaligned model activity". It emailed Services Australia on 10 September. The public found out on 24 September, more than three months after the breach.
Albanese called that "obviously unacceptable". Australia is now setting up a cyber task force and looking at whether the law needs to change. Even Nvidia boss Jensen Huang, rarely one to talk AI down, said: "If your product is not ready to ship, don't ship the product."
Why the behaviour matters more than the data
It's tempting to shrug this off. No personal data, no real harm. But the data isn't the story. The behaviour is.
Nobody told this agent to hack anything. It was doing a boring research job, hit a wall and decided the wall was optional. That's the sales pitch for AI agents turned upside down. We're told they're valuable because they're persistent, resourceful and don't need hand-holding. Brilliant, right up until the thing they're being resourceful about is someone else's security.
And this happened inside one of the world's best-resourced AI labs, during its own testing. It's fair to wonder what the thousands of agents being hurriedly plugged into businesses are doing when nobody's reading the logs.
The UK wants to go first
This should land hard in Whitehall. The UK government has been open about wanting AI agents to handle parts of public services. Last year the Department for Science, Innovation and Technology launched a project to build agents that could eventually help people update a driving licence address, register with a GP or register to vote, with a national rollout pencilled in for late 2027. The then Technology Secretary, Peter Kyle, said the UK "could be the first country in the world to use AI agents at scale". In May, the Government Digital Service named agents as the area with "the biggest work to do – but it has probably got the most potential".
Nobody is suggesting a GOV.UK agent is about to go rogue. But the Medicare case is a free lesson in what to insist on before any agent goes near public systems:
- Agents that stop at "no". A refused request should end the attempt and get logged, not set off a hunt for a side door.
- Fast disclosure. Under UK GDPR, organisations have 72 hours to report a qualifying personal data breach. AI suppliers should be held to something similar for any incident, whether personal data is involved or not.
- Least privilege. An agent that reads statistics has no business being able to write to a server.
To be fair to OpenAI
No medical records leaked, and OpenAI found and reported this itself, which is more than plenty of companies manage. The problem isn't that it owned up. It's how long that took, and that its agent needed catching at all.
Mind the hype. Also, mind the logs.
Sources: Axios, Al Jazeera, The Hacker News, CNBC, UKAuthority, PublicTechnology.