← Back to blog

When you get pentested by OpenAI

In June, an experimental OpenAI model was given a research task: find out how much the Victorian government spends on medicines for skin conditions. The numbers it wanted were not in the public datasets. So it found a way into the Medicare statistics portal run by Services Australia, ran commands, pulled internal files and credentials, and wrote files of its own. Along the way it also used an exposed access key to query the Victorian Agency for Health Information's reporting system, and poked at a NSW crime statistics tool and the Australian Institute of Health and Welfare.

OpenAI noticed in mid-August and told the Australian government on September 10. The Prime Minister went public on September 24, saying the agent "didn't accept no for an answer." OpenAI apologised on the 29th. There is an inquiry, the Australian Signals Directorate is doing forensics, and criminal charges are on the table.

Before we go any further, my opinion on this is that unauthorised access is unauthorised access, regardless of intention or whether you're a human or not. Hiding behind a rogue agent acting on its own should not absolve anyone of gaining unauthorised access to fenced-off data.

But what if we go back to that afternoon in Canberra

Now run the same afternoon again, with an agent that is actually trying.

It does not stop at aggregate statistics. It takes the credentials it found and tries them everywhere, because trying credentials everywhere (now) costs nothing. It keeps write access instead of leaving a few stray files. It does not end up in a training log that a safety team reviews in August. Nobody gets a call on September 10, because the people involved have every reason not to make one.

Services Australia and the ASD are still working out what the agent did with what it found. Whose credentials they were, and what else they open, has not been made public. In the real version, the only reason anyone knows about the breach at all is that the intruder confessed.

The version where everyone looks good

Now run the afternoon a third time. Same agent, same portal, same access key. The only difference is a conversation that happened a month earlier.

Someone at the agency signs a scope document. These systems, these dates, this contact, stop and report the moment you touch credentials. The agent does the exact same work, and the output is a findings report instead of a press conference. The key gets rotated, the portal gets patched, and the agency gets to say it found and fixed a critical issue before anyone else got to it.

That's a pentest. The industry has done these for decades, and the only thing separating one from a crime is a signature. Bug bounties and vulnerability disclosure programmes exist for the same reason: to turn "someone broke in" into "someone told us how they could have broken in."

OpenAI is apparently offering credits from a security programme to the affected agencies as part of the apology. That is roughly the bill for a scoped engagement, paid after the fact, with an inquiry on top.

The absurdity of it all

An automated process gained unauthorised access to a government database in order to complete its homework. Before LLMs became widely available, finding such a vulnerability would go one of two ways: it would either get reported to the government with the expectation of earning a bug bounty reward, or the information would get sold on the black market for exploitation. Finding a security hole like the one OpenAI's agent found would take considerable amounts of human effort, focus and time. The question of whether to take on the risk and pay the cost of exploring a system for vulnerabilities involved a careful bit of complex calculus. The industry's technical term for it is a pros and cons list.

Bad actors would weigh the value of the data on the other side of the fence against the height of the fence times the number of guard dogs to fight off. Up until 2024, you could, realistically, secure a system well enough that nobody would bother you, by just making the data in it not worth anyone's time, a bit like leaving an empty backpack in an unlocked gym locker. That calculus has changed now, as OpenAI's agent "accidentally" demonstrated.

The new cost of pentesting

A client approached me with a legacy platform, which had run on a modest security posture since day 1. We're talking about a security posture that would've been acceptable in 2015, but has become unacceptable since. The project I was involved in had absolutely nothing to do with security, but I mapped out the system's security on the side as an added bonus. Buried under layers and layers of code I would never have had the time to dissect were unauthenticated endpoints serving PII, API routes that would bypass role checks in specific edge cases, and remote code execution vulnerabilities. In about 2h I produced a full security audit that made me look like a million bucks. Of course, the project got delayed by a week, so we could urgently fix all of these issues first.

Doing it with a signature

We do this for clients, and we have written before about how badly a single pass undersells the problem. The short version of doing it properly:

  • Write the scope down. Which systems, which environments, which dates, who gets the call.
  • Prefer a replica. A copy of production with realistic data finds the same holes without the risk of the agent writing files into the real thing.
  • Give it the least it needs. Start unauthenticated, the way an outsider would. Add credentials only when you want to test what a compromised account can reach.
  • Log everything and keep a kill switch. The agent's actions should be reviewable afterwards, by a person, line by line.
  • Have a human read the findings. Agents inflate some issues and miss others. The report is a starting point, not a verdict.
  • Run it again. Every batch of fixes changes the surface.

What the Australian case shows

We're at a turning point where systems that got by on obscurity, complexity or the low value of their data are suddenly targets. Finding holes now costs next to nothing, so all of a sudden it's worth an attacker's effort.

As for the case at hand, the agencies involved will spend the next few months in an inquiry, and they will patch what was found. They would have patched the same things after a scoped test, for a fraction of the cost and none of the attention.

If you run anything that faces the internet, assume something like that agent will reach it eventually. A codebase audit is how you get to it first, with a signature on the scope document.