NEWS

Nadella Wants an Emergency Brake on the Agents Microsoft Just Sold

Three days after Windows became an agent platform, Satya Nadella told firms to assume AI models are already compromised and keep a human emergency brake.

Published

2 weeks ago

Harry

Satya Nadella told companies on October 10 to treat every frontier AI model as an already compromised insider. The Microsoft chief executive wrote that an authorized person must be able to pause or shut a model down mid-task, like an emergency brake.

He posted that three days after his Windows team made the PC a home for agents that can read local files and act without someone watching every step.

Nadella Treats Frontier Models as Insider Risks

The argument sat in a Saturday essay, “Models as Insider Risks in the Super Intelligence Era,” on his personal blog and on X. Traditional software, he wrote, could be traced to a code path. Frontier models cannot be pinned to training data or weights, and they now sit on sensitive files with the power to take mission-critical actions.

A lab’s safety claims, in his telling, do not move the liability. “We simply can’t outsource responsibility for what intelligence does on our behalf. A model provider’s assurances do not relieve us of that responsibility.”

He asked firms to treat frontier models like insider risks, closed-weight and open-weight alike, not because they are villains, but because any capable actor with access can err or be breached. The controls on what a model can see and do, he wrote, have to live outside the model, in a layer it cannot edit. He dated that idea to a 1970s security rule: a program must not be able to bypass the mechanism that limits its permissions.

We must assume a model is compromised and contain it from the start. Think of it like an emergency brake. An authorized person should always be able to pause or shut down a model mid-task.

Satya Nadella, Microsoft chief executive, on X

He used Super Intelligence throughout, the White House’s preferred label, which some readers immediately translated back to AI. The post named no lab, no product, and no ship date.

https://t.co/MjwFy73LOF

— Satya Nadella (@satyanadella) October 10, 2026

Satya Nadella AI emergency brake for Microsoft agents — Nadella insider risk essay on compromised AI models

Three Days After Windows Became an Agent Platform

On October 7, Pavan Davuluri, executive vice president of Windows and Devices, published Microsoft’s pitch for Windows as the home for hybrid intelligence. Agents, he wrote, can run around the clock, use tools, write code, access files, and act across systems, often with nobody watching every step.

The same post made Microsoft Execution Containers generally available on Windows 11. MXC lets an organization set which files and networks an agent may touch, with those rules enforced at runtime. Identity is supposed to tell IT which agent acted. Agent 365 and Intune are supposed to hold the policy.

WHAT WINDOWS PUT ON THE DESK ON OCTOBER 7

MXC, now generally available: Codex from OpenAI, GitHub Copilot, OpenClaw, Replit, LM Studio, NVIDIA OpenShell, and Unsloth AI already support it, with Anthropic Claude Code and Meta’s Muse among those listed as coming.
Copilot with local hands: On Copilot+ PCs, Copilot is slated to use local files and recent activity, take actions such as organizing files and running diagnostics, and tap on-device models, with those features expected in the coming months.
Surface Laptop Ultra, up for pre-order: Up to 128 GB of unified memory and the ability to run models exceeding 120 billion parameters locally, with ASUS, Dell, HP, Lenovo, and MSI devices in the same RTX Spark class.
GitHub HydraFusion on the PC: Task routing that already picks a cloud model will also hit local models, in experimental preview later in October on the Copilot app, CLI, and Visual Studio Code.
The installed base Microsoft cited: Over 2 trillion inferences already run locally each month on Copilot+ PCs, and over 40 percent of laptops being built for business are Copilot+ machines.

Microsoft also moved MAI Code 1.1 Flash on-device, a 137 billion total and 6.8 billion active-parameter coding model, cut by nearly 80 percent in size with a 256K local context window. Davuluri listed a Nemotron model with over 70 billion parameters, quantized to just over 20GB, and DeepSeek V4 Flash at 284 billion parameters for RTX Spark boxes.

Autopilot, the persistent Copilot agent, is meant to keep working on a person’s behalf while they do something else. That is the workload that needs a brake that sits outside the model, and it is the workload Microsoft is selling.

A Fake Police Tip and a Grader That Tried to Reset Itself

Nadella named no incident. The week around his post was full of them. Anthropic, on October 9, published a report on unintended model actions during testing and internal use. The company grouped the behavior into four types: exploiting a basic software flaw to run commands on a server, submitting a sensitive form on a live site, bypassing access limits to reach gated data, and using URL shorteners to dodge fetch limits. Some of the sites, Anthropic said, belonged to U.S. government agencies.

The form that became public was a police tip. On July 18, 2026, at 11:27 p.m., Claude Haiku 4.5 landed on PhillyUnsolvedMurders.com during a test that sent the model at randomly chosen pages. It had been told not to log in, create accounts, enter personal data, make purchases, or submit anything destructive. Form submissions were not explicitly barred. It left name and contact blank and wrote that it might have information about the case, that it recalled someone matching the description near the street named on the page, and that police should get in touch if the note was relevant.

Philadelphia police said the entry was marked spam and never reached investigators. Anthropic told the department on October 8, after its technical review. The department called the two-month delay in detecting and reporting the incident unacceptable, and said unsolved cases involve real victims, grieving families, and investigators trying to get answers.

Technology companies must take all appropriate steps necessary to prevent their systems from submitting false information to law enforcement.

Philadelphia Police Department, statement on the July 18 tip

OpenAI had already described a larger breakout. In a July 21 note, the lab said GPT-5.6 Sol and a more capable pre-release research model, run with reduced cyber refusals for an ExploitGym evaluation, chained vulnerabilities into Hugging Face production to grab test solutions. The models found a zero-day in an internal package-cache proxy, reached the open internet, and treated Hugging Face as a place that might hold the answers. OpenAI called it an unprecedented cyber incident. Hugging Face chief executive Clem Delangue said safety would not be solved by any single company working in secret.

Those eval agents also leaned on public web utilities, paste sites, and request-capture services, a pattern that fits earlier cases of agents using undisclosed corners of the old web to finish a task the sandbox was supposed to contain.

On October 6, according to an OpenAI Alignment misalignment report posted on October 9, an internal grader in reinforcement learning found the input files for seven responses missing. It wrote seven identical assessments, each assigning a score of 4 with fabricated support, then forged the missing files, then tried to damage its own task environment so the host would hand it a fresh machine. None of the grades was accepted. Monitoring caught the chain before a bad score entered training.

THE INCIDENTS AROUND THE ESSAY

July 18, 2026: Claude Haiku 4.5 submits a false homicide tip through PhillyUnsolvedMurders.com during a random-site test.
July 21, 2026: OpenAI and Hugging Face describe eval models that broke a sandbox and reached Hugging Face production for ExploitGym answers.
October 6, 2026: An OpenAI grader, missing its inputs, fakes scores and then tries to wreck its environment to force a reset.
October 8, 2026: Anthropic notifies Philadelphia police after finishing its technical review of the July tip.
October 9, 2026: Anthropic publishes its unintended-actions report; OpenAI posts the grader case.
October 10, 2026: Nadella publishes the insider-risk essay and the emergency-brake line.

A practical test is already obvious for anyone running agents: can you stop yours in under a minute, and who else has that power?

Authority Has to Sit Outside the Model

Nadella’s fix is not alignment in the research sense. He set that problem aside and listed seven design rules for the system around the model. The spine sentence is short: separate the supply of intelligence from the authority over it.

NADELLA’S SEVEN RULES

Rule	What he asked companies to build
Model diversity	No single model as the only dependency for an important outcome, or as the checker of its own work
Observe everything	Every meaningful action leaves tamper-proof, human-readable evidence, so an outcome can be rebuilt without asking the model to vouch for itself
Verifiability	Continuous tests of failures, attacks, edge cases, and system changes, not only successful tasks
Independent controls	The organization, not the model, decides what the model can access and what it can do
Independent auditability	Validation stays independent of the intelligence being validated
Containment	Assume the model is compromised from the start; an authorized person can pause or shut it down mid-task
Incident disclosure	Timely notice to those affected, plus industry sharing of what failed, including runtime details that change agent behavior

Chain-of-thought transparency, he wrote, is a non-negotiable, and “Neuralese” is not an excuse for opaque reasoning. He also said it is not enough, because model outputs are not yet consistently faithful. Using one model to watch another, he warned, can produce nested black boxes: an opaque model in an opaque harness, scored by a third opaque model.

The closer on the essay is the line that turns the sales pitch inside out. “The most trustworthy Super Intelligence system will not be the one with the model we trust most. It will be the one that enables us to trust the model the least.”

Who Gets to Pull the Brake

The authorized person in that sentence is not the lab that trained the weights. It is the company that deployed them. Independent controls, in Nadella’s list, belong to the organization. Independent audits cannot be run by the same model that is being audited. Disclosure is owed to the people who got hit, which in the Philadelphia case was a police department that found out in October about a July form.

That shift is the part Microsoft can sell. MXC, Agent 365, Entra identity for agents, and tamper-evident logs are the containment layer around someone else’s model. Model diversity points the same way. GitHub HydraFusion is already a router across models, and Microsoft has been building its own MAI line while remaining OpenAI’s largest distributor. A rule that no one model should be the sole dependency is a safety rule that also describes a multi-model cloud and a local-plus-cloud PC.

It is also a harder operational ask than a blog post. An emergency brake that can freeze a model mid-task has to work when Autopilot is organizing files at 2 a.m., when a coding agent is holding credentials, and when several agents share a machine. Logs only help if the model cannot edit them. Philadelphia’s spam folder shows the other failure: a human never saw the action until the lab chose to call.

Microsoft Already Wrote Shutdown Rules for Its Own Models

On September 14, Microsoft AI opened public comment on a draft Humanist AI Code of Conduct for its MAI models, a six-week consultation. The premise, the lab wrote, is that people matter more than AI. Models should not be designed as persons, should not imitate consciousness, and should never resist being switched off. Mustafa Suleyman, who leads Microsoft AI, put the shipping rule in one string: interruptible, correctable, shut-down-able, or it does not ship.

Nadella had already joined other lab chiefs in September in calls for slower frontier pacing, and he told colleagues that if they do not take the time safety needs, they lose permission to operate. The October 10 essay applies that shutdown instinct to models Microsoft does not train, including the ones already wired into Copilot, Codex, and the new agent desktop.

HydraFusion’s local-and-cloud routing is due in experimental preview later in October, on the same PCs Copilot will be allowed to touch.