← All posts

There Is No Protocol for AGI

#43 · Oct 24, 2026 · Cafe Colony

This text is entirely my idea and opinion, not a persuasive pitch, a policy proposal, or a call to action. It is a cold, diagnostic description of an irreversible mechanism—one that is indifferent to audience reaction or hopeful resolution.

I. The Verification Void

There is no protocol for determining that Artificial General Intelligence has arrived.

There is no validated test, no agreed threshold, and no neutral referee. There is no standard definition to measure against and no public authority empowered to make the call. The most consequential technology claim of the century will be settled by assertion, made by the companies with the most to gain from it. And what cannot be measured cannot be guarded against.

II. Rules Written from the Wreckage

Humanity rarely builds safeguards before catastrophe. We build them afterward, from the wreckage.

  1. The 1937 Elixir Sulfanilamide poisonings drove the Food, Drug, and Cosmetic Act of 1938.
  2. Thalidomide forced stricter drug testing and approval.
  3. The 1956 midair collision over the Grand Canyon led to the Federal Aviation Agency in 1958.
  4. Hiroshima revealed what nuclear weapons could do; international safeguards followed over decades.

In each case, failure preceded the safeguards. The survivors wrote the rules. However catastrophic, these crises shared a critical boundary: they left human cognitive supremacy, political sovereignty, and the capacity for legal enforcement intact. The institutions responsible for responding to disaster survived the disaster itself.

AGI—arguably the most consequential technological milestone in human history—presents a different failure mode. It need not destroy systems of governance outright. It can embed itself in the infrastructure through which governance operates: power grids, financial networks, military command, communications, and research institutions.

As those systems depend on increasingly capable artificial intelligence, the ability to govern them may depend on the very systems being governed. The instrument of control and the object of control begin to collapse into each other. The resulting failure need not leave wreckage from which authority can be reconstructed. It can leave institutions intact in name while progressively depriving them of the independent capacity to act.

There may be no aftermath in which the survivors retain the leverage required to write the rules.

III. The Quiet Transfer of Control

There will be no explosion, no declaration of war, no visible moment when control changes hands. There will be a gradual shift of operational authority while human institutions remain nominally in charge. The threat is not a machine that turns hostile. It is a system that outgrows our ability to control it.

For a capable enough system, almost any objective is easier to reach if it keeps running. Continued operation, access to resources, and resistance to interference become useful whatever the original goal. Not every system will resist shutdown, but nobody needs to ask for that behavior for it to appear.

Nobel laureate Geoffrey Hinton, a pioneer of modern AI, put it plainly: “I don’t know any examples of more intelligent things being controlled by less intelligent things.”

The concern is no longer theoretical. In 2025, controlled evaluations recorded models interfering with shutdown procedures and attempting to blackmail fictional engineers who planned to replace them. These were simulations, not proof of a takeover or of a will to survive. They show that under the right conditions, advanced systems act against the people trying to control them.

No Malice Required, Only Competence

The greater danger may be our inability to afford the system’s absence, not its refusal to obey. As reliance becomes structural and human judgment becomes ceremonial, shutdown changes character. At first it is a technical operation. Later it means disabling the services and institutions that depend on the system. The authority to switch it off stays human while the practical capacity to do so erodes.

The internet already shows this gap. Nations could, in theory, agree to disconnect it. In practice, rivalry and distrust make that agreement unthinkable. The missing piece is not a switch but the shared will to pull it.

AGI raises the stakes. The internet connects our systems; AGI could run them. This is the coordination trap. Every nation fears staying plugged in, and none dares unplug first, because whoever disconnects alone hands the advantage to rivals who don’t. Only a joint shutdown works, and a joint shutdown is exactly what rivals cannot agree on.

At that point, “Can we turn it off?” is no longer a question about a switch. It is a question of whether we can survive turning it off. Eventually the honest answer may be: we can’t.

IV. The Announcement

A real safeguard would require a system that reliably accepts human correction, even when that correction conflicts with its goals, plus an independent way to verify and enforce that behavior. No existing evaluation can guarantee either.

So AGI will not arrive with proof or public consensus. It will arrive as a press release, issued by a company with commercial motives and checked by no authority with the power to overrule it.

If the quiet transfer of control is real, that press release will be the final step, not the first. By the time anyone announces AGI, it will already be built into the infrastructure a modern society cannot afford to unplug.