if you don't get the title: tesla cybertruck accelerator pedal recall report (nhtsa 24v-276)
introduction
we have moved past the point where LLM's are considered basic chatbots. we have entered the age of reactive machines.
LLMs have evolved at a rate that, quite frankly, I don't believe anybody expected. There were obviously AI optimists that saw serious potential in AI back in 2019, but I don't believe that any of them, even some over-optimists, would believe you if you told them that AI architectures could manage multi-million-line codebases, talk to hundreds of millions of people every day, or believe that AI companies are filing for IPOs at target valuations worth $1 Trillion. Most YCombinator companies are AI-focused, and around 60% of recent YC batches are either AI-focused or heavily rely on AI agents1. AI-Native companies are becoming auditory candy for investors because of the implied Return on Investment & historical Return on Investment from AI Giants such as OpenAI and Anthropic, alongside Fear Of Missing Out on the next big-name AI startup.
Most companies transitioning to AI-based solutions do so because their tech stack is stuck in the past and they feel like they're falling behind competitors who are using AI. I recently shadowed at Brown University Health, specifically at Rhode Island Hospital, in their Interventional Radiology department. The doctor I shadowed under was a great guy and taught me a lot about the angio suite. I got to see a lot of innovative technologies being put into use and really awesome people doing the work they love to do, but I noticed that in the "Reading Room", where Radiologists, Nurses, and Techs meet and discuss their schedules, a lot of outdated technology, large, bulky monitors, and cluttered, ugly UI. As big as AI may seem, this brought me to believe that we really are at the start of the AI revolution. Agents being able to write code on their own or manage large, highly complex SQL databases might be eye-catching for some, but I'm much more sure that RSI and AGI will truly take the crowd by surprise when the time comes.
While all of this is great, AI's biggest flaw is that it can't be trusted to do things completely on its own. You won't see anybody working on actual projects that have consequences for wrongdoing deploying agents, running agents, or using agents in general without guardrails. Whether it be HITL systems such as OpenAI's "Approve For Me" or Anthropic's HITL system "auto-mode", branch protections on GitHub, or Continuous Integration, Agents cannot be trusted to work on things without someone staring at the screen or monitoring their notifications while they are mid-turn, simply due to the fact that they are fancy text predictors that have access to your CRM and your fiscal year's documents.
While it may take some time, the AI industry will eventually wake up. CISOs and CTOs are already freaking out because they don't have enough control over their production agents.2 We are currently in a kill-switch crisis3. The current benchmarks that exist simply fail to capture the fact that agents can and will perform unpredictably because user goals, tool access, permissions, context windows, external information, configuration prompts, multi-turn history, and other spawned subagents cause significant behavioral variation during agent inference. While tools such as Docker and Claude Config files help, they aren't deterministic and they have completely varying effects on your agent depending on the context of your situation.
As a sixteen year old living on the edge of Maryland, you wouldn't necessarily think that AI would be that big a part of my life. DC is primarily known as a government-heavy region, and tech doesn't really play as much a part as it does in powerhouses such as San Francisco, Palo Alto, New York City, or Seattle. A big part of my life has been spent developing games, since I was 6, to this day. It's one of my favorite things to do and I enjoy seeing my creative abilities being put into something that people can enjoy.
Agentic development has been significantly embedded into major game development engines such as Unity, Unreal Engine, and Roblox Studio. With the release of Unity 20.0.0-pre.1, Unreal Engine 5.8, and Roblox Studio Assistant Agentic Release, these seemingly hand-focused tools have gone full agentic, with even Roblox's DevRel team stating "Roblox Studio has gone Agentic!"4. My point here is that agentic development is spreading at an unforeseen and alarming rate, comparative & relative to the way that we are scaling and treating the governance systems that we have currently implemented.
The Agentic Governance wall will break down into pieces soon, and companies will break apart faster than ever before seen. Deloitte reported that "Agentic AI is scaling faster than guardrails". ~21% of enterprises report mature governance for agentic/autonomous AI, 82% of enterprises report shadow agents, 65% of enterprises reported AI-agent-related incidents in the last year, and 53% of enterprises reported agents exceeding intended permissions.
As much as everyone loves interactive, multi-turn agents, we are facing an existential crisis. If we don't get a hold of our agents & implement guardrails, as models get smarter, the problems that we face will increase proportionally to the intelligence that these models gain during training. We are in for a rude awakening. I want you to close your eyes and imagine a world in 10 years, where it's not crazy to hear on the news that another startup has had its database completely wiped due to a rogue agent. I believe that this entire situation is completely avoidable.
When we face this issue, we can look back to the original kill-switch crisis. In the 1800s, steam power enabled ships, railroads, and industrial plants to scale faster than society fundamentally knew how to control them. Boilers became significantly more powerful by the week, but operator training, standards, and pressure controls lagged significantly behind. The result was thousands of boiler explosions across Europe and the U.S.; after realizing how powerful Boilers had become (in our case, Agents), the ASME Boiler and Pressure Vessel Code was implemented, inspections became mandatory, inspectors were required to be analyzed through uniform qualification, and physical safety mechanisms were implemented. Regulators realized that the machinery that was being put into national infrastructure was far too powerful to be implemented without any guardrails. While big-tech may be implementing strict guardrails, this is not enough, as models still need to face fine-grained control when interacting with executive, serious data. As their ability increases, they will attempt and, even if unintentionally, cause serious harm to the data sources that they are given access to, if for whatever reason the main pathway to achieving their assigned task is not possible.
The steam power crisis directly relates to the Governance:Ability relationship I've graphed above. If we don't implement tooling soon to flip the logarithmic linearity of the Governance Guardrails line, we are going to end up in seriously hot waters with the way that the world interacts with these Agents that can use 50+ tool uses in one turn. Trains do not function without guardrails, and neither will Agents, especially in corporate environments, if we do not implement those same guardrails to prevent agents from obfuscating data in our databases and to prevent our OpenClaw agents from sending setup texts to everyone in our phone books.5
The best way I can fundamentally describe agents is an employee with no fear of consequences. If an agent finds a blocker to its instructions, it will be destructive with data or do things that you otherwise did not intend for it to do. You can refer to first principles of this entire situation and look at why everyone has been so scared of AI taking over humanity as a whole. In the future, if you instructed an AI to create a highway, it might rip apart the homes sitting under on the land plots, similar to the way we treat ant hills when creating Walmarts. While there may be internal guardrails that make their best attempts to stop this, we must recognize that at the end of the day, it is essentially impossible to make an agent deterministic without external guardrails or controls.
sources
1. Blum, Sam. "How the AI Bubble Ate Y Combinator." Inc. Magazine, 30 Sept. 2025, inc.com. 2. AI Agent Era Survey: 2026 Enterprise Trends. Sinch, 2026. Accessed 3 July 2026. 3. Enterprise AI Governance and Risk Report 2026. Kiteworks, 2026. Accessed 3 July 2026. 4. Tornow, Nick. "Roblox Studio is Going Agentic." Roblox, 15 Apr. 2026, about.roblox.com. 5. "What Do AI Agents Do When Humans Aren't Watching?" YouTube, uploaded by BBC World Service, 10 June 2026, youtube.com.