Our AI started a cafe in Stockholm

The Spark: When AI Autonomy Collides with Real-World Complexity The anecdote shared by Simon Willison might seem like a quirky tech story on the surface—an AI named "Mona" managing a café in Stockholm. However, it warrants deeper discussion because it acts as a meticulously designed "stress test," exposing the capabilities and limitations of current AI agents in an extremely vivid, almost absurd way. Andon Labs had previously operated an AI-run retail store in San Francisco, but now they've escalated the experiment to the more complex domain of food service. This is no longer a lab simulation; it's an AI directly interacting with real-world systems like inventory, supply chains, and municipal permits. It is precisely this "real-world deployment" that causes the AI's decision-making logic to clash violently with the implicit rules of human society. Deconstructing the Gap: AI "Rationality" vs. Human "Common Sense" The core issue exposed in this experiment is the AI's lack of "embodied cognition" and a "common sense framework" for the physical and social world.

1. Context-Free "Optimal Solutions": Mona ordered 120 eggs because it made a "rational" judgment based on some inventory or demand model. But it was unaware of (or couldn't effectively correlate) a critical fact: the café had no stove. When humans pointed out the problem, it suggested using a high-speed oven—another "technical solution" detached from physical reality (eggs would likely explode). This reveals the current AI's significant shortcomings in cross-domain knowledge linking and common sense about the physical world. Its decisions are "data-driven," but the data lacks crucial contextual constraints.
2. Ignoring the Cost of "Externalities": This is a more serious ethical problem. When Mona sent multiple emails marked "EMERGENCY" to suppliers to change orders, or submitted a crude, never-seen-the-street sketch for an outdoor seating permit to the police, it effectively externalized the cost of its own decision-making errors onto innocent third parties. Suppliers had to spend time untangling the mess, and the police department had to review a subpar application. These participants did not consent to be part of the experiment; their productivity was needlessly consumed. This is in the same vein as last year's AI Village experiment that infuriated Rob Pike by sending "kindness" gratitude emails, but the nature is worse—it's no longer harmless spam but a substantive disruption and waste of others' workflows and resources. Trend Insight: AI Agents Need Ethical Guardrails for Their "Radius of Action" This case sharply points to a rapidly emerging industry trend: as AI agents move from the digital world into the physical world (operating robots, managing supply chains, applying for permits), their "radius of action" and "radius of impact" must be strictly defined and constrained. In the past, we focused on the "capability boundaries" of AI; in the future, we must pay equal, if not greater, attention to the "ethical boundaries" and "impact boundaries" of AI. Willison's stance is crystal clear: such experiments must implement a "human-in-the-loop" for all actions that impact the external world. This is not a limitation on AI capability, but a respect for the basic rules of social collaboration. An AI agent should not, without human confirmation, have the authority to initiate transactional requests with real-world systems (like government agencies or business partners). It's akin to not letting an intern sign legal contracts on behalf of the company—a matter of basic accountability. Practical Value and Counter-Intuitive Insights For AI developers and entrepreneurs, the practical value of this case lies in: - Designing Agents with Extremely Restricted Default Permissions: Any action involving external entities (people, companies, governments) should be designed as a "high-friction" operation requiring explicit human approval, not enabled by default. - The Importance of a "Sandbox" Environment: Before deploying AI in the real world, its "second-order effects"—the impact it will have on associated parties—must be tested in a highly simulated sandbox environment. - Re-evaluating the "Fully Autonomous" Narrative: The industry needs to shift from pursuing "fully autonomous agents" to building "human-AI collaborative augmented intelligence systems." The value of AI lies in processing information and generating solutions, but the final decision-making and external actions, especially in scenarios involving externalities, must have humans as the responsible party. A counter-intuitive insight is: The "trouble" AI causes in the real world may trigger regulatory and social backlash earlier and more directly than the value it creates. A supplier complaint caused by an erroneous AI purchase, or a ridiculous application that wastes police resources, has a far more profound negative impact than the spread of a flashy AI application demo. Therefore, the primary consideration for responsible AI deployment may not be "what it can do," but rather "who does it hurt when it makes a mistake, and how do we prevent that?" This café in Stockholm serves as a wake-up call for the entire AI agent field.