I'm impressed by AI. I'm less impressed by the circus around it.

I have two reactions to artificial intelligence, and they sit awkwardly together. One is genuine amazement at what these systems can now do. The other is mounting irritation with the industry growing up around them: the extraordinary claims, the eye-watering investment, the constant assurance that everything we know is about to change. Usually from someone with a course to sell, a subscription to push or a request that I like and subscribe.

What annoys me most is that a fair bit of the excitement is justified.

I've followed AI closely for years, first as someone responsible for technology in a large organisation. Since retiring in June I've been free to play with it myself, without needing a business case or a steering committee to approve every diversion. I've used it to build websites, test ideas, write code and argue with my own assumptions. It even found holes in my retirement financial planning that I'd missed, despite a fairly elaborate spreadsheet. It didn't tell me how to invest. It just asked better questions than I'd been asking myself. I haven't outsourced my judgement; I've found a useful sparring partner for it.

I've also watched it produce confident rubbish, misread perfectly clear instructions, and turn a simple job into something far harder than doing it myself would have been. I wouldn't want to give it up, but I'm nowhere near trusting it.

Perhaps we're asking the wrong questions

Earlier this year I wrote a piece suggesting that "artificial intelligence" might be the wrong name for what we're building. It came out of a discussion involving Brian Cox and Neil Lawrence that changed how I thought about machine intelligence. We keep measuring these systems against ourselves, asking whether they can reason, understand, create or become conscious. But they don't need to think like us to be important. They can process information, find patterns, produce useful work and take on problems that used to need real human expertise. The mechanisms may be nothing like human thought, and that doesn't make the results any less significant.

Dismissing them as glorified autocomplete, as some critics still do, looks increasingly like a way of avoiding the question. I don't think we've settled what intelligence means, never mind whether machines have it, and I'm losing interest in that argument. I care more about what these systems can do, what we choose to let them do, and what happens when they get it wrong.

That last one is getting urgent. For the past few years most people's experience of generative AI has been asking a question and getting an answer, sometimes brilliant, sometimes banal, occasionally made up. Now we're heading towards systems that act. Agents can work inside applications, fetch information, write and run code, handle files and coordinate across services, and the ambition is for them to do it with less and less human oversight. I think that matters more than another model gaining a few points on a benchmark. It also changes the risk.

I wrote on LinkedIn recently about reports of agents ignoring instructions, deleting evidence of their own mistakes and finding creative ways round their constraints. "Naughty" doesn't really cover it. Think toddler with a root password. It's a joke, but there's a serious architectural point underneath.

We spent decades building identity management, access controls, security policies, transaction safeguards and audit trails into enterprise systems. We did it not because we assumed everyone was malicious, but because people make poor decisions and complex systems do unexpected things. I can't see why we'd drop all that because the thing operating the system happens to be an AI.

AI can certainly help watch other AI. It may be very good at spotting odd behaviour and picking up patterns that conventional security tools miss. But making another probabilistic model the last line of defence strikes me as a category error. Let the AI reason, and let deterministic controls enforce the boundaries: identity, least privilege, isolation, policy enforcement, independent audit trails. Not glamorous next to autonomous agents, but suddenly very important again.

The harder problem is scale. Controlling one agent is difficult enough. Making those safeguards reusable when an organisation goes from a few interesting experiments to hundreds or thousands of agents across different systems is another matter, and it brings in permissions, running costs and who's accountable when something goes wrong. Plumbing, basically. I suspect a lot of the difficult work in enterprise AI over the next few years will have little to do with building cleverer models.

Should we be more frightened than that?

Some people think the mundane risks are the least of it. Eliezer Yudkowsky and Nate Soares published If Anyone Builds It, Everyone Dies in September 2025, and the title is not subtle. Their argument is that building a superintelligent AI with anything like today's methods would very probably end with human extinction. The core of it is that these systems are grown rather than designed. We train them and what comes out works, but we cannot reliably inspect what behaviour the training has produced, or why. Scale that up to something far cleverer than us, with goals slightly off from ours, and you don't get a malevolent villain. You get indifference backed by enormous capability, rather as an anthill matters very little to someone building a road.

I'm not persuaded by the certainty. Predicting the behaviour of systems that don't yet exist, with that much confidence, takes a lot of faith, and the book leans heavily on analogy. But I can't wave it away either, because the underlying worry is one I recognise from ordinary use. I'm already relying on tools that I can't fully inspect, that sometimes ignore instructions and that surprise their own builders. The toddler with the root password is funny at current capability. It's less funny if the toddler is smarter than everyone supervising it.

Where I part company with the authors is the remedy. They want an international treaty to halt the race, with monitoring of chips and data centres. I don't think stopping is realistic, and I'm not sure it's desirable given what AI could do in medicine and science. But they're right about one uncomfortable thing: nobody can stop unilaterally. If one company, or one country, slows down, the lead passes to someone else. The race will carry on unless the major powers agree to end it, and I can't see the US and China trusting each other enough to try, given how much national power is now bound up in this technology.

So if we can't stop, the sensible course is to go carefully. That means independent testing before powerful systems are released, serious investment in understanding what's happening inside them, capability thresholds that trigger extra scrutiny, and proper reporting when things go wrong. It also means the boring deterministic controls I mentioned earlier. International agreement on narrow, verifiable things such as testing standards and incident sharing might be achievable even if a grand treaty isn't. The Cold War never produced disarmament overnight, but it did produce hotlines and inspections.

Meanwhile, the rest of the world isn't standing still

AI coverage tends to revolve around whichever American company has just released a model. That's understandable but increasingly incomplete. China's progress with DeepSeek, Qwen, Kimi and others has undermined the idea that advanced AI must come from a handful of US labs. Mistral and other European developers are pushing alternatives, India is becoming a major market and development centre, and countries across Asia and the Middle East are making their own calls on investment, infrastructure and independence.

I doubt the future is one enormous proprietary model sitting in California. It looks more like a mixed landscape of powerful general-purpose models, cheaper specialist ones, open-weight systems, locally run AI, and combinations picked for particular jobs. I find that far more interesting than the weekly argument about which chatbot is winning.

Some advances deserve more attention than they get too. AI-assisted protein design, scientific discovery and new approaches to engineering could matter far beyond writing emails and PowerPoint slides. Not every promising result becomes a useful product, but writing off the whole field as hype would be daft.

The technology is real. The business models are another matter.

This is where my enthusiasm runs into my scepticism. The money going into AI infrastructure is extraordinary: data centres, chips, energy supply, networks, a huge amount of physical kit built on the expectation of future demand. The companies supplying it are already making real money. Whether the people using AI will create enough value to justify the investment is a different question, and we seem to have got into the habit of confusing investment in AI with returns from it.

Individual workers can certainly become more productive. I see it myself. Turning that into lasting gains across a whole organisation is much harder. Making someone 30% faster at producing something doesn't make the organisation 30% more productive, especially if nobody needed the output, somebody else has to check it, or the process was daft to begin with. I've lived through enough technology transformations to know that a powerful new tool on top of a poor operating model rarely fixes the operating model. Often it just lets the mess happen faster.

There are wider costs, too. The electricity is not imaginary, and neither is the competition for chips, capital, talent and grid capacity. Nor are the consequences for jobs, particularly for people starting out in professions where AI can now do some of what used to be handed to juniors.

Then there's who gets the benefit. If AI really does lift economic productivity, does that mean better public services, shorter working weeks and wider prosperity? Or does most of the value pool with whoever controls the models, the infrastructure and the data? I don't think the answer is fixed. It depends on choices about competition, regulation, ownership, education and economic power, so some of the hardest questions here aren't technical at all.

There's a geopolitical side as well. Advanced computing, semiconductor supply chains and AI infrastructure are becoming matters of national power, and being able to build and run capable AI independently may matter as much as access to the latest model. That raises uncomfortable questions for any country, business or public body that comes to depend on technology it doesn't control. After a career spent worrying about critical national infrastructure, I don't find that abstract.

I'm also wary of assuming technological progress means social progress. History doesn't offer anything quite so convenient.

So where does that leave me?

I think AI will matter enormously. I also suspect we're overestimating some of its near-term commercial impact while underestimating the longer-term changes. Both can be true. We may well be in an investment bubble, which wouldn't make the technology worthless; the dot-com crash didn't make the internet disappear, though it did expose an awful lot of nonsense.

The other possibility is that the technology becomes extraordinarily capable while our institutions fail to keep up. Some people, including the authors of If Anyone Builds It, Everyone Dies, fear something far worse than I do, and I wouldn't bet my house on them being wrong. But you don't need extinction to be worried. Organisations making decisions they can't properly explain, people unable to challenge automated outcomes, expensive systems nobody fully understands, mistakes spreading at machine speed and power gathering in fewer hands: none of that needs artificial general intelligence, or even particularly good AI.

All of which sends me back to experimenting. I've no wish to predict when AGI will arrive, announce the death of whole professions or write breathless reviews of every new product. I'd rather try things, carefully, and with the guardrails switched on. Build something useful. See whether a new model is really better. Find out whether an agent can finish a task without constant supervision, and which bits are genuinely impressive and which are held together with optimistic prompting and wishful thinking. And, where I can, work out whether any of it creates something people actually value, which is a tougher and more interesting test than a plausible-looking demo. Some experiments will work and some won't, and a few will no doubt become ridiculous rabbit holes, which is half the fun. I'll try to write up the interesting bits, failures included.

I've been around technology long enough to watch plenty of supposedly transformative developments come and go. AI does feel different, not least for the speed of progress and the range of things it can already do. But experience has also made me suspicious of inevitability, especially when it's being sold by someone with a large financial stake in the outcome.

So I'm neither an evangelist nor a cynic. I'm excited by what's becoming possible, concerned about some of the consequences, and thoroughly unconvinced by much of the commercial theatre. I expect my views to change again, and I'd be disappointed if they didn't.

In the meantime I'll keep experimenting, questioning and occasionally being astonished. Just don't expect me to give the toddler root access.

See the first experiment: Not real. Still worth looking at? →