In some ways, Gartner’s annual IT conference was a departure from previous years. There was a lot more discussion of disappointing returns on AI investments thus far, alongside suggestions for how enterprises should plan for AI playing a more important role over the next few years.
Gartner analysts Alicia Mullery and Daryl Plummer acknowledged that it’s easy to dismiss the impact of AI since most firms have yet to see the transformative impact. However, they cautioned against falling for the hype and instead backed a “golden path” whereby value can be found in between those two extremes.
Looking for Return on AI Investment
In 2025, Gartner put the odds of an AI initiative achieving a return on investment at one in five, Mullery said. The odds of an AI initiative achieving true transformation was just one in 50.
Plummer noted that a recent study of chief financial officers showed that 74% say they are seeing productivity gains, but only 11% are seeing return on investment.
AI Readiness Positioning (Credit: Michael J. Miller)
To counter this, Mullery and Plummer proposed a new “positioning system” looking at both the readiness of AI systems and the readiness of humans to take advantage of that. When neither is ready, Plummer said, you should be highly skeptical of AI. When both are ready, “we flip a switch into AI optimism.” Right now, he said, AI systems are almost halfway up the scale, but humans are only a quarter of the way. This makes value hard to achieve.
The Need for an ‘Accuracy Survival Kit’
Alicia Mullery (Credit: Michael J. Miller)
Mullery noted that most AI systems currently have questionable readiness; for instance, generative AI systems have an error rate of up to 25% depending on the use case. That may be good enough for some applications, but not all. Worse, she said, Gartner surveys suggest that 84% of CIOs and IT leaders say they don’t have a formal process to check accuracy, beyond having a “human in the loop.”
“AI can make mistakes faster than we humans can catch them,” she said, calling on IT leaders to create their own “accuracy survival kit” with things such as formal metrics, a two-factor error checking, and a “good enough ratio.” This is “harder to achieve than you think,” she argued.
AI Agents Are Not Created Equal
Daryl Plummer (Credit: Michael J. Miller)
Plummer noted that AI agents “are at the top of the hype cycle” now, with 17% of organizations saying they have adopted agents already and another 42% expecting to adopt them in the next 12 months. “But not all agents are created equal,” he said, noting that 88% of the IT leaders are focused on conversational agents, not ones involved in decision making.
For that, he said, we’ll need multi-agent systems, with reasoning and autonomous decision making that can actually perform the “jobs to be done” such as one that could create the slides for a presentation by just listening to him talking.
This kind of agent won’t be cheap. Mullery noted that the big problem is that even if you think you know the costs of running AI on day one of the project, you don’t know what it will cost on day 100, unlike most IT projects. In fact, 74% of organizations are breaking even or even losing money on their AI investments. That’s partly because they don’t recognize the ancillary costs, such as managing access credentials, acquiring new data sets, or accuracy tools, as well as time spent on training and change management, Plummer explained.
Picking the Right Vendor
Then there is the issue of picking the right AI vendor. Plummer noted that large AI vendors are acting like “Digital Nation States,” spending more on AI infrastructure per quarter than the annual GDP of 47% of the world’s countries. Mullery suggested that if you’re planning a massive rollout of AI to your enterprise, you should bet on a major hyperscaler such as Microsoft, Google, Amazon, Alibaba, or Oracle. But if you want industry-specific use cases, you’ll need to partner with startups. And for leading edge capabilities, you should look to “wild card” vendors such as OpenAI, Anthropic, Meta, DeepSeek, and Mistral AI.
Within this is also the question of “AI sovereignty”—not just the data, but also the models, results, and users. Plummer said the US, China, and the European Union are building sovereign AI solutions with proprietary models that they control and predicted that 35% of countries will be locked into region-specific AI platforms using proprietary contextual data by 2027. He warned that if you’re a global enterprise, you might get locked out of a long-term partner relationship, so he suggested big companies look into digital tokenization solutions that anonymize data, so the real data doesn’t leave your shore.
Job Chaos, Not Job Loss
When it comes to human readiness, Mullery noted that 71% of CIOs and IT leaders say their workforce is not ready for AI “because AI unleashes a toxic mix of steep learning curve and the primal fear that AI is going to replace us.”
But despite the scare headlines, while 60% of CFOs are expecting they can reduce head count due to AI, in reality only about 1% of job loss is due to AI today. “It’s not about job loss. It’s about job chaos,” Plummer said, noting that job and role restructuring is having 20 times more impact than layoffs. Instead, he said, Gartner is seeing hiring restraint for junior level jobs.
Get Our Best Stories!
Your Daily Dose of Our Top Tech News
By clicking Sign Me Up, you confirm you are 16+ and agree to our Terms of Use and Privacy Policy.
Thanks for signing up!
Your subscription has been confirmed. Keep an eye on your inbox!
So instead of looking for a “talent remix,” Mullery urged executives to think about a “value remix” —doing things such as cutting the backlog and increasing roles that require human empathy. While there may be some layoffs, demographic shifts and labor force stagnation will balance it out.
“By 2027, AI will create more jobs than it destroys. But the sad fact is, there won’t be enough new people to hire,” she said. So, the best thing you can do is train your workers to use AI well, with one intent being creating “Swiss army knife” workers who can pick the tools to solve any given task. For that, she said, you need AI literacy and experiential knowledge. They both discussed how it was important to be on the lookout for behavioral changes driven by AI.
Perhaps no department is as likely to see disruption as IT itself. “In 2030, your IT organization will look completely different,” Mullery said.
Recommended by Our Editors
According to a Gartner study of CIOs, 81% of IT work is done by humans without AI. By 2030, CIOs expect 75% of IT work will be done by humans augmented with AI, and 25% will be done by AI alone. Zero percent of IT work will be done without AI.
This will change your entire IT estate, she said, and create extra capacity, so it’s up to leaders to demonstrate extra IT value. One issue Plummer identified was using this extra capability to make up for worker shortages caused by the demographic changes.
Other changes will impact the business more directly, creating what Plummer called “shock waves.” For instance, he said, as AI takes over more diagnosis and virtual care allows more recovery at home, hospitals will become primarily treatment centers.
Today, we are gradually adding machine autonomy around the edges of our organizations, Mullery said, moving us towards autonomous business. This is the new “agentic battleground” where multiple vendors compete to deliver teams of machine workers ready at a moment’s notice.
But Plummer insisted that “If we keep fighting to protect our roles of the past, we’ll never reimagine our roles for the future.” He said the day may come when we collectively decide we will choose not to do things like managing operations or configuring system and instead choose to have people primary nurture human connectedness. “We decide what humans will do and what AI will,” he said.
AI Daryl (Credit: Michael J. Miller)
He even showed a digital twin of himself – AI Daryl – that could present ideas the way he would when he’s not available. Such technology might uncover value hidden in the data you already have and prevent the “retirement brain drain” by training such systems on the documentation, code reviews, and meeting records of retiring employees.
“When humans and AI are ready, we can transcend and ride the AI shockwaves,” Mullery concluded. “Let’s bring it all together and retrace our path to find, capture, and sustain value. You have to get both Ai and humans ready.”
About Our Expert
Michael J. Miller
Former Editor in Chief
Experience
Michael J. Miller is chief information officer at Ziff Brothers Investments, a private investment firm. From 1991 to 2005, Miller was editor-in-chief of PC Magazine,responsible for the editorial direction, quality, and presentation of the world’s largest computer publication. No investment advice is offered in this column. All duties are disclaimed. Miller works separately for a private investment firm which may at any time invest in companies whose products are discussed, and no disclosure of securities transactions will be made.
Until late 2006, Miller was the Chief Content Officer for Ziff Davis Media, responsible for overseeing the editorial positions of Ziff Davis’s magazines, websites, and events. As Editorial Director for Ziff Davis Publishing since 1997, Miller took an active role in helping to identify new editorial needs in the marketplace and in shaping the editorial positioning of every Ziff Davis title. Under Miller’s supervision, PC Magazine grew to have the largest readership of any technology publication in the world. PC Magazine evolved from its successful PCMagNet service on CompuServe to become one of the earliest and most successful web sites.
As an accomplished journalist, well versed in product testing and evaluating and writing about software issues, and as an experienced public speaker, Miller has become a leading commentator on the computer industry. He has participated as a speaker and panelist in industry conferences, has appeared on numerous business television and radio programs discussing technology issues, and is frequently quoted in major newspapers. His areas of special expertise include the Internet and its applications, desktop productivity tools, and the use of PCs in business applications. Prior to joining PC Magazine, Miller was editor-in-chief of InfoWorld, which he joined as executive editor in 1985. At InfoWorld, he was responsible for development of the magazine’s comparative reviews and oversaw the establishment of the InfoWorld Test Center. Previously, he was the west coast bureau chief for Popular Computing, and senior editor for Building Design & Construction. Miller earned a BS in computer science from Rensselaer Polytechnic Institute in Troy, New York and an MS in journalism from the Medill School of Journalism at Northwestern University in Evanston, Illinois. He has received several awards for his writing and editing, including being named to Medill’s Alumni Hall of Achievement
Latest By Michael J. Miller
Read Full Bio
