Last year felt like the Year of the AI Pilot. Companies bought LLM subscriptions, managers checked on employee usage, and coffee chats abounded with the “AI wrote my memo” motif.
Looking around today, there is widespread disappointment with the impact of these AI pilots. Add to this the recent sell-off in SaaS stocks, and the question is no longer “Are we using AI?” but rather “Is this thing working?”
AI is an invention that is in the process of becoming an innovation. An invention is a new capability; it is not an innovation until it has a business model. In that light, experimentation last year was the sensible move.
It is becoming clear now that the form that innovation takes will be AI systems trusted with real decisions — what Peter Drucker would call executives, and what are today referred to as agentic AI.
As we turn to the question at hand, Is this thing working?, we can look to one of Drucker’s intellectual disciples for a framework to take us forward. Andy Grove, the legendary former CEO of Intel, turned Drucker’s writings into a hard-nosed, pragmatic approach to managing knowledge-worker organizations. His book, “High Output Management,” provides the classic framework for measuring the outputs of middle managers. This is not an easy thing to measure. But Grove is relentless in insisting it can and must be measured.
As we address the question of whether AI agents are delivering tangible value, we have to shift our focus away from activities, anecdotes and initiatives. These are inputs.
Grove argues that organizations must instead focus on outputs. If we try to think like Grove, we would first define the business outcome we wanted to achieve, and then measure our agentic AI only by whether this performance metric is better.
A mathematical approach
As we began working on this several years ago across our software portfolio, I had the great good fortune to meet Dario Fanucchi, a mathematician who was using AI to solve real-world problems in a very similar way. He is also co-founder and CTO of Isazi 1, a decade-old, 70-plus-person team of mathematicians and engineers who have completed hundreds of projects for leading companies around the world.
His approach to these has a singular focus: improving core business metrics.
Isazi came to the same idea of measuring outputs, although starting from the field of mathematics rather than organizational behavior. The idea is to approach AI projects as though they are mathematical optimization problems: Define a target measure (such as throughput or working capital), ask what variables influence that metric, and model the mechanism by which the target measure is moved.
Then all initiatives are aligned to this target measure, and success is measured by its improvement. This aligns well with how AI models are built and improved: benchmarks and evals are always the core measure of success. Here, these evals are directly aligned to business metrics.
You must begin with the output you want to measure. And then you watch that output measurement, as a gauge, and see how long it takes until that gauge is reading changes, how much it changes, in what direction, and whether it sustains.
The time it takes to see (and sustain) a material movement is called “Time To Production.” Our theory on why so many pilots fail is that companies tend to pick an AI tool and a pilot duration and qualitatively check in with users at the end of that time.
While we at Strattam and Isazi appreciate experiments and pilots, we have found that results are best when that process is reversed. We choose the output we want to see improved, vary the AI tools until one moves the dial, and measure the time it takes to change the output positively and in a sustainable way. The shorter the Time To Production, the better.
A real-world example
Let me share an example.
One of Strattam’s portfolio companies, Trax Technologies, is in the business of helping very large multinationals manage their global shipping. A key part of the offering is ensuring that freight bills are complete, match the contract, are approved for payment, and are properly accounted for.
Trax works across all geographies and all shipping modes, with thousands of carriers. Discrepancies between the bill and the shipper contract are common. Handling those “exceptions” at scale is a key part of the service, and historically, Trax has had a large in-house team that resolves those.
In 2024, it identified AI’s ability to resolve some of those exceptions as a key opportunity and developed the AI Audit Optimizer in-house. The output goal was clear: the fraction of exceptions resolved without human intervention.
The first quarter after its release, the Trax AI Audit Optimizer resolved some 826,000 exceptions that otherwise would have required human intervention. That was a good start, but not worth writing home about just yet.
In Q2, however, the system remained stuck at that same level, rather than improving. So Trax rapidly experimented to see what would improve outcomes. In Q3, the company discovered that a human prompt engineer interacting with the system made a big difference. As a result, in Q4, resolved exceptions tripled to 2.5 million.
Now we’re talking.
With the output gauge firmly in mind, Trax is moving forward by adjusting interaction points of the prompt engineer and the system. It used data from successful and unsuccessful resolutions to retrain the system. The company also set quarterly goals; next quarter, it will aim for the Trax AI Audit Optimizer to resolve more than any previous quarter.
This story shows how studying an output gauge allowed the company to tune and adapt the AI tooling to deliver the outcomes that actually matter. Trax is intent on fixing its customers’ problems so it can earn market share. Its use of AI helped it do that, and its output measurements prove the real-world value of the AI innovation.
Measure what matters
Amidst all the hype, we all care that our companies actually adapt, actually deliver customer value, and actually succeed. We know that we cannot keep doing what we are doing as we have been doing it, that our futures may well depend on our ability to adapt. But this is different from actually adapting.
To adapt successfully, resist the urge to buy tools and run pilots and tell anecdotes and report on activities. Those are just inputs. Instead, determine the outcome measurement that matters, and watch it like a hawk to see if AI is delivering cold hard business results. If it’s not, change your AI until the dial moves. Drawing on the time-tested wisdom of Drucker and Grove in this way, you’ll ensure AI earns its keep at your firm.
Bob Morse co-founded Strattam Capital in 2014 and is managing partner. He has served on numerous private and public technology company boards, and currently is a director of CloudHesive, Contegix, Daxtra Technologies, Green Security, Resource Navigation and Trax Group. Previously, he was a partner and member of the investment committee at Oak Hill Capital Partners. He also worked at GCC Investments and Morgan Stanley. Morse serves on the board of directors of Austin PBS and as member of the advisory board for the HMTF Center for Private Equity Finance at The University of Texas at Austin McCombs School of Business. He attended Princeton University, graduating summa cum laude with a B.S.E., and Stanford Graduate School of Business, where he earned his MBA and was an Arjay Miller Scholar. Morse lives in Austin.
Dario Fanucchi contributed to this article. He is chief technology officer at Isazi, a Johannesburg-based applied artificial intelligence firm purpose-built to deliver production-grade AI software solutions for clients. Fanucchi has excelled academically in the fields of computer science, mathematics and physics throughout his career.
Related reading:
Illustration: Dom Guzman
Stay up to date with recent funding rounds, acquisitions, and more with the
Crunchbase Daily.
