TetianaKtv | shutterstock.com
Intelligent systems are increasingly being used productively. If this is the case, many companies quickly realize that this poses problems in terms of accountability. Ultimately, AI tools do not work like traditional enterprise software: their dynamic interaction with data, APIs and business workflows can produce unpredictable results.
David DuChene, manager at AI service provider SHI International, sums up the status quo for many users: “If something goes wrong with AI, responsibility is usually assigned to the person who was closest to the problem.”
And because AI systems are increasingly no longer taking on the role of advisor but rather that of actor in workflows, accountability can no longer be enforced through guidelines alone. Rather, it is the task of IT managers to integrate these directly into the structure of their operational processes. For example in the form of the following five measures.
1. Clearly define responsibilities
Various AI user companies (especially in the enterprise environment) are still convinced that AI accountability is a task for the entire workforce. However, experts argue that this assumption turns out to be false once the systems are used productively. For example, Joe Wilson, SVP and CIO at software provider CSG: “Shared accountability is not accountability. A direct owner is essential.”
According to the manager, this is how his employer operates. In addition, AI initiatives go through governance reviews in which management is also involved. In addition, CSG also relies on “CIO representatives” who are embedded in the business areas and product groups. In this way, according to its CIO, the software provider wants to ensure that accountability covers the entire life cycle of AI initiatives.
However, formalized accountability structures like this have so far been missing in most companies, DuChene notes: “On paper there may be those responsible, but as soon as a system actually fails, everything is renegotiated.”
According to the manager, whether organizations are really prepared when it comes to accountability can be clarified by asking a diagnostic question: “If your AI deployment generates an incorrect answer that costs the company money, who will write the postmortem? If managers cannot answer this question directly, accountability structures probably do not yet exist in practice.”
2. Engage governance in a timely manner
In recent years, many companies have adopted AI systems – before they had the necessary governance and operations foundations in place, according to DuChene: “The biggest problem we regularly see is related to the correct order of actions. In many cases, a lot of houses were built with the walls already up before the foundation was poured.”
This subsequently leads to costly retrofitting measures, says the manager: “The teams in these companies often discover far too late that important things such as data classification systems, AI-related IAM controls or escalation channels for errors are not in place.”
According to Seth Dobrin, CEO at AI modeling provider Arya Labs and former Global AI Leader at IBM, governance often fails because companies view it as a pure policy layer rather than integrating it directly into operational workflows. “If you don’t get this right, the whole thing will fall apart,” states the AI expert. He points to the example of an insurance company that spent 18 months building an intelligent system before the legal department paralyzed the deployment: “The problem wasn’t the technology itself, but that governance didn’t play a role in the early stages of the project. In the end, the company had to abandon the project.”
At this point, CSG manager Wilson argues that governance should support teams in companies to manage complexity rather than restrict their ability to act: “Governance is more of a chassis system than a braking mechanism. Things should go faster, but they also have to work when you get into rough terrain.”
In this context, data also matters, warns Quais Taraki, CTO of EnterpriseDB. Many companies regularly underestimate how difficult it is to maintain accountability once AI systems interact with fragmented data environments in the enterprise environment: “An AI assistant that summarizes customer interactions, for example, could retrieve regulated or confidential data from systems that were never intended for it.”
To prevent problems like these from occurring in the first place, strong data governance practices are essential. Specifically:
- Lineage,
- Provenance Tracking,
- Classification systems and
- Access controls.
In addition, these measures also create the basis for accountability should something go wrong. Because they make it possible to determine which data an AI system has accessed, how the outputs were generated – and also whether sensitive information influenced a decision. “Without data lineage and provenance, root cause analysis is impossible – meaning you don’t know what needs to be changed and you have no insight into how things have changed in unexpected ways,” Taraki says.
The CTO advocates aligning accountability with regulated data products – rather than organizational silos: “If ownership is distributed across infrastructure, data science and developer teams, it can be difficult to clarify who was responsible after a mistake. Clear responsibilities for the data products that power the AI systems help ensure accountability across the entire AI lifecycle.”
3. Integrate observability
Classic enterprise monitoring systems are primarily designed to monitor availability, infrastructure health and application performance. However, artificial intelligence raises new challenges: these systems require tracking of reasoning paths, decision chains and behavioral deviations.
Nik Kale, member of the Coalition for Secure AI (CoSAI), recommends a so-called “Investigation Graph”. This could provide information about what an AI system observed, what tools it accessed, what conclusions it reached and what actions it ultimately took. “When something doesn’t work, the first impulse is always to ask why the AI made this decision. But you should rather ask how the system actually acted – after all, it’s not the AI model that acts, but the system around it.”
This broader accountability perspective is also changing the perception of observability: Instead of monitoring AI models in isolation, companies increasingly need visibility across all the systems they interact with – including data sources, APIs, applications, security controls and downstream workflows. In practice, this means keeping comprehensive records – namely:
- Prompts,
- Outputs,
- tool calls,
- Data access events as well
- The agent action.
Combined with traditional application and infrastructure telemetry, this creates auditable logs of how AI systems behaved and why decisions were made.
This transparency becomes particularly important when IT managers are dealing with unauthorized AI use. While governance policies define which tools employees should useObservability helps uncover which tools they actually use. By extending observability beyond AI models to the entire enterprise environment, IT can detect shadow AI earlier, investigate it more quickly, and close the accountability gaps it creates.
4. Establish escalation mechanisms
The most important accountability question, however, is when an AI system should stop and ask for help. Unfortunately, in AI expert Kale’s experience, this is an area that receives little attention in most AI implementations in companies. The manager argues that companies need explicit escalation paths, human decision points and clearly defined stopping mechanisms for AI systems that are used in productive operations: “A ‘nod’ would be out of place here – keyword ‘human in the loop’. This person should be named and also have the authority to stop the AI.”
According to CIO Wilson, such emergency stop mechanisms are also necessary with regard to incident response: “A conventional IT incident is typically an ‘up’ or ‘down’ scenario. AI failures are a little more subtle: models can develop drift over time or workflows can deliver results that are unexpected but not technically noticeable.”
According to the manager, this results in a growing need for interdisciplinary response processes in which legal, communications, security, audit, business and IT operations teams are involved in parallel.
5. Treat AI like employees
Traditional software can often be reviewed and approved at the time of release because its behavior remains relatively stable between versions. This is different with AI systems: models evolve, prompts change, retrieval systems are updated, and the information available to agents also changes continuously. However, quite a few companies still view AI like traditional applications.
However, Kale believes that the technology behaves less like deterministic software and more like employees: “You can’t just deploy AI once and leave it at that. Similar to the workforce, a certain level of oversight is needed here – for example in the form of performance reviews, feedback rounds or to put a stop to deviant behavior.”
This challenge goes beyond internally developed systems, as the IT decision maker notes: “Companies also need to keep an eye on the third-party AI services they rely on, as they may update software and functions behind the scenes.”
In order to clarify the responsibilities between user companies, software and model providers and infrastructure operators, Kale refers to CoSAI’s “AI Shared Responsibility Framework” (PDF). (fm)
This article is im Original published by our sister publication Computerworld.com.
