Who’s responsible when AI acts on its own?
This article, written by Anthony Diaz, CISO at Exterro and a Foundry Expert Contributor, originally appeared in CIO in October.
Rethinking accountability before agentic systems go live.
When an AI acts independently, such as executing trades, approving loans or negotiating contracts, the question isn’t just what went wrong, but who’s responsible.
That question is becoming more urgent as AI shifts from advisory to agentic systems that plan and execute multistep tasks autonomously. When those agents go off-script, accountability can’t be an afterthought.
As both CIOs and CISOs are aware, the answer hinges on two key factors: control and intent. Understanding where these lie across the AI life cycle: development, deployment and oversight, is the key to managing risk, providing due diligence and avoiding costly regulatory exposure.
Where liability starts and how it shifts
At the start of any AI system’s life, responsibility sits with the manufacturer or developer. Their obligations are foundational: secure coding, safe model training, robust testing and transparency about limitations. If an AI acts harmfully due to flawed training data or unsafe design, the defect liability begins there.
But the risk rapidly transfers once the enterprise deploys that AI. The deploying organization owns the operational context, its policies, oversight and configuration decisions. If an autonomous trading bot overextends its portfolio because internal governance failed to cap exposure, that’s an enterprise failure, not a vendor defect.
This is where most risk lives today: in the gap between what vendors deliver and how enterprises govern it.
IBM’s 2025 “Cost of a Data Breach” report reveals that AI is outpacing security and governance, driving do-it-now adoption. The findings show that ungoverned AI systems are more likely to be breached and more costly when they are. According to the report, 63% of organizations lack AI governance policies to manage AI or prevent the proliferation of shadow AI.
From a CISO’s standpoint, three emerging liability gaps matter most:
- The trust and control gap: Weak oversight that allows autonomous damage.
- The audit trail gap: Inability to explain or reconstruct AI decisions.
The third-party gap: Vendor interactions that create unclear fault lines.
Reframing accountability across the AI life cycle
CIOs and CISOs can’t treat accountability as a single point of failure. It must form a chain of ownership that follows the AI through its ModelOps life cycle.
Data owner (input stage)
Responsible for data integrity and bias in training datasets. Poor data lineage creates foreseeable harm. Each AI should have an AI factsheet documenting its data sources, bias testing and governance approvals. This is a best practice reinforced in the NIST AI Risk Management Framework.
Model owner (business stage)
The line-of-business leader using the AI must own the business outcome — and any resulting harm. Before deployment, the model must undergo adversarial testing to validate safety guardrails. According to a recent survey, 82% of organizations say they’re using AI across functions, yet only 25% report having a fully implemented AI governance program.
Control owner (oversight stage)
This role is accountable for ongoing monitoring, drift detection and escalation. This directly addresses IBM’s trust and control gaps. Leading enterprises are formalizing a cross-functional AI governance committee (AIGC), jointly led by CIO, CISO and legal, to ratify high-risk use cases and assign oversight responsibility.
How do you operationalize trust and control? Reframing accountability also means translating governance into enforceable technical controls:
- Least privilege for AI: Just as humans don’t get admin rights by default, agentic systems must operate with minimum necessary access. If a customer-service bot can alter financial records, that’s not an AI failure — it’s a security policy failure.
- Explainability as a legal control: For high-impact use cases (hiring, lending, healthcare), explainability isn’t optional; it’s evidence. IBM’s AI governance principles emphasize that audit trails and decision logs are now integral components of compliance.
Proving due diligence when AI causes harm
When an autonomous system acts independently, “we had a policy” won’t satisfy regulators or courts. Due diligence now requires proof: documented evidence that governance was operationalized before the harm occurred and that controls were functioning during it.
Proof 1: Pre-condition governance
Show that the AI was classified by risk and autonomy level, approved by the AIGC and red-teamed for vulnerabilities. High-risk systems (such as financial, medical and legal) require continuous monitoring and clear human accountability before deployment.
Proof 2: Control effectiveness
Demonstrate that safety constraints were technically enforced, such as logs showing least-privilege restrictions, drift detection and human override mechanisms (e.g., kill switch) working as intended.
Proof 3: Post-action audibility
Maintain explainable logs that reconstruct the AI’s reasoning chain. Regulators are already moving in this direction; both the U.S. and EU proposals expect documentation proving “reasonable organizational behavior.” Insurers, too, increasingly require forensic justification before covering AI-related losses.
Balancing innovation with liability: Sandboxes and kill switches
Despite the liability fears, enterprises aren’t halting innovation; they’re reframing it. Most organizations are testing agentic AI in low-risk, high-value domains: customer experience, knowledge summarizations, and internal automation.
A recent survey found that 44% of organizations plan to implement it within the next year to save money, improve customer service, and reduce the need for human intervention.
To manage exposure, they’re adopting what I call the constrained autonomy model:
- Sandbox first: Agentic AI runs in a closed environment with no production-write access until validated.
- Role-based access control (RBAC): AI is treated like a new employee with limited scope and supervised duties.
- Kill switches: Mandatory, human-triggered stop mechanisms that work even if the AI’s own systems fail.
- Tiered autonomy: Agents may process refunds of up to $500 autonomously, but any amount higher is routed to human review.
The goal is to earn the right to innovate safely, demonstrating quick ROI while building the governance muscle memory for higher-risk deployments.
Consumer AI: The liability squeeze
In consumer-facing applications, the brand (the deployer) bears immediate accountability. The vendor may be legally liable for core defects, but the brand owns the customer relationship and the headlines.
Vendors face growing pressure under evolving frameworks, such as the EU’s proposed AI Liability Directive, which expands the definition of “product” to include software. The courts are effectively splitting fault: model-level defects belong to the vendor; deployment-level mismanagement belongs to the enterprise.
CIOs and CISOs must plan for both by enforcing AI responsibility clauses and audit rights in vendor contracts. Liability caps should scale with risk — no more blanket limits tied to subscription fees.
Contracts and SLAs: The new risk-allocation toolkit
AI liability is now as much a contract problem as a technical one. SLAs must evolve beyond uptime and performance guarantees to measure trust, safety and drift.
- Bias and data warranties: Require vendors to certify the integrity and fairness of their training data.
- Audit and transparency rights: Mandate access to model documentation and decision logs upon failure.
- Incident response SLAs: Define vendor response times and obligations for AI-specific breaches or autonomous misbehavior.
Leading legal experts are calling these “AI Responsibility Clauses,” i.e., contractual language ensuring accountability from pre-deployment through post-incident investigation. In the next two years, we’ll see measurable accountability. AI liability norms are entering an enforcement era marked by four irreversible shifts:
- Model- vs. deployment-level fault: Courts will split liability between vendor defect and enterprise misuse.
- Regulatory fragmentation: The EU AI Act will set the global compliance floor, while U.S. states adopt sector-specific laws.
- Financialization of AI risk: Insurers will price policies based on governance maturity, not revenue size.
- Mandatory explainability: “Black box” defenses will collapse. Audit logs and chain-of-thought documentation will become the new regulatory minimum.
The FTC’s Operation AI Comply and global regulatory momentum are clear signals: AI risk management is no longer optional; it’s an enterprise control discipline. CIOs and CISOs must embed governance not as a compliance overlay, but as an engineering function that spans data, model and control layers.
by Anthony Diaz
Anthony Diaz has a two-decade career as a security and data risk expert working with Fortune 500 organizations. He currently serves as CISO at Exterro and was previously a managing director for US Cyber & Strategic Risk at Deloitte. Anthony held senior security leadership roles at Merrill Lynch, Johnson & Johnson, IBM, Optiv Security, Ernst & Young and HSBC.