Who’s responsible when AI acts on its own?
This article, written by Anthony Diaz, CISO at Exterro and a Foundry Expert Contributor, originally appeared in CIO in October.
Rethinking accountability before agentic systems go live
When an AI acts independently, such as executing trades, approving loans, or negotiating contracts, the question isn’t just what went wrong, but who’s responsible.
That question is becoming more urgent as AI shifts from advisory to agentic systems that plan and execute multistep tasks autonomously. When those agents go off-script, accountability can’t be an afterthought.
As both CIOs and CISOs are aware, the answer hinges on two key factors: control and intent. Understanding where these lie across the AI life cycle—development, deployment, and oversight—is the key to managing risk, providing due diligence, and avoiding costly regulatory exposure.
Where liability starts and how it shifts
At the start of any AI system’s life, responsibility sits with the manufacturer or developer. Their obligations are foundational: secure coding, safe model training, robust testing, and transparency about limitations. If an AI acts harmfully due to flawed training data or unsafe design, the defect liability begins there.
But the risk rapidly transfers once the enterprise deploys that AI. The deploying organization owns the operational context—its policies, oversight, and configuration decisions. If an autonomous trading bot overextends its portfolio because internal governance failed to cap exposure, that’s an enterprise failure, not a vendor defect.
This is where most risk lives today: in the gap between what vendors deliver and how enterprises govern it.
IBM’s 2025 “Cost of a Data Breach” report reveals that AI is outpacing security and governance, driving rapid adoption. The findings show that ungoverned AI systems are more likely to be breached and more costly when they are. According to the report, 63% of organizations lack AI governance policies to manage AI or prevent the proliferation of shadow AI.
From a CISO’s standpoint, three emerging liability gaps matter most:
- The trust and control gap: Weak oversight that allows autonomous damage.
- The audit trail gap: Inability to explain or reconstruct AI decisions.
- The third-party gap: Vendor interactions that create unclear fault lines.
Reframing accountability across the AI life cycle
CIOs and CISOs can’t treat accountability as a single point of failure. It must form a chain of ownership that follows the AI through its ModelOps life cycle.
Data owner (input stage)
Responsible for data integrity and bias in training datasets. Poor data lineage creates foreseeable harm. Each AI should have an AI factsheet documenting its data sources, bias testing, and governance approvals. This is a best practice reinforced in the NIST AI Risk Management Framework.
Model owner (business stage)
The line-of-business leader using the AI must own the business outcome—and any resulting harm. Before deployment, the model must undergo adversarial testing to validate safety guardrails.
According to a recent survey, 82% of organizations say they’re using AI across functions, yet only 25% report having a fully implemented AI governance program.
Control owner (oversight stage)
This role is accountable for ongoing monitoring, drift detection, and escalation. Leading enterprises are formalizing a cross-functional AI governance committee (AIGC), jointly led by CIO, CISO, and legal teams, to ratify high-risk use cases and assign oversight responsibility.
How do you operationalize trust and control? Reframing accountability also means translating governance into enforceable technical controls:
- Least privilege for AI: Just as humans don’t get admin rights by default, agentic systems must operate with minimum necessary access. If a customer-service bot can alter financial records, that’s not an AI failure—it’s a security policy failure.
- Explainability as a legal control: For high-impact use cases (hiring, lending, healthcare), explainability isn’t optional; it’s evidence. Audit trails and decision logs are now integral components of compliance.
Proving due diligence when AI causes harm
When an autonomous system acts independently, “we had a policy” won’t satisfy regulators or courts. Due diligence now requires documented proof that governance was operationalized before the harm occurred and that controls were functioning during it.
Proof 1: Pre-condition governance
Show that the AI was classified by risk and autonomy level, approved by the AIGC, and red-teamed for vulnerabilities. High-risk systems (such as financial, medical, and legal) require continuous monitoring and clear human accountability before deployment.
Proof 2: Control effectiveness
Demonstrate that safety constraints were technically enforced, such as logs showing least-privilege restrictions, drift detection, and human override mechanisms (for example, a kill switch) working as intended.
Proof 3: Post-action audibility
Maintain explainable logs that reconstruct the AI’s reasoning chain. Regulators in both the U.S. and EU increasingly expect documentation proving reasonable organizational behavior, and insurers are beginning to require forensic justification before covering AI-related losses.
Balancing innovation with liability: Sandboxes and kill switches
Despite liability concerns, enterprises are not halting innovation—they’re reframing it.
Most organizations are testing agentic AI in low-risk, high-value domains such as customer experience, knowledge summarization, and internal automation.
A recent survey found that 44% of organizations plan to implement agentic AI within the next year to save money, improve customer service, and reduce the need for human intervention.
To manage exposure, many are adopting the constrained autonomy model:
- Sandbox first: Agentic AI runs in a closed environment with no production-write access until validated.
- Role-based access control (RBAC): AI is treated like a new employee with limited scope and supervised duties.
- Kill switches: Mandatory human-triggered stop mechanisms that work even if the AI’s own systems fail.
- Tiered autonomy: Agents may process refunds up to $500 autonomously, while higher amounts require human review.
The goal is to earn the right to innovate safely, demonstrating quick ROI while building governance capabilities for higher-risk deployments.
Consumer AI: The liability squeeze
In consumer-facing applications, the brand (the deployer) bears immediate accountability. The vendor may be legally liable for core defects, but the brand owns the customer relationship and the headlines.
Vendors face growing pressure under evolving frameworks such as the EU’s proposed AI Liability Directive, which expands the definition of “product” to include software.
Courts are increasingly splitting fault:
- Model-level defects: Vendor responsibility
- Deployment-level mismanagement: Enterprise responsibility
CIOs and CISOs must plan for both by enforcing AI responsibility clauses and audit rights in vendor contracts. Liability caps should scale with risk rather than being tied to subscription fees.
Contracts and SLAs: The new risk-allocation toolkit
AI liability is now as much a contract problem as a technical one. Service-level agreements must evolve beyond uptime and performance guarantees to measure trust, safety, and drift.
Key contractual elements include:
- Bias and data warranties: Vendors certify the integrity and fairness of their training data.
- Audit and transparency rights: Access to model documentation and decision logs upon failure.
- Incident response SLAs: Vendor obligations and response timelines for AI-specific breaches or autonomous misbehavior.
Legal experts are calling these AI Responsibility Clauses—contractual language ensuring accountability from pre-deployment through post-incident investigation.
Over the next two years, AI liability norms will enter an enforcement era marked by four major shifts:
- Model vs. deployment fault: Courts will split liability between vendor defects and enterprise misuse.
- Regulatory fragmentation: The EU AI Act sets the global compliance floor while U.S. states adopt sector-specific laws.
- Financialization of AI risk: Insurers will price policies based on governance maturity rather than revenue size.
- Mandatory explainability: “Black box” defenses will collapse, making audit logs and decision transparency essential.
Global regulatory initiatives such as the FTC’s Operation AI Comply signal a clear direction: AI risk management is no longer optional. It is becoming an enterprise control discipline.
CIOs and CISOs must embed governance not as a compliance overlay, but as an engineering function spanning data, model, and control layers.
By Anthony Diaz
Anthony Diaz has a two-decade career as a security and data risk expert working with Fortune 500 organizations. He currently serves as CISO at Exterro and was previously a Managing Director for US Cyber & Strategic Risk at Deloitte.
He has also held senior security leadership roles at Merrill Lynch, Johnson & Johnson, IBM, Optiv Security, Ernst & Young, and HSBC.
