Why existing AI governance frameworks are insufficient for agentic systems — and the preliminary documentation architecture required to address the gap.
The governance frameworks that currently structure enterprise AI risk management — NIST AI RMF 1.0, EU AI Act, ISO/IEC 42001 — were designed for AI systems that produce outputs for human review. A loan underwriting model produces a credit decision; a human reviewer makes the credit approval. A diagnostic imaging model produces a finding; a physician makes the diagnostic conclusion. Human oversight is assumed to be structurally available at the point of consequential action.
Agentic AI systems — AI that takes sequences of autonomous actions in complex environments, plans across multi-step tasks, uses tools and external systems, and produces consequences that may not be easily reversible — challenge this assumption structurally. The human oversight assumed by current governance frameworks is not available at the point of consequential action in an agentic system, because the consequential action is not a single classifiable output: it is an action in an environment, executed by an agent that has been given a goal and the autonomy to pursue it.
This working paper argues that agentic AI requires a distinct documentation architecture — one that addresses the documentation obligations created by agent autonomy, multi-step action sequences, tool use, environmental interaction, and the compounding risk of error propagation across agent reasoning chains. It presents a preliminary framework for agentic AI documentation and invites practitioner and academic response.
The term "agentic AI" encompasses a range of systems that share a common structural characteristic: they take actions autonomously in pursuit of a goal, using tools and interacting with external environments in ways that produce real-world consequences. Retrieval-augmented generation (RAG) systems that search, retrieve, and synthesize information autonomously; coding agents that write, test, and deploy code; customer service agents that access backend systems to resolve issues; research agents that conduct multi-step information gathering — all exhibit the agentic characteristic of autonomous multi-step action with real-world consequence.
The governance documentation challenge specific to agentic AI has three dimensions that are structurally different from the documentation challenges of non-agentic AI systems.
First, the action sequence problem. Current AI governance documentation frameworks require documentation of model inputs, outputs, and the decision logic connecting them. For agentic systems, the "input" is a goal specification; the "output" is a sequence of actions that may span dozens of steps, involve multiple tool calls, and produce environmental changes that are themselves inputs to subsequent steps. Documenting an agentic AI system's behavior requires documenting the action sequence, the decision logic at each step, the tools available and how their use is governed, and the conditions under which the agent escalates to human oversight.
Second, the reversibility problem. Non-agentic AI outputs are typically reversible: a credit denial can be appealed, a diagnostic finding can be reviewed by a second physician, a content recommendation can be ignored. Agentic AI actions may not be reversible: a deployed code change, a sent communication, a financial transaction, a deleted file. Governance documentation must address how irreversibility risk is managed — which action types require human approval before execution, which action types can be executed autonomously and reversed if incorrect, and which action types cannot be reversed and therefore require pre-execution validation.
Third, the error propagation problem. In a multi-step agentic reasoning chain, errors compound. A misclassification at step 3 of a 20-step agent task may not be detectable until step 18, at which point it has propagated through 15 subsequent decisions. Governance documentation must address how error propagation risk is managed, how the agent's reasoning chain is logged in a form that makes error propagation traceable, and how the agent is designed to fail safely when errors are detected.
The NIST AI RMF's four functions — GOVERN, MAP, MEASURE, MANAGE — provide the correct governance vocabulary for agentic AI but insufficient documentation specificity for the agentic context. GOVERN function requirements for organizational accountability and governance structure apply to agentic AI systems without modification. MAP function requirements for categorizing AI systems by risk are applicable to agentic AI but require extension: the MAP function must classify not just the agentic system's purpose and context, but the range of actions the system can take and the reversibility profile of each action class. MEASURE function requirements for performance monitoring must be extended to measure agentic system reasoning chain quality, not just input-output accuracy. MANAGE function requirements for risk treatment must address the specific risk patterns of agentic systems: error propagation, action reversibility, and autonomous decision-making scope.
The EU AI Act's technical documentation requirements (Article 11, Annex IV) apply to high-risk AI systems — and many agentic AI deployments will qualify as high-risk under Annex III categories. But Annex IV was written for systems with defined input-output behavior, not for systems whose behavior emerges from goal-directed multi-step reasoning. The technical documentation requirement for "description of the system's performance" is clear for a binary classifier; it is underspecified for an agent whose performance is measured across a sequence of actions in a dynamic environment.
The ELDR Institute's preliminary position is that existing frameworks provide the governance skeleton for agentic AI documentation but require three specific extensions: an action taxonomy documenting what action classes the agent can execute and their reversibility profiles; a reasoning chain logging specification documenting how the agent's decision process is recorded in auditable form; and an oversight threshold specification documenting the conditions under which autonomous action is suspended and human review is required.
"Governing agentic AI requires documenting not what the system produces, but what it does — across sequences of actions whose consequences compound and may not be reversible."
The following preliminary documentation framework addresses the three agentic AI governance gaps identified above. It is presented as a working hypothesis for practitioner comment, not as a final governance methodology.
Component 1: Agent Specification Document. Analogous to EU AI Act Annex IV technical documentation, the Agent Specification Document describes the agent's designed purpose, goal specification format, available tool set with each tool's action capabilities and side effects, environmental interaction scope (which external systems the agent can read from and write to), action authorization model (which action classes require human approval, which can be executed autonomously, and which are prohibited), and failure behavior specification (what the agent does when it encounters an error, an ambiguous goal, or an action that would violate defined constraints).
Component 2: Reasoning Chain Logging Specification. Agentic AI systems generate reasoning traces — logs of the agent's decision process at each step of a task. The Reasoning Chain Logging Specification defines what information is captured at each reasoning step, how logs are stored and retained, who has access to reasoning chain logs, and how logs are used in governance review, incident investigation, and performance assessment. Without a Reasoning Chain Logging Specification, agentic AI incidents cannot be reconstructed and audited — which means governance review cannot demonstrate that the agent operated within its defined constraints.
Component 3: Oversight Threshold Policy. The Oversight Threshold Policy specifies the conditions under which autonomous action is suspended and human review is required. Thresholds may be defined by action irreversibility (any action that cannot be reversed within 24 hours requires human approval), by action consequence magnitude (any action affecting more than N records or N dollars requires human approval), by reasoning chain uncertainty (any reasoning step where the agent's confidence falls below a defined threshold requires human escalation), or by action category (any action in a defined category of sensitive actions requires human approval regardless of other factors).
Component 4: Error Propagation Risk Assessment. For each agent deployment, a structured assessment of error propagation risk: the longest reasoning chain the agent is expected to execute, the point in the chain at which errors are most likely to compound, the mechanisms for detecting error propagation before it reaches irreversible consequence, and the agent's designed response to detected error propagation.
This working paper presents preliminary analysis and invites practitioner and academic response to the following research questions, which will inform subsequent ELDR Institute research on agentic AI governance.
While this working paper presents early-stage research, several preliminary practitioner observations warrant immediate attention for organizations deploying agentic AI systems in regulated environments.
First, deploy no agentic AI system in a regulated environment without a documented action taxonomy. The minimum governance requirement for agentic AI deployment is a documented inventory of the actions the agent can take, the consequences of each action class, and the reversibility profile of each action class. This is achievable in any deployment timeline and is the single most important documentation investment for agentic AI governance.
Second, treat reasoning chain logging as infrastructure, not audit preparation. Organizations that implement reasoning chain logging only when an incident occurs will not have the historical data needed for meaningful incident investigation. Logging should be designed and implemented before deployment, as governance infrastructure, not after an incident as remediation.
Third, design oversight threshold policies before deployment, not in response to incidents. The conditions under which human oversight is required should be defined before the agent is deployed — not discovered empirically through incidents that reveal where the autonomous action boundary should have been drawn.