AI Agent Audit Trail Playbook

By "exact prompt": system prompt, user instruction, retrieved context included in prompt, tool call specifications, any dynamic content inserted at runtime, and the complete prompt string as sent to the model API. The common gap: logging the template but not the filled-in prompt. Dynamically generated sections vary per decision — the template documents intent, the actual prompt proves what the agent was instructed to do for that specific case. Without the exact prompt, you cannot prove what instructions the agent was operating under or verify the prompt didn't contain errors or unauthorized modifications.

Access logs show the agent queried Credit Bureau API at 3:42 PM and accessed financial statement documents 9284-FS-2023 and 9284-FS-2024. That proves access. What you need for auditability: the actual credit score returned, the specific paragraphs from the financial statements the agent analyzed, the API response content it reasoned over. Re-querying those systems now returns current data, not what the agent saw at decision time. Credit reports change. Financial statements are updated. The data the agent analyzed at 3:42 PM is not necessarily the data available today.

Version stamping covers: exact model name and version, model deployment timestamp, policy document version and effective date, prompt template version, and configuration state at decision time. The failure pattern: approval rate drops from 65% to 45% in March. Regulator asks why. You know rates changed but cannot show which version of the model or policy was running in February versus March, whether the change was a model update, a policy tightening, or a configuration change, or which specific decisions were made under which version. Without version stamps per decision, you cannot reconstruct causality for behavioral changes.

A replay bundle contains: decision ID and timestamp, complete prompt, retrieved context, model and policy versions, tool calls and responses, intermediate reasoning steps, and final decision with rationale. The test: a regulator requests decision packets for 20 sampled applications. Without replay capability, each one requires 5–8 hours to reconstruct — pulling logs from multiple systems, inferring reasoning from available data, writing narrative explanations. The result is inference presented as evidence. With replay capability: query the audit system with decision IDs, export bundles, 15 minutes total. The difference is not just efficiency — it is the difference between "this is what happened" and "this is what we think happened."

A correlation ID is a unique identifier per decision, present in all relevant log entries, linking prompt logs, context retrieval logs, tool call logs, model invocation logs, and final output. Without it: reconstructing a single decision requires finding the timestamp, searching each system for entries within a narrow time window, manually verifying the entries relate to the same decision, and hoping no other decisions occurred simultaneously. With it: one query returns every log entry for that decision across all systems. The difference is 30–60 minutes of manual detective work versus 30 seconds of a database query.

Tamper-evident logging includes: cryptographic hashing of log entries, immutable append-only storage, digital signatures, and independent timestamping. The litigation scenario: a denied applicant claims discrimination. You produce logs showing the decision. Their legal team asks how they can verify the logs weren't modified after the suit was filed — your IT team has database access. "We have policies against modifying logs" is not an answer to that question. Cryptographic integrity is. Each entry hashed and chained means any modification breaks the chain, which can be verified independently without trusting your assurances.

Decision rationale includes: key factors considered, how factors were weighted, why this decision versus alternatives, confidence level, assumptions made, and risk considerations. The regulatory gap: a regulator asks why your agent denied an application and you can show "Decision: DENY, Timestamp: 15:42:35" — but not the factors that led there. Without documented rationale, the regulator cannot verify the decision was appropriate, check whether protected characteristics influenced it, or determine if policy was applied correctly. "We'd have to infer from the application data" is the answer that triggers deeper investigation.

Intermediate steps include: data validation, initial screening results, factor scoring, risk classification, policy application logic, confidence assessment, and final synthesis. The difference between input-to-output logging and step capture is the difference between "it approved" and "it validated data, screened against minimum criteria, assessed financial strength, scored risk, applied policy limits, and assessed confidence before concluding approval." The regulator's concern with input-to-output only: "I can see it approved. I can't see the reasoning process. Did it consider all relevant factors? Did it weight them appropriately?" You cannot verify soundness of process from conclusion alone.

Reviewable means: plain language explanations, structured scannable format, includes context not just data dumps, and can be understood by domain experts who are not engineers. The test: hand your audit evidence to a loan officer, a compliance auditor, and a legal team member. Can they understand what happened, verify the decision was appropriate, and make an independent determination without an engineer in the room? If the answer is "they'd need engineering to interpret the logs" or "we'd create a separate summary for regulators," your audit trail fails at its primary use case — being reviewed by the people who need to review it.

Behavioral baseline includes: expected approval or denial rates, typical confidence distributions, normal reasoning patterns, expected factor weightings, and standard processing time. The failure mode: model updated in March, approval rate dropped from 65% to 45%, nobody noticed for six weeks. Discovered when the regulator asked why rates changed. By then, 2,000+ decisions had been made under the changed behavior. With baseline monitoring: day one alert — "approval rate dropped to 48%, expected 65% plus or minus 5%." Investigation within hours. Policy interpretation issue identified in new model version. Fifty decisions under anomalous behavior, reviewed immediately. The difference is scale of exposure.

Deviation flagging covers: low confidence decisions below threshold, decisions near policy limits, unusual factor combinations, reasoning inconsistent with similar past decisions, and atypical patterns. The production failure: agent approved a $500K line — the largest ever granted. Standard log entry, no flag. Reviewed: never, treated as routine. Discovered eight months later during portfolio review when the borrower defaulted. The decision was at the extreme edge of policy and should have been escalated to a senior underwriter. No flagging mechanism existed for decisions at policy limits. With flagging: automatic trigger, senior review same day, approval confirmed with proper documentation, defensible decision on record.

Freeze mechanisms include: automatic pause on major behavioral shifts, kill switch for detected errors, human approval required for flagged decisions, and batch review queue before execution. The failure pattern without freeze: anomaly detected at 95% approval rate overnight, alert sent to engineering team, agent approved 400 applications before freeze implemented manually, prompt template error removed credit score check, 400 potentially inappropriate approvals requiring retroactive review, some loans already funded. With freeze: anomaly detected on first 10 decisions, automatic move to review queue, zero inappropriate approvals executed, error identified and fixed within two hours. The mechanism is not monitoring — it is the gate that monitoring triggers.

A review workflow includes: regular sampling (for example, 2% of decisions monthly), risk-based sampling weighted toward high-value decisions, edge cases, and low confidence outputs, documented review criteria and checklist, sign-off process, issue escalation, and corrective action tracking. The difference: regulator samples 50 decisions and finds 8 with questionable rationale. If your answer is "we weren't aware of these issues, we don't have a regular review process," that triggers a deeper examination of your governance model. If your answer is "here are our monthly quality review reports showing what we sample, what we found, and what corrective actions we took," that closes the inquiry. Systematic review is the difference between discovering problems and having them discovered.

Regulators have response deadlines, often 30 days. Sampling requests typically include 20–50 decisions, but a full audit may require hundreds. Manual reconstruction at 2–4 hours per decision means 50–100 hours of effort and 2–3 weeks of calendar time for 25 decisions — a response that nearly misses the deadline and arrives as inference rather than evidence. Automated export means querying the audit system with decision IDs, exporting replay bundles in 15 minutes, QA review in two hours, responding the same day. The regulator's concern with slow reconstruction isn't just operational — it signals that the evidence doesn't actually exist in a form that proves what happened.

Regulatory readiness validation includes: compliance team review of the audit approach, legal review for regulatory adequacy, external auditor testing of evidence, mock regulatory inquiry, and gap identification with remediation before the real examination. The failure pattern: first test is an actual regulatory exam. Discovery: cannot produce decision rationale, logs don't capture reasoning. Result: regulatory finding, remediation required under pressure, months of work. The alternative: mock inquiry six months before the expected audit, compliance team requests evidence for ten sample decisions, reconstruction reveals gaps in six of ten, remediation implemented, second test passes cleanly, actual audit finds no audit trail deficiencies. The gap between "we assume our logs are sufficient" and "we have tested our logs are sufficient" is where most regulatory findings originate.