AI Auditing Tools & Techniques 21% of exam

Scoping defines the boundary of the engagement: which AI system(s), which environments (training, staging, production), which time period, and which auditable units are in and out. For AI this is harder than a classic app because the "system" sprawls across data pipelines, a model, MLOps infrastructure, governance processes, and downstream decisions — and a third-party foundation model may sit at its core.

Auditor's angle: a vague scope is the root of most failed AI audits — you either boil the ocean or miss the unit that mattered. The control to look for is a documented, risk-aligned scope agreed in the engagement letter; the evidence is the model inventory entry, architecture diagram, and data-flow map you use to draw the boundary.

• Fix the system identity precisely: name, version family, owner, and the decision it drives.
• State exclusions explicitly (e.g., "the third-party LLM's internal weights are out of scope; its integration and outputs are in scope").

Before any control is touched, understand what the AI actually does and what it decides: the population affected, the consequence of a wrong output, and the degree of human oversight (human-in-the-loop, on-the-loop, or fully automated). This maps to the risk-tier language from Domain 1 (e.g., EU AI Act prohibited / high-risk / limited / minimal), and the tier should drive audit depth.

Auditor's angle: matching audit effort to risk tier is risk-based planning. The risk is over-auditing a marketing-copy generator while under-auditing a credit-decisioning model. Look for an approved risk classification in the model inventory; if none exists, that absence is itself a governance finding.

• A model that recommends to a human is lower risk than one that decides autonomously.
• Consequence severity (credit, health, employment, safety) outranks transaction volume when tiering.

Every engagement needs objectives (what the audit will conclude on) and criteria (the measurable benchmark). For AI, criteria come from internal AI policy and model-risk standards, regulation (EU AI Act, sector rules), recognized frameworks (NIST AI RMF, ISO/IEC 42001), and contractual/SLA commitments. Without explicit, agreed criteria there is no objective basis for a finding.

Auditor's angle: "the model feels biased" is an opinion; "the selection rate for the protected group was 0.62, below the org's own 0.80 four-fifths threshold" is a finding. The independence trap: the auditor agrees and applies criteria but must not invent the control standard management should have set — doing so means auditing your own benchmark next year.

• Criteria must be defined before testing, in writing, and agreed with management.
• If no criterion exists, the missing criterion is a reportable governance gap, not something you fill in.

A defining Domain 3 skill is breaking an AI system into its auditable units so nothing material is missed: data, model, infrastructure/MLOps, governance, outputs/use, and monitoring. Each unit carries its own risks, controls, and evidence.

Auditor's angle: the risk is a "complete-looking" audit that silently omitted a unit (commonly monitoring or third-party models). The control is a unit-by-unit risk-and-control matrix; the evidence is that each in-scope unit has at least one designed procedure.

Materiality for AI is rarely a dollar figure; it is the threshold at which an issue becomes significant enough to report — expressed as number of affected individuals, severity of a wrong decision, regulatory exposure, or a tolerated error/bias rate. The audit program then translates objectives and risks into specific procedures, evidence to obtain, sample sizes, and timing. The engagement letter also fixes the engagement type (assurance vs advisory).

Auditor's angle: without a materiality definition you cannot rate findings or decide what to report; the control to look for is a documented materiality basis tied to harm and regulation, and an audit program that traces every procedure back to a risk and criterion.

• For AI, "qualitative materiality" (harm to a protected group, safety) often dominates quantitative thresholds.
• The program should state, for each procedure, the assertion tested and the evidence sought.

You do not need to be a data scientist, but you must understand the system well enough to audit it. Where the team cannot competently evaluate a fairness metric, a robustness test, or a model's statistics, the auditor should engage a specialist (internal data-science assurance or an external expert) and remain responsible for scoping their work and integrating their conclusions.

Auditor's angle: pretending to assess something you cannot understand is an audit-quality failure; so is letting a specialist's opinion pass unchallenged. The control is a documented competency assessment during planning; the evidence is the specialist's terms of reference and the auditor's evaluation of their work.

• The auditor remains accountable for the conclusion even when a specialist did the technical test.
• A specialist informs the audit; they do not relieve the auditor of judgment or independence.

The bedrock distinction the exam tests constantly: a test of design (TOD) asks "is the control capable of working if it operates as intended?"; a test of operating effectiveness (TOE) asks "did the control actually work, consistently, over the whole period?" Both apply to AI controls.

Auditor's angle: the classic trap is concluding from design alone. "There is a drift dashboard" proves design; it does not prove anyone read it or acted on an alert. Always push to operating effectiveness with period-spanning evidence.

Statistical sampling uses random selection and lets you project results to the whole population with measurable confidence — use it when you must quantify an error or bias rate defensibly. Judgmental (non-statistical) sampling targets high-risk items (edge cases, the protected subgroup, post-retrain decisions); efficient but not statistically projectable.

Auditor's angle: the method must match the conclusion. If you intend to state "the error rate is X% with 95% confidence," you need a statistical sample; if you only want to probe whether a specific risk exists, judgmental selection is fine — but you must not over-claim projectability from a judgmental sample.

• Statistical → defensible quantified rate; needs proper sample-size calculation.
• Judgmental → fast risk probing; cannot be extrapolated to the population.

The heart of AI sampling is representativeness. A sample of model outputs that omits minority subgroups can hide exactly the bias you are looking for. Stratify the population by subgroup, time period, and decision type so the sample mirrors the risks, not just the volume.

Auditor's angle: a simple random sample drawn from a skewed population reproduces the skew — if the protected group is 3% of records, 60 random items may contain almost none, and "no errors found" proves nothing about fairness. The control is stratification aligned to the audit objective; the evidence is the documented sampling plan showing each stratum's coverage.

• Stratify by the dimensions that carry the risk (protected attribute, retrain epoch, decision outcome).
• Ensure each high-risk stratum has enough items to support a conclusion about it.

Reperformance — independently re-running the model (or a control) and comparing results — is the most powerful AI test: re-score a sample of inputs and check the outputs match what production produced and what validation reported. It is also where reproducibility bites — if you cannot reproduce a result because the model version, data, or random seed was not pinned, that inability is a finding.

Auditor's angle: reperformance is auditor-generated evidence, so it sits at the top of the reliability hierarchy. The risk is non-reproducibility masking poor control; the evidence to demand is a pinned model version, a data snapshot, and a runnable environment.

• Match auditor-reproduced outputs against both production logs and the validation report.
• Treat "we can't reproduce it" as a control finding, not a footnote.

Testing governance gates means confirming approvals genuinely preceded deployment and were given by an independent, authorized role. Testing monitoring controls means confirming alerts are configured, fire when thresholds breach, and are acted on — not merely that a dashboard exists.

Auditor's angle: both are prime "design vs operating effectiveness" territory. The risk is a beautiful dashboard nobody watches and an approval workflow that is bypassed for "urgent" releases. The evidence is alert-fire records tied to investigation tickets, and approval timestamps preceding go-live.

• For monitoring, trace a real breach all the way to its investigation and resolution.
• For gates, test the exceptions (emergency changes) where controls usually fail.

Because models drift and are retrained, point-in-time testing ages quickly. The exam expects you to recognize that AI often calls for more frequent or continuous testing, and that timing matters: test the model version actually in production now, and align sample periods with retraining events.

Auditor's angle: the risk is concluding on a version that has since been replaced, or sampling a period that predates a retrain that changed behavior. The control is testing cadence tied to the retrain schedule; the evidence is alignment between your sample window and the deployment/retrain log.

• Always confirm which version is live today before concluding.
• Anchor sample periods to retrain boundaries so before/after behavior is comparable.

The standard is unchanged: evidence must be sufficient (enough of it) and appropriate (relevant + reliable). A conclusion is only as good as the evidence behind it, and one weak source rarely supports a strong opinion.

Auditor's angle: the risk for AI is concluding on thin or self-reported evidence (a single interview, a self-authored model card). The control mindset is to combine sources so that relevance and reliability are both met, and to corroborate anything self-reported.

• Sufficiency = quantity and coverage; appropriateness = relevance + reliability.
• More high-quality evidence is needed where risk and materiality are higher.

The four classic techniques all apply, and the exam wants you to rank their reliability and pick the right one for the assertion: reperformance (strongest — auditor generates it) > inspection (documents/records) > observation (only proves it happened while you watched) > inquiry (weakest — must be corroborated).

Auditor's angle: when two pieces of evidence conflict, prefer the more reliable source. The trap is resting a conclusion on inquiry alone.

AI adds rich new evidence sources the auditor must know how to use and verify: model cards, datasheets for datasets, training logs, validation reports, monitoring dashboards, version-control history, approval records, data lineage, and decision logs.

Auditor's angle: each source proves something different and has a different reliability — a model card states intended use (self-reported), while git history and decision logs are independent system records. The risk is treating a self-authored artifact as proof of reality; verify the card against logs and reperformance.

The AI-specific habit that separates a credible audit from a weak one: record exactly which model version, data snapshot, and configuration you tested, with hashes or version IDs. Because models are retrained and inputs shift, evidence is meaningless without pinning the version.

Auditor's angle: the risk is "evidence drift" — your finding refers to a model that no longer exists, so it cannot be defended or remediated. The control is version pinning; the evidence is a documented version ID and data-snapshot hash attached to every test result.

• Capture version ID, data-snapshot hash, config, and (where relevant) the random seed.
• Tie every workpaper result to the exact artifacts it was produced from.

Reproducibility itself is evidence. If the team can hand you a pinned environment and you reproduce their validation results, that strongly corroborates their governance. If results cannot be reproduced, the inability is a finding about controls — not a footnote.

Auditor's angle: reproducibility tests the maturity of the MLOps and governance controls behind the model, not just the number. The risk is a team that cannot recreate its own results, signaling weak versioning and change control. The evidence is a successful (or failed) independent reproduction.

• Successful reproduction corroborates governance; failed reproduction is a control finding.
• Ask the team to reproduce their own result first — inability is itself revealing.

Garbage in, garbage out is a control objective here. Auditors assess the data the model trains and runs on against the classic dimensions: completeness (missing records, fields, periods — missing data on a subgroup bakes in bias), accuracy (do values and labels reflect reality — mislabeled training data corrupts the model silently), validity (formats, ranges, business rules), and timeliness (stale data + a shifting world = drift).

Auditor's angle: data quality is upstream of every model risk, so a weak dataset undermines even a well-governed model. The control is documented data-quality checks in the pipeline; the evidence is reconciliation, profiling, and label-audit results — which the auditor can reperform.

• Subgroup completeness is a fairness issue, not just a data-hygiene issue.
• Label accuracy is the silent killer — sample and re-check labels against ground truth.

Computer-assisted audit techniques (CAATs) and analytics let the auditor move beyond samples. Because AI data and decision logs are already digital and large, the auditor can often test the full population — every decision, every record — which gives far stronger coverage of bias and exceptions than a sample. Anomaly analytics (including Benford-style tests) can surface manipulated, fabricated, or out-of-pattern values.

Auditor's angle: sampling is a response to populations you cannot economically test in full; when full-population analysis is feasible, it is more defensible — especially for rare-event bias that a sample would miss. The control/evidence is a documented, repeatable CAAT script run over the complete log.

• Prefer full-population testing when data is digital and complete.
• Full-population analysis catches rare disparities a sample of 60 never would.

An auditor may use AI/ML tools to triage evidence, summarize documents, or detect anomalies — but doing so introduces its own risks the auditor must manage: over-reliance (treating output as conclusion rather than a lead to verify), explainability of the audit's own tools (if you can't explain why it flagged something, you can't defend the finding), confidentiality (feeding client data into an external model), and bias/error in the tool itself.

Auditor's angle: the trap the exam plants is reporting an AI tool's output as a finding without independent verification. Always verify a sample (or all) of the flags and be able to explain how the tool reached them.

Analytics enable continuous auditing/monitoring: automated checks that flag drift, fairness breaches, or anomalous outputs on an ongoing basis rather than once a year. For systems that retrain and drift, point-in-time assurance ages too fast to be enough alone.

Auditor's angle: the risk is annual-only assurance on a model whose behavior changes monthly. The control is auditor-run (or auditor-relied-upon) continuous tests; the evidence is the ongoing exception log and the response to each flag. Independence note: if the auditor relies on management's monitoring, that is testing a control, not the auditor's own continuous audit.

• Continuous auditing suits AI because drift makes one-off testing stale quickly.
• Keep the auditor's continuous tests independent of the controls they assess.

Every well-formed finding has the same anatomy: Condition (what is), Criteria (what should be), Cause (why the gap exists), Effect (so what — the risk/impact), plus a Recommendation owned by management.

Auditor's angle: weak findings stop at condition vs criteria; strong ones nail the root cause (so the fix is real) and the effect (so management cares). The boundary: the auditor recommends; management decides and owns the action.

Findings are rated by severity (high/medium/low or critical/significant/minor) from likelihood and impact against materiality. AI introduces issue types you must name and explain: bias/unfairness, drift, lack of explainability, weak or absent governance, data-quality defects, and third-party/model-supply-chain risk.

Auditor's angle: severity must be consistent and tied to the materiality basis set in planning, not to how alarming the issue feels. The risk is rating by gut; the evidence is a rating rubric mapping likelihood × impact to a defined band.

• Tie each rating back to the planning-phase materiality definition.
• Name the AI issue type precisely — "bias" vs "drift" vs "explainability" drive different fixes.

The skill the exam tests is translating AI findings for two audiences: a technical team needs the metric, the model version, and the test detail; the board needs the business risk in plain language ("the model may be systematically declining a protected group, exposing us to regulatory penalty and reputational harm").

Auditor's angle: the risk is a report that is unreadable to the people who must act — technical jargon for the board, or hand-waving for the engineers. The control is a layered report (executive summary + technical detail); the evidence is that both audiences can act on it.

• Same finding, two registers: business impact up, technical specifics down.
• Never bury a critical bias finding in statistics the board cannot parse.

An honest AI report states its scope limitations (e.g., "we could not reproduce the May validation results; vendor model internals were not accessible") and the residual risk that remains after recommended actions. Reporting residual risk respects that management — not the auditor — owns the decision to accept it.

Auditor's angle: the risk is an over-stated conclusion that hides what you could not test, exposing the auditor and misleading management. The control is explicit limitation and residual-risk statements; the evidence is their presence in the report.

• Disclose what you could not access or reproduce — never imply coverage you didn't have.
• State residual risk; let management formally accept (or reject) it.

Reporting is not the end: the auditor tracks remediation to closure and re-tests that fixes work. A finding is only resolved when evidence shows the control now operates effectively — not when management says it's done.

Auditor's angle: the risk is "closed-on-promise" — items marked resolved on management's word without re-testing. The control is a tracked remediation log with auditor re-verification; the evidence is the re-test result confirming operating effectiveness.

• Re-test the fix; don't close on management's assertion alone.
• Track each finding with owner, due date, and verification status.

Two heavily tested distinctions close the domain. Assurance gives an independent opinion against criteria; advisory helps management improve. Advisory can erode independence if the auditor later has to audit what they helped design, so the engagement type must be set up front and disclosed. And the cardinal rule: the auditor never designs, owns, operates, or signs off the controls under audit — they recommend; management decides, owns, and accepts residual risk.

Auditor's angle: objectivity is impaired if the audit function configured the very controls under review, or used an AI tool built/tuned by the team it now audits — disclose and manage it, and use a different auditor for the assurance.