AI Operations 46% of exam

Supervised-learning data is partitioned into three sets, each with one job. The training set is what the model learns its parameters (weights) from — usually the largest slice. The validation set is used during development to tune hyperparameters and choose between candidate models; the model sees it indirectly because you keep adjusting based on it. The test set is a locked holdout used once, at the very end, to give an honest estimate of real-world performance.

Auditor's angle: If the test set influences any design decision it is contaminated and the reported accuracy is fiction. Confirm the test set was held out, used a single time, and never used for tuning or model selection. The evidence is the documented split methodology and a record of when/how the test set was scored.

• You tune on validation; you report on test.
• Reusing the test set to pick a model is over-fitting to the test set and inflates the numbers.
• Splits should be reproducible (fixed random seed or documented rule).

Production AI data flows through a pipeline: ingestion → cleaning → transformation → feature creation → serving. A feature store is a managed repository of engineered features shared across models and across training and serving. Its main control value is preventing training–serving skew — the situation where a feature is computed one way during training and a different way in live serving, so the model degrades silently.

Auditor's angle: Undocumented or duplicated feature logic is a hidden risk; the same "customer tenure" computed two ways gives two models. Look for a single source of feature definitions, consistency between training and serving paths, and access controls on the pipeline. Evidence: pipeline diagrams, feature-store definitions, and skew-monitoring results.

• Feature store = consistency between training and serving (the same code path computes the feature).
• Pipelines need monitoring, error handling, and access control like any production system.
• Training–serving skew is a top cause of "great in the lab, weak in production."

In supervised learning the labels are the "ground truth" the model imitates. If human labelers were untrained, inconsistent, rushed, or biased, the model faithfully learns the wrong answers. Quality is measured with inter-annotator agreement (do independent labelers agree?), governed by written labeling guidelines, and verified by QC sampling of labels.

Auditor's angle: Poor labels are an invisible defect — the model can be technically sound yet wrong because the truth it learned was wrong. Ask for the labeling guidelines, the inter-annotator agreement metric, the QC sampling rate, and how disagreements were adjudicated. Evidence: annotation guidelines, agreement statistics (e.g. Cohen's/Fleiss' kappa), and a sample of audited labels.

• Low inter-annotator agreement = ambiguous task or untrained labelers.
• Single-labeler datasets have no agreement check at all — a red flag for high-stakes use.
• Label bias becomes model bias.

Data versioning pins exactly which version of which dataset trained which model. Lineage (provenance) traces where data came from, how it was transformed, and where it flowed. Together they answer the question "what did this model learn from?" — essential for reproducibility, incident investigation, and demonstrating lawful, approved data use.

Auditor's angle: Without data versioning and lineage you cannot reproduce a model, investigate an incident, or prove what data was used. Look for a data version control system, a model-to-data mapping in the model registry, and end-to-end lineage records. Absence is itself a finding.

• Reproducibility requires pinning code and data and the model.
• Lineage supports privacy (where did personal data come from?) and incident root cause.
• "We retrain on the latest data" with no snapshot = no reproducibility.

A model can only be as fair and accurate as its data is representative of the population it will serve. If a hiring model is trained mostly on one demographic, it under-performs for everyone else — sampling bias baked into the foundation. Representativeness also covers coverage of conditions: rare but important cases (edge weather, fraud variants) must appear often enough to learn.

Auditor's angle: Unrepresentative data produces a model that is accurate on average but discriminatory or blind in segments. Ask for a representativeness/bias analysis comparing the training population to the deployment population, broken out by relevant subgroups. Evidence: distribution comparisons and subgroup performance figures.

• "Accurate overall" can hide poor performance for under-represented groups.
• Compare training distribution to the real serving population, not to a textbook ideal.
• Rare-but-critical cases need deliberate sampling/over-sampling.

Data leakage is when information that would not be available at prediction time sneaks into training — a feature that is a proxy for the answer, the same customer in both train and test, or a value populated only after the outcome. The model looks brilliant in testing and collapses in production. It is one of the most common, dangerous, and testable AI data faults.

Auditor's angle: The signature is "too good in test, poor in production." The architecture of the experiment, not the model, is broken — more data leaks just as badly. Request the feature list with availability timestamps and confirm the split was by entity (e.g. by customer) where appropriate, not by row.

• Target leakage: a feature encodes the answer (a "collections flag" set only after default).
• Train/test contamination: the same entity appears in both sets.
• Temporal leakage: future information used to predict the past.

Synthetic data is artificially generated records used to fill gaps, balance rare classes, or protect privacy by not using real personal data. The trade-off: it can encode the biases of the generator, fail to capture rare real-world cases, and create a false sense of coverage if it is too similar to the real data it was derived from.

Auditor's angle: Synthetic data needs governance like any other data source — approval, documentation of how it was generated, and validation that it preserves the relationships that matter without leaking real records. Evidence: generation method, fidelity/utility tests, privacy checks, and approval records.

• Synthetic data inherits the generator's biases and blind spots.
• Privacy benefit only holds if it cannot be reverse-engineered to real individuals.
• Validate fidelity (does it match real distributions?) and utility (does a model trained on it work on real data?).

Data poisoning is a deliberate attack: an adversary corrupts training data to degrade the model or implant a hidden backdoor that triggers on a specific input pattern. It is especially dangerous when training data is sourced from the open web, user contributions, or unvetted third parties.

Auditor's angle: The control is data provenance and integrity — trusted sources, validation/anomaly detection on incoming training data, and a model bill of materials for third-party data and pre-trained models. Evidence: source approvals, data validation logs, integrity/signature checks, and anomaly-detection alerts on training inputs.

• Poisoning targets the training stage, unlike adversarial examples which target inference.
• Backdoors are dormant until a trigger appears — hard to find by ordinary testing.
• Web-scraped and user-contributed data are the highest-risk sources.

A useful end-to-end sequence: problem framing (define the business problem, success metrics, and whether AI vs a simple rule is even appropriate) → data collection & preparation → feature engineering → model training → validation & evaluation → deployment → monitoring → retirement/decommission. Each stage has a control that belongs to it.

Auditor's angle: A frequent right answer is "build the control into the lifecycle at the correct stage" rather than "add a manual check later." Verify each stage exists, has an owner, and produces evidence (sign-offs, test results, monitoring dashboards).

• Problem framing decides whether AI is appropriate at all — skipping it is a root cause of many failures.
• Bias testing belongs at validation, not after a complaint.
• Retirement is a real stage: models that no longer fit purpose must be retrained or decommissioned.

MLOps extends DevOps to machine learning: automated, repeatable pipelines for training, testing, deploying, and monitoring models, with version control over code, data, and models. CI/CD for models means automated build → test → deploy pipelines so changes are validated and traceable. MLOps is what makes an AI system auditable; its absence is itself a finding.

Auditor's angle: Manual, ad-hoc model deployment with no automated testing or versioning means no reproducibility, no rollback, and no audit trail. Look for a model registry, automated test gates in the pipeline, and traceability from a production model back to its commit and data version.

• MLOps versions three things: code, data, and the model artifact.
• CI/CD gates automate the "did it pass the tests?" check before promotion.
• No MLOps → no reproducibility → unverifiable assurance.

CRISP-DM (Cross-Industry Standard Process for Data Mining) is the classic six-phase methodology: Business Understanding → Data Understanding → Data Preparation → Modeling → Evaluation → Deployment. Crucially it is iterative, not linear — teams loop back (e.g. from Evaluation to Business Understanding) as they learn.

Auditor's angle: CRISP-DM gives a recognizable structure to check a project against. The most-skipped phase is Business Understanding; a project that jumped straight to modeling often lacks a clear success metric and stakeholder alignment. Evidence: documented phase outputs and the rationale at each gate.

• Six phases, starting and re-anchoring on Business Understanding.
• Iterative — evaluation can send you back to reframe the problem.
• Maps cleanly onto the broader ML lifecycle (2.1).

Model selection balances accuracy against interpretability, cost, latency, and risk — a highly accurate black box may be the wrong choice for a high-stakes, regulated decision where explainability is required. Hyperparameter tuning (learning rate, tree depth, regularization) is performed on the validation set, never the test set.

Auditor's angle: The "best" model is not always the most accurate one; the choice should match the risk and regulatory context. Check that selection criteria were documented and that interpretability requirements for high-stakes decisions were weighed. Evidence: model comparison records, selection rationale, tuning logs on validation data.

• Accuracy is one axis; interpretability, cost, latency, and risk are others.
• Tuning happens on validation; the test set stays locked.
• High-stakes/regulated decisions may justify a slightly less accurate but explainable model.

Reproducibility is the ability to recreate a model exactly from pinned code, pinned data, and fixed random seeds. It is a control objective: if you cannot reproduce a model, you cannot truly assure it, investigate it, or roll it back with confidence.

Auditor's angle: Non-reproducibility is a fundamental MLOps gap that quietly undermines every other control. The test is simple — can the team regenerate the production model and get the same result? Evidence: pinned dependencies, fixed seeds, data version, and a successful reproduction run.

• Reproducibility = code + data + seed/config all pinned.
• It underpins incident investigation and safe rollback.
• "It works on my machine" is the opposite of reproducible.

Two artifacts the exam expects you to recognize. Model cards are a standardized summary of a model — intended use, performance across subgroups, limitations, ethical considerations, and evaluation data. Datasheets for datasets document a dataset's motivation, composition, collection process, and recommended uses. Both are prime audit evidence.

Auditor's angle: These documents tell you whether the team recorded intent, limits, and validation — or shipped a black box. A missing or empty model card on a high-stakes model is a finding. Evidence: the model card and datasheet, checked for completeness (especially subgroup performance and stated limitations).

• Model card = the model's "nutrition label": use, limits, subgroup performance.
• Datasheet = the dataset's provenance and composition record.
• Empty/boilerplate cards are nearly as bad as none.

Model versioning registers every model artifact with a unique version and its metadata (data version, metrics, approval, owner) in a model registry. It is the backbone that makes rollback, audit trails, and incident investigation possible — you cannot restore or compare what you did not version.

Auditor's angle: Without versioning there is no rollback path and no way to know which model made a given decision. Check that every promoted model has a registry entry and that production always points to a specific, known version. Evidence: the model registry and the mapping from live endpoint to model version.

• Each version carries metadata: data version, metrics, approver, date.
• Production should reference an explicit version, never "latest" implicitly.
• Versioning enables rollback and decision attribution.

Models are retrained to fight drift, but when matters. Retraining triggers should be defined and controlled — scheduled (e.g. monthly), drift-based (fired when a drift/KRI threshold is breached), or event-based (after a known data-source change) — rather than ad-hoc "whenever someone feels like it."

Auditor's angle: Ad-hoc retraining is an uncontrolled change; each retrain is a new model needing validation and approval. Confirm triggers are defined, that a retrain still passes the validation gate, and that the retrained model is versioned. Evidence: the retraining policy, trigger logs, and post-retrain validation results.

• Retraining = a change, even with identical code.
• Drift-based triggers tie retraining to monitoring (Subtopic 4).
• Every retrain must re-pass validation before promotion.

Three core change controls. Approval gates require sign-off before a model is promoted. Rollback is the ability to restore a known-good prior model instantly. Segregation of duties (SoD) ensures the person who builds a model is not the one who unilaterally promotes it to production.

Auditor's angle: A builder who can self-approve and self-deploy is a classic SoD failure. Confirm an independent approval step, a tested rollback path, and that builder and approver are different people. Evidence: approval records with approver identity, rollback test results, and access controls enforcing SoD.

• Approval gate before promotion; rollback after, if it goes wrong.
• SoD: build ≠ approve ≠ deploy held by the same person.
• A rollback path that has never been tested is not a control.

Safe deployment patterns reduce the blast radius of a bad model. In shadow deployment the new model runs alongside the old on real traffic but its outputs are not used — only compared (lowest risk). In canary deployment the new model serves a small slice of traffic first; if metrics hold, you ramp up. (Blue/green does a full switch with the old version kept ready for instant rollback.)

Auditor's angle: A direct, full cutover with no staged rollout is high risk. Confirm the rollout strategy matches the stakes, that success criteria for ramping are defined, and that monitoring runs during the rollout. Evidence: deployment strategy docs, canary metrics, and ramp/abort criteria.

• Shadow = compare without using outputs; safest for validation in production.
• Canary = small live slice, then ramp on good metrics.
• Blue/green = instant switch with the old version on standby.

Environment promotion means a model moves through separate dev → test → prod environments, and production is never trained or tuned directly. Each environment has appropriate data and access, and promotion happens only after the prior stage's checks pass.

Auditor's angle: Training or tuning directly in production, or developers having standing write access to prod, are classic findings. Confirm environment separation, that prod is not used for experimentation, and that promotion is gated. Evidence: environment configuration, access controls per environment, and promotion logs.

• Production is for serving, never for experimentation.
• Access narrows as you move toward prod.
• Promotion is gated, not a copy-paste.

For LLM-based systems, the prompt or system message and configuration (temperature, model choice, tool access, retrieval settings) dramatically change output — yet they are easy to edit informally with no record. The exam's point: an uncontrolled prompt edit is a real production change with no rollback.

Auditor's angle: Prompts and configs must be version-controlled, tested, and approved like code. Confirm prompts live in version control, that changes go through change management, and that there is a rollback. Evidence: prompt version history, test results for prompt changes, and approval records.

• A prompt edit is a behavioral change to a production system.
• Prompts and configs need versioning, testing, and rollback.
• Informal "quick prompt fixes" are uncontrolled changes — flag them.

Ongoing monitoring continuously watches a live model's inputs, outputs, performance, and infrastructure against baselines established at deployment. It is the early-warning system that detects drift, degradation, anomalies, and abuse before they cause harm.

Auditor's angle: "Deploy and forget" is a top AI failure mode — models degrade silently. Confirm monitoring exists for inputs and outcomes (not just uptime), with defined baselines and alerting. Evidence: monitoring dashboards, alert configuration, and a log of alerts and responses.

• Monitor inputs, outputs, performance, and infrastructure.
• Baselines come from the deployment-time validation results.
• Monitoring without alerting/ownership is just dashboards no one watches.

Drift is the gradual divergence between the world the model was trained on and the world it now operates in. Data drift (covariate shift) is when the distribution of the inputs changes while the underlying rule stays the same. Concept drift is when the relationship between inputs and the correct output changes — what used to predict the outcome no longer does. Distinguishing them is one of the most heavily tested points in the whole domain.

Auditor's angle: The two need different detection. Data drift is caught by monitoring input distributions vs the training baseline; concept drift is caught only by monitoring live accuracy against actual outcomes. A team that monitors only inputs will miss concept drift entirely. Evidence: both input-distribution monitoring and outcome-based performance monitoring.

Performance degradation is the decline in a model's effectiveness over time — driven by drift, data-quality breaks, pipeline failures, or a bad retrain. It can be gradual (creeping drift) or sudden (a broken upstream feed). Detecting it requires comparing live performance to the deployment baseline against defined thresholds.

Auditor's angle: Degradation often goes unnoticed because there is no ground-truth feedback or no threshold to breach. Confirm there is a way to obtain actual outcomes, a baseline, and a threshold that triggers action. Evidence: performance trend charts, the degradation threshold, and the response when it is hit.

• Gradual degradation = drift; sudden degradation = often a data/pipeline break.
• You need actual outcomes (ground truth) to measure performance, sometimes with a lag.
• A threshold turns "looks worse" into an actionable trigger.

Levels of oversight: human-in-the-loop (a human approves each decision), human-on-the-loop (a human monitors and can intervene), and human-in-command (humans retain ultimate authority). High-stakes decisions (credit, hiring, healthcare, anything affecting rights) demand meaningful human review and a working override mechanism.

Auditor's angle: The danger is oversight that exists in form but not substance — rubber-stamping. Confirm reviewers have time, information (including the model's reasoning), authority, and incentives to actually overturn decisions, and that overrides are logged and monitored. Evidence: override logs, approval times, and override rates as a KRI.

• In-the-loop / on-the-loop / in-command = decreasing per-decision involvement.
• Meaningful review needs time, information, and authority to overturn.
• Track override rate — too low can signal rubber-stamping.

Output validation checks the model's outputs before they are acted on — range/sanity checks, business-rule guardrails, and (for LLMs) groundedness/format checks. Feedback loops are a related risk: when a model's own outputs become its future training data and amplify its biases (e.g. a policing model sending patrols where it already predicted crime, generating more arrests there).

Auditor's angle: Unvalidated outputs can cause harm directly; unmanaged feedback loops cause harm that compounds. Confirm outputs pass validation/guardrails and that the team has assessed whether outputs feed back into training. Evidence: output-validation rules, guardrail logs, and a feedback-loop risk assessment.

• Validate outputs against ranges, business rules, and (for LLMs) groundedness.
• Feedback loops let a model's bias reinforce itself over time.
• Self-fulfilling predictions are a subtle, compounding risk.

Supervision needs metrics and a path to act on them. KPIs track whether the model is delivering value (e.g. uplift, business outcome). KRIs (key risk indicators) — drift scores, error rates by subgroup, override frequency, complaint volumes — give early warning. When a KRI breaches its threshold, a defined escalation path routes it to the right owner and, if needed, to incident response.

Auditor's angle: Metrics with no thresholds or no escalation are decoration. Confirm KRIs are defined with thresholds, owners, and an escalation path that can trigger investigation, retraining, or rollback. Evidence: the KRI register with thresholds, the escalation procedure, and examples of escalations that fired.

• KPIs = value delivered; KRIs = early warning of risk.
• Each KRI needs a threshold, an owner, and an escalation route.
• Escalation connects supervision to change management and incident response.

Every prediction a classifier makes is a true positive (TP), true negative (TN), false positive (FP) or false negative (FN). Arranged in a 2×2 grid this is the confusion matrix, and every headline metric is derived from it. Accuracy = (TP+TN)/all; Precision = TP/(TP+FP) — of those flagged positive, how many really were; Recall (sensitivity) = TP/(TP+FN) — of the real positives, how many were caught.

Auditor's angle: a single accuracy figure on a report is not assurance. The risk is that the team optimised the wrong metric for the business cost. Demand the confusion matrix and the threshold behind the number, and confirm the chosen metric matches which error (FP or FN) actually hurts.

• Recall matters when false negatives are costly (missed fraud, missed disease).
• Precision matters when false positives are costly (blocking good customers, false accusations).
• Always ask which metric was the optimisation target and why.

Most classifiers output a probability; a threshold turns it into a yes/no. Moving the threshold trades precision against recall. The ROC curve plots true-positive rate against false-positive rate across all thresholds, and AUC (area under it) measures how well the model ranks positives above negatives independent of any one threshold — 0.5 is random, 1.0 is perfect.

Auditor's angle: a great AUC does not mean the deployed threshold is right. The risk is a defensible model wrapped around an arbitrary or undocumented cut-off. Ask for the threshold rationale and confirm it was tied to error costs, not left at a library default of 0.5.

• AUC compares models; the threshold sets operating behaviour.
• On imbalanced data, prefer the precision–recall curve over ROC, which can look flatteringly good.
• The chosen threshold is a change-controlled configuration (links to change management).

A model can be accurate overall yet systematically worse for a subgroup. Fairness testing slices performance and outcomes across protected groups using metrics such as demographic parity (equal positive rates), equal opportunity (equal recall) and equalised odds. These definitions conflict — you usually cannot satisfy all at once — so the team must choose and justify one.

Auditor's angle: the risk is discriminatory impact and legal exposure hidden behind a good aggregate score. Look for a documented fairness metric appropriate to the use case, subgroup results (not just overall), and sign-off that the chosen definition was deliberate.

• Disaggregate every key metric by protected attribute.
• Fairness testing belongs at validation, before deployment — not after a complaint.
• "We don't use the protected attribute" is not a defence — proxies (zip code, name) leak it.

Robustness testing asks whether the model holds up under noisy, shifted, malformed or deliberately crafted inputs. Adversarial testing goes further, generating perturbations — often imperceptible to humans — designed to flip the prediction. A model that is accurate on clean data can be brittle and unsafe in the wild.

Auditor's angle: the risk is a fragile model that fails (or is attacked) the moment reality deviates from the test set. Look for a robustness test plan, adversarial test results, and evidence that the team tested degraded inputs — not just the happy path.

• Perturbation, noise injection, and corrupted/edge inputs.
• Adversarial training (feeding crafted examples back in) is a mitigation, not just a test.
• Safety-critical and security-facing models need this most.

Explainability tools show why a model made a prediction. SHAP (Shapley values) fairly attributes a prediction to each feature and works globally and locally; LIME builds a simple local surrogate model around one prediction to approximate its drivers. Testing explanations confirms the model relies on sensible features, not spurious correlations.

Auditor's angle: the risk is a model whose accuracy comes from a leak or a forbidden proxy. Explainability is also a compliance control where individuals have a right to an explanation. Look for explanation artifacts on high-stakes models and confirm top features make business sense.

• SHAP = consistent global + local attribution; LIME = quick local surrogate.
• Use explanations to detect leakage: a dominant suspicious feature is a red flag.
• Explanations support adverse-action notices and regulator queries.

Generative systems need their own evaluation. Red-teaming has adversaries deliberately try to break guardrails (toxicity, jailbreaks, data leakage, dangerous instructions). Hallucination / groundedness testing checks whether outputs stick to provided or verifiable facts rather than confidently inventing them — central to RAG systems where answers must be traceable to retrieved sources.

Auditor's angle: the risk is plausible-sounding falsehoods, harmful content and bypassed safety controls reaching users. Because output is open-ended, look for a combined evaluation: automated scoring plus human review, a documented red-team exercise, and a groundedness metric for any factual use case.

• Groundedness = output supported by cited source context.
• Red-team coverage: prompt injection, jailbreaks, PII leakage, bias, unsafe instructions.
• Track an ungrounded-response rate as a release gate and ongoing KRI.

A/B testing compares two model variants on live traffic with a controlled split and a pre-declared success metric. Benchmarking measures a model against standard datasets or a baseline/incumbent. Edge-case and stress testing probe rare, extreme or out-of-distribution scenarios the training data barely covered, plus load behaviour under volume.

Auditor's angle: the risk is cherry-picked comparisons and untested rare cases that cause real-world harm. Look for a pre-registered A/B metric, statistically valid sample sizes, an apples-to-apples benchmark, and an explicit edge-case catalogue with results.

• A/B success metric and sample size must be set before the test.
• Benchmark against the current model and a relevant standard, not a strawman.
• Edge cases = the rare-but-high-impact tail; document and test it deliberately.

Adversarial examples are inputs perturbed by tiny, often imperceptible amounts engineered to push a model across a decision boundary into a wrong, attacker-chosen output — a few pixels that turn a "stop" sign into "speed limit," or token tweaks that flip a spam filter.

Auditor's angle: the risk is an attacker reliably forcing misclassification in a security- or safety-critical model. Controls: adversarial training, input sanitisation, robustness testing, anomaly monitoring on inputs. Evidence: robustness test results and monitoring alerts for unusual input patterns.

• Imperceptible perturbation, large behavioural effect.
• Highest concern where outputs gate safety or security.
• Mitigated by adversarial training + runtime input monitoring.

Data poisoning corrupts training data to degrade the model or implant a hidden backdoor — a trigger pattern that makes the model behave as the attacker wants while looking normal otherwise. Risk is acute when training data is crowd-sourced, scraped, or supplied by third parties.

Auditor's angle: the risk is a silently compromised model. Controls: data provenance, trusted sources only, integrity checks, anomaly detection on training data, and curation of feedback that becomes training data. Evidence: data lineage records, source approvals, data-validation logs.

• Two modes: availability (degrade) vs targeted backdoor (trigger).
• Continuous-learning systems that ingest user feedback are prime targets.
• Provenance + integrity verification is the core defence.

Model inversion reconstructs sensitive training data from a model's outputs (e.g. recovering a recognisable face). Membership inference determines whether a specific record was in the training set — itself a privacy breach, since membership can reveal that someone was, say, in a medical-condition dataset.

Auditor's angle: the risk is leakage of personal/confidential training data through the model interface, with direct privacy-law exposure. Controls: differential privacy, output/confidence limiting, strict API access control and rate limiting. Evidence: privacy design docs, DP parameters (epsilon), API logging and rate-limit config.

• Overfitted models leak more — they "memorise" individuals.
• Returning raw confidence scores aids the attacker; limit output detail.
• Differential privacy bounds what any single record contributes.

Model extraction queries an exposed model enough to train a near-copy, stealing the IP, the competitive edge, and a sandbox in which to craft further attacks. The attacker never sees the weights — they reconstruct behaviour from input/output pairs.

Auditor's angle: the risk is theft of a costly proprietary asset via the public API. Controls: authentication, rate limiting, query monitoring/abuse detection, output coarsening, and watermarking. Evidence: API access logs, throttling configuration, abuse-detection alerts, and watermark records.

• High-value, publicly queryable models are the targets.
• Detect by volume and systematic input-space coverage per client.
• Watermarking helps prove a stolen clone later.

Prompt injection feeds an LLM crafted input that overrides its instructions. Direct injection is the user typing "ignore your rules"; the deeper danger is indirect injection — malicious instructions hidden in a document, email or web page the LLM later reads. Jailbreaks bypass safety guardrails; training-data exfiltration coaxes the model to reveal memorised secrets.

Auditor's angle: the risk multiplies when an LLM has excessive agency (tools, data access, the ability to act). Controls: separate system instructions from untrusted content, input/output filtering, least-privilege tool access, and human approval for consequential actions. Evidence: guardrail config, red-team reports, agency/permission scoping, output-handling controls.

• Indirect injection turns any ingested content into an attack vector.
• Prompt-level guardrails are bypassable; architectural limits hold.
• "Prompt injection is the new SQL injection" — treat all input as untrusted.

Most teams build on pre-trained models, open-source datasets and third-party components they did not create. Any of these can carry hidden backdoors, poisoned data, malicious code in model files, or licence/IP problems — supply-chain and provenance risk.

Auditor's angle: the risk is inheriting a compromise you cannot see. Controls: vet third-party models/datasets, verify signatures/hashes, scan model files, maintain a model bill of materials (SBOM/MBOM), and pin versions. Evidence: vendor due-diligence records, integrity verification, the model/data SBOM, licence approvals.

• Pickle/serialized model files can execute code on load — scan them.
• "Downloaded from a hub" is not provenance — record source, version, hash, licence.
• Provenance underpins reproducibility and incident root-cause.

The OWASP Top 10 for LLM Applications is the reference list the exam may cite. Recognise the key items: Prompt Injection, Sensitive Information Disclosure, Supply Chain, Data and Model Poisoning, Improper Output Handling, Excessive Agency, System Prompt Leakage, Vector/Embedding Weaknesses, Misinformation, and Unbounded Consumption (resource abuse / "denial of wallet"). Secure MLOps wraps the pipeline in controls: signed artifacts, access-controlled registries, scanned dependencies and monitored deployment.

Auditor's angle: use the list as a coverage checklist for any LLM app. The risk is treating an LLM like ordinary software and missing whole categories. Evidence: a threat model mapped to the OWASP list with a control and test per item.

Beyond breaches and outages, an AI incident includes: harmful, biased or discriminatory outputs; an LLM producing dangerous, defamatory or hallucinated content; severe performance degradation from drift; a successful poisoning or prompt-injection attack; or an unexplained mass shift in decisions. An AI incident response plan defines these triggers, roles (including data science and legal), severity tiers and a decision tree.

Auditor's angle: the risk is that nobody classifies a misbehaving model as an "incident," so it never enters the response process. Look for an AI-specific IR plan with explicit AI triggers, named roles, severity criteria and links to regulatory reporting.

• Many AI incidents are quality/ethics failures, not breaches.
• The plan must name who can disable a model and who notifies regulators.
• Define severity by impact on people/rights, not just system downtime.

Detection for AI relies on output and performance monitoring, drift/KRI alerts, complaint and escalation channels, and external/red-team reports — not just infrastructure alarms. A model can be "up" and healthy by ops metrics while making harmful decisions.

Auditor's angle: the risk is external parties (customers, media, regulators) detecting the failure before the organisation does. Look for live output monitoring, drift/accuracy KRIs with thresholds and owners, and a working complaint-to-IR pathway.

• Monitor outputs and outcomes, not only uptime/latency.
• KRIs: drift score, error rate by subgroup, override rate, complaint volume.
• "Social media noticed first" is a detection finding.

The AI-specific containment move is to disable or roll back the model to a known-good version, or fall back to a human/rules process. This depends entirely on having a model registry, versioning and a rollback path — so weak change management (Subtopic 3) directly cripples incident response.

Auditor's angle: the risk is that the only containment option is shutting the whole service off — far costlier. Look for a tested rollback capability, a documented fallback mode, and a clear authority to invoke it under time pressure.

• Rollback to prior model version is the fastest, lowest-impact containment.
• A human/rules fallback keeps the service running while the model is off.
• Containment authority must be pre-assigned and exercisable fast.

Model failures need model-aware investigation: was it data drift, a data-quality break, a poisoned input, a bad retrain, a configuration/threshold change, or an upstream feature pipeline change? Reproducibility and versioning of code, data and model are what make a real root cause possible.

Auditor's angle: the risk is "we restarted it and it seems fine" with no true cause, guaranteeing recurrence. Look for the ability to reproduce the failing model from pinned artifacts and a structured RCA that distinguishes the candidate causes.

• Differentiate drift vs data break vs attack vs bad change.
• Reproducibility (pinned code+data+seed) is the RCA enabler.
• Link RCA back to the control that should have prevented it.

When a model has already acted, containment is not enough — you must remediate the harm: retract or correct outputs, notify affected individuals, and reverse or re-adjudicate decisions already made (e.g. re-process wrongly denied applications, refund wrong charges).

Auditor's angle: the risk is fixing the model but leaving victims with the consequences. Look for a process to identify all affected subjects/decisions during the incident window, a redress mechanism, and records of corrections made.

• Scope the blast radius: which decisions, over what window, for whom.
• Provide redress: reverse, re-decide, compensate, and inform.
• Retain evidence of what was corrected for regulators.

AI incidents often trigger disclosure obligations: privacy-breach notifications, sector regulators, and emerging AI-specific reporting (e.g. serious-incident reporting under the EU AI Act for high-risk systems). After the fire is out, post-incident remediation and lessons learned fix the data/model/controls, tighten monitoring thresholds, and feed findings back into governance.

Auditor's angle: the risk is missed legal deadlines and a recurring failure because no systemic fix was made. Look for a disclosure decision (with legal input and timelines met), a tracked remediation plan with owners, and a closed-loop update to controls and the IR plan itself.

• Know that EU AI Act high-risk systems carry serious-incident reporting duties.
• Lessons learned must change a control, not just close the ticket.
• Tie the incident back to upstream gaps (testing, change mgmt, monitoring).