Testing Controls, Oversight & Assurance

Writing a control is easy. Proving that it works is the hard part. Assurance is the work of testing controls and keeping the evidence. It is what lets you tell a customer, an auditor or your own board that you checked. This lesson covers how to test controls, how to make human oversight real and how to prepare for audits.

What assurance means

Assurance gives people reasons to trust. It has three parts: the control exists, it is applied and it works. A policy that says people review output is not assurance. Evidence that reviews happened, and that reviewers caught real errors, is.

Three lines of defense

• First line: the teams that build and run the systems. They own the risks and the daily controls.
• Second line: governance, risk and compliance. They set the rules, advise and monitor.
• Third line: internal audit. It checks independently and reports to leadership.

Smaller organizations may not have all three. Aim at least for some separation between those who do the work and those who check it.

Test your controls

CONTROL: Tier 3 systems have a trained reviewer who approves risky outputs
WHAT COULD GO WRONG: Reviewer approves everything without reading

TEST DESIGN
- Pull 30 recent approvals
- Plant 5 known-bad outputs (in a test environment) and see if they are caught
- Check reviewer training records and time spent per item
- Interview two reviewers about how they decide

RESULT
- Caught 3 of 5 planted errors; median time per item 4 seconds

CONCLUSION: Control exists, but is not effective
ACTIONS: Reduce volume per reviewer, add a checklist, retrain, retest in 60 days. Owner: {{name}}

Make human oversight real

Oversight is only real if the person can understand the output, has time, has authority to override and is accountable. Test each of these.

• Understanding: can the reviewer see the sources and reasons, not only the answer?
• Time: is the volume low enough that review is possible?
• Authority: can the reviewer reject or change the result without penalty?
• Skill: has the reviewer been trained on common errors?
• Evidence: do you record who reviewed, what changed and how often they disagreed with the AI?

Evidence

• Keep records in one place: policies, inventory, risk assessments, test reports, approvals, logs, training records and incident reports.
• Date and version everything.
• Keep what you need for as long as laws and contracts require, and no longer.
• Make it easy to retrieve. An audit is smoother when evidence is organized.

Preparing for an audit or customer review

• Know the scope: which systems, which standards and which period.
• Map your controls to the standard's requirements.
• Run a dry run before the real one, and fix the gaps.
• Be open. Auditors respect honest gaps with a plan more than hidden ones.
• Track findings to closure, with owners and dates.

Measure governance itself

• Share of AI systems in the inventory with an owner and a current review.
• Time to complete reviews by tier.
• Number of controls tested this quarter, and the share that pass.
• Open findings and how old they are.
• Incidents and near misses, and time to respond.
• Staff training completion and survey results.