What a tuning exercise actually involves

"We need to tune the monitoring system" covers everything from an afternoon adjusting two parameters to a six-month programme with an independent validation report attached. Teams commissioning one for the first time often do not know which they are buying, and the gap between the two shows up later, usually in an examination.

This is what the substantial version actually contains.

Phase one: understand the estate

An inventory of every rule in production — what it is meant to detect, what typology it maps to, when it was last changed, and by whom. In most firms this document does not exist, and assembling it is the first real finding.

Alongside it: coverage mapping. Which typologies in the risk assessment are addressed by which rules, and which are addressed by nothing. Gaps found here matter far more than anything the tuning itself will produce, and they are the reason to start with coverage rather than volume.

Phase two: data quality assessment

Every tuning exercise is limited by the data underneath it. Before analysis, establish completeness of the fields the rules depend on, consistency of customer segmentation data, whether transaction codes mean what the documentation says, and how reference data behaves at the edges.

Skipping this is the most common reason a tuning exercise produces confident conclusions that turn out to be wrong. Parameters calibrated on a field that is populated eighty per cent of the time are calibrated on a fiction.

A tuning exercise is a data quality exercise wearing a statistics costume.

Phase three: the analysis

For each rule in scope: the current alert population, the disposition history, the distribution of the underlying behaviour across the customer base, and how volume and detection respond as the parameter moves.

The two pieces of evidence that carry the weight are above-the-line testing — what a proposed setting catches — and below-the-line testing, a sampled review of what falls into the band being given up. Below-the-line is the part that is often skipped because it is laborious, and it is the part an examiner asks for first.

Phase four: the decision and the file

Recommendations go to whoever owns the risk, not to whoever administers the platform. What the file needs to contain:

• Scope and method, including why the sample period was chosen
• Data quality findings and their effect on confidence
• Results across a range of tested settings, not just the recommendation
• Below-the-line results and the residual risk accepted
• Approval — named, dated, at the right level of seniority
• Implementation plan, including how the change will be verified in production
• Review triggers

Phase five: implementation and confirmation

Deploy in a controlled way, then confirm the production behaviour matches what the analysis predicted. Volumes that diverge sharply from the model mean something was wrong in the assumptions, and it is much cheaper to find that in week two than at the next review.

How long it takes, honestly

For a mid-sized estate with reasonable data, the substantial version runs roughly two to four months of elapsed time. The analysis is not the long pole; data access, extract quality and getting decisions made are.

Anyone promising a full tuning exercise with defensible below-the-line evidence in two weeks is either working with an unusually clean environment or not doing below-the-line testing.

Two things worth deciding up front

Who validates it. If the same people who ran the analysis also sign it off, the exercise carries far less weight. Independence does not require a large second line — it requires someone outside the delivery line with authority to disagree.

What "success" means. If the objective is stated as a volume reduction target, the exercise will hit it and the control may be worse. State it as: a defensible calibration, with coverage evidenced, and volume reduced where it can be reduced safely. The distinction sounds pedantic until someone asks you to justify the result.

More insights