Confidence levels: how we tell clients what we know

A research report is read and then acted on. A general counsel decides whether to proceed with a transaction, an editor whether to publish, an investor whether to commit. Those decisions often turn on a single sentence. If that sentence rests on a court judgment, it can bear a great deal of weight. If it rests on one unnamed source, it can bear much less. A reader who cannot tell the two apart has to give every sentence the same weight, and that is rarely justified.

The four levels

Every finding in our reports carries one of four stated confidence levels.

• Confirmed. Supported by two or more independent, reliable sources, or by an authoritative primary record.
• Probable. Supported by credible evidence, with nothing reliable to the contrary, though it has not been independently confirmed.
• Possible. Consistent with the evidence, while resting on a single source or on inference.
• Unverified. Reported or alleged, and either untested or incapable of being tested. Included only where it is relevant to the reader.

The levels describe the evidence for a finding. They say nothing about how important the finding is, how likely a future event may be, or whether the conduct described is lawful. A minor detail can be Confirmed and a serious allegation can be Possible, and the report should make both clear.

Why we state them

The problem is an old one. In the 1960s Sherman Kent, writing for US government analysts, described how colleagues had read the phrase 'serious possibility' in an earlier estimate to mean very different odds. He proposed agreed meanings for what he called words of estimative probability. The proposal was not adopted at the time. Several intelligence communities have since adopted standard estimative terms. Journalism deals with a related problem through attribution. A careful report separates what the reporter has established from what a source has said, and both from what has merely been alleged.

Our levels serve the same purpose for findings of fact. They allow a reader to see, sentence by sentence, where the evidence is strong and where it is thin.

Readers act on sentences, so each sentence should show how much weight it can bear.

How levels are assigned and reviewed

The researcher who writes a finding proposes its level and records the sources behind it. A second member of the team then reviews the finding against those sources. The review tests whether sources described as independent really are independent. Two articles that repeat the same original claim count as one source. A company statement and a registry entry filed by the same company may also amount to one. Where independence cannot be shown, the level comes down.

An authoritative primary record confirms what the record itself shows. A registry entry confirms that a filing was made in certain terms on a certain date. It does not, by itself, confirm that the content of the filing is accurate, and the finding is worded accordingly. Levels can also change. New evidence can raise a finding or lower it, and every report is dated so that readers know when the assessment was made.

Reading a report that mixes levels

Most reports contain findings at several levels, and the summary carries the same labels as the body. Each level should be read with its own sentence. A paragraph of Confirmed findings lends no strength to the Possible finding beside it. Where a decision depends on a Possible or Probable finding, the sensible response may be further research, or a decision that allows for the finding being wrong.

Unverified material is included sparingly. It appears where a reader is likely to meet the allegation elsewhere, or where its existence is itself relevant, for example because it is circulating among counterparties or in the press. Recording it, clearly labelled and with what is known of its origin, serves the reader better than leaving them to encounter it without context. It is never presented as a finding of fact.

For us, the confidence level is part of the finding. It tells the reader what we know, how we know it and where the limits lie. A report that is candid about its uncertainties is easier to rely on than one that presents everything with the same assurance, and that is the standard we hold our own work to.