Shining the Spotlight on Expert Judgement
Expert judgement is inherent in actuarial work but is something that often receives less attention than may be appropriate. This article provides an introduction to the subject within the context of actuarial models, drawing on research carried out by Roger Austin and the IFoA[1].
Models and expert judgement
Perhaps a good place to start is to consider “what is a model?”. A model is effectively a simplification of reality, and thus some judgement is required in all models. The level of impact these judgements have will vary; some judgements have small impacts whilst others have significant impacts.
So what is the difference between a judgement and an expert judgement? This will come down to the company’s expert judgement policy, but materiality will be an important factor. Expert judgements are just a subset of judgements, and sometimes there can be a grey area as to whether a judgement should be considered an expert judgement. The following diagram illustrates this by considering judgements relating to mortality for an annuity portfolio:
Most will agree that mortality improvement lies within the expert judgement space, whereas some data manipulation/cleansing lies within the judgement space. Mortality risk factors could potentially sit in either and are an example of judgements lying within the grey area.
Expert judgements can cover quite a range of different aspects, but generally fall into one of three broad areas: methodology, assumptions and approximations.
Consequences
Now that we have explored what an expert judgement is, it is worth thinking about what is the consequence of a judgement being considered an expert judgement? It comes down to additional rigour, and this additional rigour is required across four main areas:
- Approach to forming a judgement: How in-depth and structured is the thought process used to reach decisions? The additional rigour may manifest itself via having a more formal process around the judgement, for example.
- Documentation: The steps in forming the judgement will need to be documented in more detail for expert judgements.
- Validation: The additional documentation will also help with the increased rigour around validation of the judgement.
- Monitoring: Expert judgements will need to be monitored more closely, not just on a scheduled basis but also triggers for non-scheduled review, to ensure they remain valid.
Forming expert judgements and decision-making
It is interesting to think about the expert judgement process at a high level, as illustrated by the following diagram. Identifying potentially useful sources of information is an important part of the process. These sources of information are used to inform the views of the experts. The decision-makers then make use of both the views of the experts and the sources of information to reach a decision:
Having a good process around expert judgement is very important, but there is not a “one size fits all” answer; rather, the approach should be tailored and proportionate in line with materiality.
Framework
Drawing these ideas together, it feels sensible to put in place an expert judgement framework to ensure expert judgement is managed appropriately within the company. Elements of such a framework are likely to include:
- Expert judgement policy: including the definition of expert judgement.
- Governance structure: consistency of approach helps with overall credibility.
- Strong process: helps to create rigour.
- Documentation: evidence and rationale behind decisions again demonstrate consistency.
- Validation: good process and appropriate documentation exposes the thought processes behind the judgements, enabling them to be reviewed and challenged.
- Proportionality: avoid using a sledgehammer to crack a nut!
Process
Picking up on the process, the following diagram provides an overview of an approach which can be used to deal with expert judgement:
So what would be involved in each of these stages?
- Preliminary assessment of judgement: Does the judgement meet the definition of an expert judgement? How material is it? If it is not very material, then it is unlikely to be sensible to expend the same amount of time and effort as for an expert judgement.
- Defining the problem: There are several aspects to this important stage e.g. articulating the need for the judgement, identifying suitable experts, and preparing a brief for those experts.
- Elicitation of expertise: This involves deciding on the best approach to obtaining the information from the experts, analysing their views and then going back to ask for clarity where necessary.
- Decision making: Decisions makers will be scrutinising the views of the experts along with their own sources of information to reach an informed decision.
- On-going monitoring: This will involve regular scheduled reviews as mentioned previously, but also the monitoring of triggers which would result in the need for unscheduled reviews e.g. the occurrence of a pandemic.
Uncertainties around judgements
While experts may often state a point estimate of their expert judgement, it is important to recognise that there will usually be a range around this point estimate which they would consider to be plausible. As such, it is useful to ask each expert for a plausible range around their estimate e.g. what would they consider to be the 25th and 75th percentiles around their central estimate. This converts into that expert’s plausible range. When you then get the views of several experts, all their individual plausible ranges can be combined to give a sense of an overall plausible range for the expert judgement in question; in effect, an indication of the level of uncertainty around that judgement. The plausible range then feeds into the impact range, providing an indication of the sensitivity of the output metrics of interest to the uncertainty around the judgement, as illustrated in the diagram below:
Conclusions
Expert judgement is inherent in models. It is something that will always be required, and we should embrace it rather than shy away from it. The framework that we have touched upon in this article (i.e. having an expert judgement policy; a sensible governance structure; a robust, but proportionate process tailored to the company’s needs; all backed up by good documentation, validation and monitoring) should facilitate a successful approach to expert judgement.
I would like to finish with one final thought: by its nature, the correct answer is unknown – possibly even unknowable – and to think otherwise is potentially dangerous. It is important to always keep this at the forefront of our minds when considering expert judgements.
Conor Marshall
February 2021
[1] ‘Expert Judgement’ paper by Roger Austin and other members of the IFoA’s Solvency & Capital Management Working Party