London

81 Chancery Lane
London
WC2A 1DD
0207 112 8493
View map

Edinburgh

1 Lochrin Square
92 Fountainbridge
Edinburgh
EH3 9QA
0207 112 8493
View map

Dublin

24A Baggot Street Upper
Ballsbridge
Dublin
D04 V970
0207 112 8493
View map

Registered Office

7 Bell Yard,
London,
WC2A 2JR
0207 112 8493
View map

Send us a message

CLOSE X
Contact Us
11.06.2024

Who are the Super-Predictors from Round 1 of the APR Life Conference Competition?

Background 

 

The IFoA Life Conference in November last year had many highlights, but we’re sure that all would agree that among them was the debut of the APR Prediction Competition. As Diamond Sponsors of the event, APR had to bring their A-game: not satisfied with merely testing visitors on their physical dexterity by means of a buzzing wire game, APR upped the stakes by challenging attendees to a fiendish test of their forecasting abilities. 

The rules of the competition were straight-forward. Participants were challenged to forecast the value of five metrics at two future timepoints (31st March 2024 and 30th June 2024). These metrics were carefully chosen to encompass a range of different demographic and economic assumptions important to actuarial work and are discussed in the following section of this article. 

For each participant, a ‘prediction score’ is calculated based on how close their predictions are to the actual values. There are then three winners – the entrant with the lowest prediction score for March, the entrant with the lowest prediction score for June, and the entrant with the lowest combined prediction score summed across the two dates. In addition to bragging rights, all of the winners will receive Virgin Experience Day gift vouchers. 

 

What are the metrics? 

 

Arguably the most important interest rate in the country, the Bank of England base rate is the rate of interest that the BoE charges to banks and other financial institutions; it has existed (in some form) as far back as 1694! The Monetary Policy Committee meet eight times a year to determine its value. 

Sliced bread is one of the items used in the ‘basket of goods’ for calculating the value of Consumer Prices Index, a measure of consumer price inflation. This is tied to Metric 1, in that the choice of bank rate is largely driven by the monetary policy aim of keeping inflation at a low and stable rate. 

Now for a nice demographic assumption! We chose Scotland for this metric for two key reasons: firstly, as a nod to APR’s Edinburgh office, and secondly, because National Records of Scotland helpfully release these statistics on weekly basis 

Currency risk is a concern for any institution which operates internationally, or which has assets or liabilities denominated in multiple currencies. In this post-Brexit world, the relative strength of the pound vs. the euro is a topic of some contention; and may be a very real concern for staff from APR’s Dublin office when visiting their London counterparts! 

Long-term government bond yields should be familiar to any actuary from their use in constructing yield curves, and gilts are of course an essential building block of any institutional investor’s portfolio. The value of this metric depends on the expectations of future interest rate movements as well as the supply and demand for bonds of various maturities, and is a key determinant in levels of business investment. 

 

How was the prediction score calculated? 

 

Suppose there are N participants in the competition. Let Pn,i be the nth participant’s prediction for the value of metric i, and let Ai be the ‘actual’ value of metric i at the time point of interest. The nth participant’s overall prediction score Sis then calculated as: 

What this effectively says is that we consider the absolute difference between a participant’s prediction and the true value for each metric, normalize by dividing by the mean across all participants, and then sum across the five metrics. Like golf, lower scores are better! 

One nice feature of the prediction score is that (by construction) the mean prediction score is equal to 5 exactly – so determining whether a score is better or worse than the average can be done at a glance. 

Other more robust or more complex statistical approaches are certainly possible – for example, using historical data to calculate an estimate of the volatility of these metrics. However, for our purposes we think the methodology we’ve used is sufficient to give a reasonable indication of a participant’s accuracy relative to their peers. (Do you agree?) 

 

What were the results? 

 

We’re saving a more detailed analysis of each of our metrics for a future article coinciding with the release of the June results – so you’ll have to keep your eyes peeled! However, in brief: 

With an impressive prediction score of 1.427, the winner for March is Alex Gilbank – congratulations! Alex was dead on with his predictions of Metrics 1 and 2, and performed better than average on the remaining three, making him the only entrant who received a sub-2 score. 

It was a tight race for second and third position, with six individuals falling in the 2 to 2.5 band, but the honour goes to Anthony Lee with 2.124 and Tasneem Harnekar with 2.276 respectively. Though not receiving a prize (yet!), they should still be proud of a podium finish and are in a strong position for the overall prize. 

 

What can we learn? 

 

Competitions such as these provide fertile ground for musing on the ‘wisdom of crowds’. If one had, for every metric, taken the median value across all participant’s predictions, this would have resulted in a prediction score of 2.411 – not unimpressive, coming in at somewhere between 7th and 8th place. Of course, this is operating with a reasonably limited number of datapoints, and with a reasonably homogeneous group – a more sizeable cohort could perhaps return even more impressive results.

Another consideration could be whether the historical data provided had an impact on the estimates provided, by means of some sort of priming or anchoring effect. Participants who visited APR’s booth were provided with: 

As such, participants’ predictions were likely influenced by this context: the trends which happened to be visible within this period, as well as the most recent values at the time that the competition took place.  

The time period considered can have a stark impact on the ‘story’ that a set of data tells. As an extreme example, consider the Bank of England base rate. If we’d looked at just the period from 2022 to 2023 (and not thought too hard about what we were looking at), a naïve analysis might have expected the rate to carry on climbing and reach 7% by March 2024 and nearly 8% by June! 

 

If we look over the period from 2001 to 2023, however… 

…this would likely prompt a rather different analysis. 

 

Conclusion 

 

We hope you’ve enjoyed this exposé on the APR Prediction Competition. Given the kinds of roles that actuaries get involved in, we believe a good sense for numbers is a key skill; indeed, it is something that we look for in any new recruits at APR and on which our clients have often complimented our staff. 

If you would like to delve further into this topic, we have found the following an interesting book: Superforecasting: The Art and Science of Prediction by Philip Tetlock and Dan Gardner. Do let us know any thoughts you have on this article or more widely. 

We’ll be in touch again soon with the final results when the data for the end of June is in. Until then, happy forecasting! 

 

 

Jacob Warbrick

June 2024