APR Prediction Competition – It’s The Final Update!
Background
Time flies when you’re making predictions! It’s been nearly a year since APR debuted its ‘Prediction Competition’ at the IFoA Life Conference – but with summer firmly behind us and autumn drawing in, the final results for the contest are now here.
We explained the premise for the competition in our earlier article this year announcing the March results, but in case you missed this, let’s quickly summarize:
Participants were challenged to forecast the value of five metrics at two future timepoints (31st March 2024 and 30th June 2024). These metrics were carefully chosen to encompass a range of different demographic and economic assumptions important to actuarial work and are discussed in the [previous] article.
For each participant, a ‘prediction score’ is calculated based on how close their predictions are to the actual values. There are then three winners – the entrant with the lowest prediction score for March, the entrant with the lowest prediction score for June, and the entrant with the lowest combined prediction score summed across the two dates.
In addition to bragging rights, all of the winners will receive Virgin Experience Day gift vouchers.
The Metrics
The values of the five metrics of choice at 30th June 2024 were as follows:
Metric 1: The Bank of England base rate
The Bank of England base rate was at a value of 5.25%¸having remained unchanged since August 2023.
At the Monetary Policy Committee meeting in August 2024, it was dropped to 5% – but unfortunately too late for anybody who predicted this!
Metric 2: The price of a loaf of sliced white bread
The price of a loaf of sliced white bread for June 2024 is 139p; similarly to Metric 1, this saw meagre variation over the period of the competition, and is in fact identical to its value at March 2024.
The standard basket of goods for calculating the Consumer Prices Index and Retail Prices Index change every year to reflect various factors, including availability of data and consumer trends; this year[1] saw the addition of air fryers, vinyl music, gluten free bread, and edible sunflower seeds!
Metric 3: The number of births registered in Scotland
The number of births registered in Scotland from April to June was 11,022 (compared to 11,553 in March 2024). We’d expect a certain amount of seasonality in the distribution of births – the following article by the ONS suggests some reasons why, such as bank holidays affecting the timings of both conceptions and deliveries.
Our historical graph highlights an important subtlety in this metric – the difference between the registration of a birth versus the birth itself. These are normally expected to track fairly closely, as, in Scotland, births must be registered within 21 days. However, during the pandemic (as can be seen) this delay was often much greater, illustrating that care must be taken when using one metric as a proxy to another.
Metric 4: The GBP – Euro spot rate
The GBP-EURO spot rate had a value of 1.1796 as of end-June 2024 (compared to 1.1694 in March 2024).
Foreign exchange rates are driven by a variety of factors, both economic and geopolitical. Though not the most volatile currency pairing, we would expect the strength of the pound and the strength of the Euro to be somewhat less tightly coupled post-Brexit.
Metric 5: 10-year UK government bond yields
10-year UK government bond yields were at 4.1644 measured at June 2024 (compared to 4.0330 in March 2024).
Predicting 10-year gilt yields is an interesting one, as in a sense it’s a ‘prediction of a prediction’ – in a few months’ time, what will we think about the 10-year time horizon? Adding in possible quirks with supply-and-demand for certain terms based on the timing of obligations makes for quite a complex picture.
The Winners!
But now, for what you’ve really been waiting for – the winners!
With a score of 1.343, Orry Pitts takes home the prize for the lowest prediction score for June – the lowest score achieved at either time point.
Scores were noticeably lower at the top this time round; Jenny Wood, with a score of 1.520, and Kleona Menti, with a score of 1.780, took silver and bronze respectively for June. A combined score of 4.071 means Kleona also came 2nd in the prediction contest over-all, with Kevin Tang close on her heels with a combined score of 4.355.
However – building on his win from March – the entrant with the lowest combined prediction score across March and June is Alex Gilbank with an impressively low 3.408. Despite several impressive performances in June, none were quite strong enough to dethrone Alex – the only competitor to achieve sub-2 scores at both time points. Congratulations!
A Bit of Data Analysis
The predictions made by our participants used a combination of intuition and on-the-spot reasoning, as we did not give any opportunity for people to take the data away and provide more complex calculations. But what if we had?
When making a prediction on the value of some metric at a future timepoint based on historical time series data, a common tool in the statistician’s toolbelt would be fitting an ARIMA (autoregressive integrated moving average) model.
To do this, we’ll be leaning on the auto.arima function from R’s forecast package, which takes out some of the heavy-lifting of parameter estimation by looping through several possible ARIMA models and finding the one with the lowest AIC (Akaike Information Criterion) value.
metric_n_arima <- auto.arima(metric_n_historical_data)
As this is a fairly informal analysis (and in the interest of fairness!) we’ll limit the data we supply our model with to that which the contestants had access to via the data sheets: the start of 2018 until the most recent data available at November 2023. In a true prediction exercise, more thought would likely be given to the period of historical data used, as well as potentially other adjustments needed to the data.
Unfortunately… this automated approach performed quite poorly! For its predictions for June 2024, it got a combined prediction score of 8.779; this puts it decidedly towards the rear of the pack! Drilling into this further, it seems like it actually performed reasonably well on all metrics except metric 5 – where it significantly overestimated.
It seems that the fatal flaw with this ‘robo-predictor’ who just turns the handle on the data is that it lacks appropriate context. Indeed, it seems that it fell into a similar pitfall to that outlined in our first article – of assuming that an increasing trend in gilt yields would continue to increase indefinitely.
Our poor robot also doesn’t, without further guidance, understand some of the concepts that our human contestants understood intuitively – that the Bank of England base rate will generally only move in steps of 0.25%, for example.
Certain naïve approaches – such as assuming that the value at the previous time step is maintained – performed significantly better than our ARIMA models, and the ‘wisdom of crowds’ approach of taking the median across all competitor values would’ve been in second or third place at either time step with a final ‘combined’ score of 3.996.
Final thoughts
First off, a final congratulations to all of our winners, who will, by the time of this article’s release, will no doubt have redeemed their prize!
If you took part and enjoyed our prediction competition in 2023, we are excited to announce that APR will once again be coming to Life Conference in 2024 (October 14th-16th), and we will run the competition again this year! So if you get the chance to drop by, please do take the time to submit an entry! If you didn’t take part last year, our condolences for missing one of the most significant events of 2023… but you can make up for it this year!
Finally, it isn’t just Life Conference APR will be at this year! Envious actuaries in the non-life sector can rejoice as we confirm our participation at GIRO, where we will be hosting another prediction competition (one slightly tweaked to reflect the utter apathy you must have for Life-centric metrics)!
We look forward to seeing as many of our readers there as possible, please drop by!
[1] Office for National Statistics (ONS), released 11 March 2024, ONS website, article, Consumer price inflation basket of goods and services: 2024
Jacob Warbrick
October 2024
