London

71-75 Shelton St
Covent Garden
London
WC2H 9JQ
01235 821 160
View map

Edinburgh

1 Lochrin Square
92 Fountainbridge
Edinburgh
EH3 9QA
01235 821 160
View map

Administrative Office

BH Office
Church Street
Ardington
Wantage
Oxfordshire
OX12 8QA
01235 821 160
View map

Send us a message
CLOSE X
Contact Us
15.01.2020

September Exam Results – A global analysis

The December IFoA exam results brought some pre-Christmas cheer at APR, as our students continued their excellent run of success. Across all of APR’s IFoA students, 87% of exams taken resulted in a pass.  In this article, FIA study co-ordinator Adam Smith and fellow student Deven Rickaby report back on the challenge that partner Gary Heslop presented to them:

Is there statistical evidence to support the premise that our exam success is anything more than good luck?

A basic approach: per exam analysis

In September 2019, 24 APR staff sat a total of 39 IFoA exams. 22 of these exam entries were from the Core Principles series and the remaining 17 from the Core Practice, Specialist Principles and Specialist Advanced series. A table showing the entries and results per exam is shown below (a mapping from the exams under the old syllabus to the new can be found at the bottom of this article for those of you who are more familiar with the old CT naming conventions):

We can see the earlier exams had a pass rate amongst APR students of 95% and 76% for the later exams.

As a first step, it is useful to compare our results against overall IFoA September 2019 pass rates, which can be calculated from the publicly available information on the numbers of entries and passes for each exam.

This shows how our students outperform the average in every exam, except for SA2. Whilst this initially appears encouraging, the low numbers of entries in individual exams renders a simplistic exam-by-exam analysis unsuitable, as we could just have had a good sitting.

Aggregate modelling and stochastic trialling

A more robust approach to analysing this problem is to aggregate all exam results and look at these aggregate results as a whole.  A sensible question to ask is:

Given the number of entries in each of the exams, what number of passes can we expect to achieve?

To answer this, we can think of the result of each exam as an independent random variable, taking the value 1 if the exam is passed and 0 if it is failed. For each exam entry, the probability of passing the exam can be set as the overall IFoA pass rate for the exam in the September 2019 sitting.  The number of passes in total is then the sum of these random variables, which is a random variable itself.  We care about the distribution of this random variable.

Each exam entry represents a Bernoulli trial, each independent but not identically distributed.  We assume independence because someone passing one exam shouldn’t affect another exam being passed[1]. The Bernoulli trials are not identically distributed because some exams are easier than others – for example 68% of the total population sitting CB2 passed but CP1 had a pass rate of only 34%. If the trials were identically distributed, the total passes would have a binomial distribution, which is fairly easy to calculate probability values for. However, as they are not identically distributed, the resulting distribution of the total passes is a Poisson binomial distribution, which is trickier to deal with.

When we approach the analysis stochastically, however, our question becomes relatively simple to answer. By simulating APR’s sitting a large number of times, we can generate a sample distribution of how we would expect APR to perform, assuming the probability of passing each exam is given by the overall IFoA September 2019 pass rate. We built a model to do just this, which followed the following steps for each simulation:

These steps were performed 10,000,000 times, with the key output being the sampling distribution i.e. how many times n exams were passed, with n ranging from 0-39. This sampling distribution is shown below:

Further analysis showed that based on industry pass rates:

This provides good evidence that APR’s pass rates are significantly better than the average IFoA pass rates from the September 2019 sittings and therefore our success is not just luck.  The results achieved by APR are impressive, although this model should be viewed in light of the assumptions and limitations discussed above.

If you wish to see the model please do get in contact with us to request a copy[3].

Possible reasons for this success

This set of results certainly isn’t a one-off; APR students have consistently achieved excellent results in recent years[4]. A number of our qualified actuaries have achieved the extremely rare feat of qualifying without failing a single exam, and several current students are not far from joining them. We are very proud of our exam record and it is often highlighted by new staff as one of the things that made them choose APR.

We believe there are a number of factors that all contribute in some way to our continued success:

It’s genuinely difficult to pick out individual achievements from the current results but here are some highlights:
One left

Special mentions

Appendix

Table mapping the new syllabus:

Core principles

For a more detailed review of the new syllabus you can always check out our previous article on this topic here.

Adam Smith & Deven Rickaby

January 2020

Adam Smith

[1] You could argue that since not every exam is sat by a different person, this assumption doesn’t hold exactly. For example, someone sitting two exams could do poorly in the first one, and become demoralised to the extent they do worse in the second one. However, assumptions have to be made to simplify the problem, and we think this is a sensible one.

[2] https://www.discovermagazine.com/the-sciences/death-by-meteorite#.VroMTvHY85I

https://www.stat.berkeley.edu/~aldous/Real-World/million.html

[3]Authors’ contact details:
Adam Smith: adam.smith@aprllp.com
Deven Rickaby: deven.rickaby@aprllp.com

[4] Overall pass rates: Sept 2019 87%; Apr 2019 89%; Sept 2018 72%; Apr 2018 89%.