September Exam Results – A global analysis

The December IFoA exam results brought some pre-Christmas cheer at APR, as our students continued their excellent run of success. Across all of APR’s IFoA students, 87% of exams taken resulted in a pass.  In this article, FIA study co-ordinator Adam Smith and fellow student Deven Rickaby report back on the challenge that partner Gary Heslop presented to them:

Is there statistical evidence to support the premise that our exam success is anything more than good luck?

A basic approach: per exam analysis

In September 2019, 24 APR staff sat a total of 39 IFoA exams. 22 of these exam entries were from the Core Principles series and the remaining 17 from the Core Practice, Specialist Principles and Specialist Advanced series. A table showing the entries and results per exam is shown below (a mapping from the exams under the old syllabus to the new can be found at the bottom of this article for those of you who are more familiar with the old CT naming conventions):

We can see the earlier exams had a pass rate amongst APR students of 95% and 76% for the later exams.

As a first step, it is useful to compare our results against overall IFoA September 2019 pass rates, which can be calculated from the publicly available information on the numbers of entries and passes for each exam.

This shows how our students outperform the average in every exam, except for SA2. Whilst this initially appears encouraging, the low numbers of entries in individual exams renders a simplistic exam-by-exam analysis unsuitable, as we could just have had a good sitting.

Aggregate modelling and stochastic trialling

A more robust approach to analysing this problem is to aggregate all exam results and look at these aggregate results as a whole.  A sensible question to ask is:

Given the number of entries in each of the exams, what number of passes can we expect to achieve?

To answer this, we can think of the result of each exam as an independent random variable, taking the value 1 if the exam is passed and 0 if it is failed. For each exam entry, the probability of passing the exam can be set as the overall IFoA pass rate for the exam in the September 2019 sitting.  The number of passes in total is then the sum of these random variables, which is a random variable itself.  We care about the distribution of this random variable.

Each exam entry represents a Bernoulli trial, each independent but not identically distributed.  We assume independence because someone passing one exam shouldn’t affect another exam being passed[1]. The Bernoulli trials are not identically distributed because some exams are easier than others – for example 68% of the total population sitting CB2 passed but CP1 had a pass rate of only 34%. If the trials were identically distributed, the total passes would have a binomial distribution, which is fairly easy to calculate probability values for. However, as they are not identically distributed, the resulting distribution of the total passes is a Poisson binomial distribution, which is trickier to deal with.

When we approach the analysis stochastically, however, our question becomes relatively simple to answer. By simulating APR’s sitting a large number of times, we can generate a sample distribution of how we would expect APR to perform, assuming the probability of passing each exam is given by the overall IFoA September 2019 pass rate. We built a model to do just this, which followed the following steps for each simulation:

• For each person sitting an exam at APR, a random number was generated.
• If this random number was less than the pass rate, the exam was recorded as a pass.
• The total number of passes across all 39 sittings was summed and recorded.

These steps were performed 10,000,000 times, with the key output being the sampling distribution i.e. how many times n exams were passed, with n ranging from 0-39. This sampling distribution is shown below:

Further analysis showed that based on industry pass rates:

• The number of exams that APR were expected to pass as a group was 20 (this is the distribution mean).
• The 99.99% 1-tailed confidence interval for passes achieved was 0-30.
  • In the context of a hypothesis test, this shows that we can say, with 99.99% confidence, that the APR pass rate is higher than the industry average.
• In the 10,000,000 simulations, 34 passes were achieved 10 times – in other words, it was a one-in-a-million sitting. Other (roughly) one-in-a-million events include[2]:
  • One of the next 24 babies born in the U.S. becoming President.
  • Being killed by an asteroid strike.

This provides good evidence that APR’s pass rates are significantly better than the average IFoA pass rates from the September 2019 sittings and therefore our success is not just luck.  The results achieved by APR are impressive, although this model should be viewed in light of the assumptions and limitations discussed above.

If you wish to see the model please do get in contact with us to request a copy[3].

Possible reasons for this success

This set of results certainly isn’t a one-off; APR students have consistently achieved excellent results in recent years[4]. A number of our qualified actuaries have achieved the extremely rare feat of qualifying without failing a single exam, and several current students are not far from joining them. We are very proud of our exam record and it is often highlighted by new staff as one of the things that made them choose APR.

We believe there are a number of factors that all contribute in some way to our continued success:

• Recruitment – APR prides itself on recruiting the highest calibre graduates and experienced hires.
• Study support – students benefit from a generous study package to provide them with the best chance of exam success. There is also a strong culture of taking your full study allocation and managers protecting their staff when this is threatened by other priorities.
• Success breeds success – there is a culture of taking study seriously, qualifying quickly, and friendly rivalry no doubt helps.
• Success is valued and celebrated – APR recognise and call out exam success (in articles such as these amongst other things) and financially reward staff who progress well along the route to qualification.

It’s genuinely difficult to pick out individual achievements from the current results but here are some highlights:

Special mentions

• Ajay Kotecha for passing all 3 exams taken.
• Joe Barnett for picking up 3 of his remaining 4 exams.
• Our September 2016 intake of Adam Smith, Dennis Wang and Deven Rickaby all moving to within one exam of qualification without a failed exam between them.
• Our March 2018, Sept 2018 and Jan 2019 intakes all passing all exams taken.
• Chris Nash and Morgan Smith-Woodhams continuing their record of not having failed a single CAA or FIA exam between them.

Appendix

Table mapping the new syllabus:

For a more detailed review of the new syllabus you can always check out our previous article on this topic here.

Adam Smith & Deven Rickaby

January 2020

[1] You could argue that since not every exam is sat by a different person, this assumption doesn’t hold exactly. For example, someone sitting two exams could do poorly in the first one, and become demoralised to the extent they do worse in the second one. However, assumptions have to be made to simplify the problem, and we think this is a sensible one.

[2] https://www.discovermagazine.com/the-sciences/death-by-meteorite#.VroMTvHY85I

https://www.stat.berkeley.edu/~aldous/Real-World/million.html

[3]Authors’ contact details:
Adam Smith: adam.smith@aprllp.com
Deven Rickaby: deven.rickaby@aprllp.com

[4] Overall pass rates: Sept 2019 87%; Apr 2019 89%; Sept 2018 72%; Apr 2018 89%.