APR 2020 Qualifiers – Skill or Luck?
In September 2016, APR hired an intake of three fresh-faced student actuaries – Adam Smith, Deven Rickaby and Dennis Wang. In April 2020, Adam, Deven and Dennis took their final actuarial exams, and each passed to qualify. In the years in between, not one exam was failed between them. Not a single one.
It is rare for actuaries to pass all of their exams first time – doing so is a laudable achievement individually – but it is almost unheard of for a whole intake to do this together. This is a first for APR and we are extremely proud of the 2016 intake for having done so. Their result is a testament to both their own capabilities and APR’s culture of exam excellence.
A Statistical Analysis
Anyone who has done the actuarial exams knows they are demanding, but perhaps the most fitting way of demonstrating Adam, Deven and Dennis’ achievement is through statistics.
Let’s ask ourselves the question:
If Adam, Deven and Dennis were three perfectly representative actuarial students, what is the probability that they would pass all of their FIA exams without a failure between them?
By “perfectly representative students”, we mean students whose likelihood of passing each exam is precisely equal to the IFoA global pass rate. For the statisticians out there, this is essentially a hypothesis test where the null hypothesis is “Adam, Deven and Dennis’ pass rates are equal to the global mean”, against an alternative of their pass rates being better than the global mean, and we are calculating the p-value of the test statistic.
Each of Adam, Deven and Dennis’ exam routes are below (excluding CT9 – now CB3), along with the corresponding global pass rates. The pass rates are specific to the sitting shown in the middle column.
| Adam | Deven | Dennis | ||||||
| Exam | Sitting | Pass Rate | Exam | Sitting | Pass Rate | Exam | Sitting | Pass Rate |
| CT1 | Apr-17 | 63.2% | CT1 | Apr-17 | 63.2% | CT1 | Apr-17 | 63.2% |
| CT2 | Apr-17 | 58.3% | CT3 | Apr-17 | 59.6% | CT3 | Apr-17 | 59.6% |
| CT3 | Apr-17 | 59.6% | CT5 | Apr-17 | 51.4% | CT5 | Apr-17 | 51.4% |
| CT4 | Sep-17 | 54.5% | CT4 | Sep-17 | 54.5% | CT6 | Sep-17 | 54.6% |
| CT5 | Sep-17 | 50.6% | CT7 | Sep-17 | 63.2% | CT7 | Sep-17 | 63.2% |
| CT6 | Apr-18 | 59.5% | CT2 | Apr-18 | 55.3% | CT2 | Apr-18 | 55.3% |
| CT7 | Apr-18 | 73.8% | CT6 | Apr-18 | 59.5% | CT4 | Apr-18 | 42.5% |
| CT8 | Apr-18 | 31.6% | CT8 | Apr-18 | 31.6% | CT8 | Apr-18 | 31.6% |
| CP3 | Sep-18 | 44.6% | CP3 | Sep-18 | 44.6% | CP3 | Sep-18 | 44.6% |
| ST5 | Sep-18 | 50.8% | ST5 | Sep-18 | 50.8% | ST2 | Sep-18 | 42.0% |
| CP1 | Apr-19 | 39.9% | CP1 | Apr-19 | 39.9% | CP1 | Apr-19 | 39.9% |
| CP2 | Oct-19 | 61.0% | CP2 | Oct-19 | 61.0% | CP2 | Oct-19 | 61.0% |
| SP6 | Oct-19 | 59.2% | SP6 | Oct-19 | 59.2% | SP5 | Oct-19 | 42.5% |
| SA7 | Apr-20 | 39.1% | SA7 | Apr-20 | 39.1% | SA2 | Apr-20 | 48.9% |
We make a key simplifying assumption – each exam attempt across the board is independent, both within each individual and between individuals. Adam passing CT1 has no impact on the probability of Adam also passing CT8, Dennis passing ST2 has no impact of the probability of Deven passing SA7 and so on.
This is not entirely realistic for a number of reasons. For example, students take exams in blocks/sittings and it is natural for students to have good or bad sittings on the whole which introduces some dependence within each sitting. Adam, Deven and Dennis also work together and have regularly taken the same exams in a sitting – we would expect they might collectively have learnt the material better for some subjects than others indicating cross-dependence between individuals.
It follows, therefore, that this model is not exact. Nevertheless, whilst the assumption is not perfect, it makes for a suitable approximation and having made it, we can more easily estimate the probability that an individual will pass every exam first time. This simplifies to the product of all the pass rates for that person’s exam set. The probability of all three passing everything first time is then the product of those three.
P(Adam passes without a single fail) = 0.632*0.583*0.596…*0.391 = 0.0001073
P(Deven passes without a single fail) = 0.632*0.596*0.514…*0.391 = 0.0000884
P(Dennis passes without a single fail) = 0.632*0.596*0.514…0.489 = 0.0000469
P(All pass without a single fail) = 4.4 * 10-13
That is about 1-in-2.4-trillion! If a group of three representative students kept taking the FIA exams over and over, we’d expect them to do what Adam, Deven and Dennis did around once in every 2 trillion attempts at the full course. Given that it took three years from start to finish, those poor students would have to keep at it for 7.2 trillion years in total. Let’s hope they stay in good health because that’s about 500 times the age of the universe!
So, is it fair to say we witnessed a 1-in-2-trillion event in April 2020? No! The explanation is of course much more logical – Adam, Deven and Dennis are much better than average (at passing actuarial exams) and their likelihoods of passing were much higher than the global pass rates. In statistics parlance, we have credible evidence to reject the null hypothesis at a very high level of significance. The results of April 2020 were not a colossal miracle – just the work of three exceptional students.
Ajay Kotetcha
November 2020
