Testing Actuarial Models
Today, we’re talking about testing – specifically, some tips and tricks for testing actuarial models that are compiled to some variety of source code.
Let’s start with a question: which of these actuarial models is easier to test?
Exhibit A
def actuarial_model(
data
) -> result:
# Keep every operation inside this function
...
Exhibit B
import numpy as np
import pandas as pd
from dataclasses import dataclass
@dataclass
class RatingParameters:
base_rate: float
...
def read_property_data(
db,
) -> pd.DataFrame:
...
def compute_exposure(
tiv_pct: np.ndarray,
...,
) -> np.ndarray:
...
...
def actuarial_model(
rating_parameters: RatingParameters,
...,
):
# Delegate operations to other functions inside this function
properties = read_property_data(...)
...
exposures = compute_exposure(...)
...
You’ll have to pardon the abstraction but hopefully you get the gist of what I am about to say.
To be sure, Exhibit B is more abstracted but it’s not necessarily more complicated as some may believe:
- First, you can read actuarial_model() in plain English and broadly understand what it is trying to do: each functional call represents a different stage of the calculation or, as programmers would call it, a different unit of software that makes up the whole
- Second, you can still test actuarial_model() as a whole, if you want to, but you now have added flexibility: you can test all of the units that make up actuarial_model() in isolation
- Third, and this is really important, we are carefully defining the input arguments required at each stage of the calculation. This is in stark contrast to Exhibit A, which unfortunately falls victim to the so-called God-object phenomenon: one big, opaque dataset with little explanation. When you write code in this way you’re making it hard to understand and/or test. It’s better to keep functions focused and determinable. The first example is bad because what even is data? How is it defined? What columns should it have? We don’t understand the contract from the code alone and that is a problem.
As we’ll come onto, all of these concerns make testing easier.
Design Matters (a lot!)
Unfortunately, a large number of actuarial models tend to adopt the monolithic Exhibit A approach and it’s not without reason,
- Most codified models were previously housed in Excel which fundamentally encourages a monolithic design approach. You can’t really ‘split’ an Excel model into distinct atomic units – what are the units exactly? It’s understandable that in the process of migrating something from Excel into an open source environment you end up basically migrating the same design flaws. I spoke about this at the RStudio Conference back in 2022 at length, and I’d encourage anyone interested in learning more to give this a watch!
- Actuarial testing tends to be quite standardised: assemble a batch of insurance policies, churn them through the model, look at the output – does it match our expectations? In other words, does it reconcile to another implementation we made in Excel / other named vendor platform? If so, we’re golden… if not, we need to investigate further (but we likely do not know the root cause until we do so!)
Let me be clear: nothing is wrong with batch testing policies and subjecting them to reconciliation. It’s a really important part of validating actuarial models. I think we should always ensure that models satisfy reconciliation checks.
That said, it is my view that this sole approach to testing is not without risk: it is opaque and not transparent enough on its own to ascertain why a system works as expected. Sure, the numbers match but do you know why? What about mismatches? You have no choice but to trawl through the entire calculation yourself to uncover why something doesn’t work in the way you expect it to: an extremely time-consuming exercise!
We can do better and it starts with better design first and foremost: you cannot assemble an effective test suite without it which is why I stress to you the importance of this first and foremost!
Then what?
Ok, so you have a good design i.e. you’ve split your model into different units. Remember: this doesn’t mean that we are about to abandon batch reconciliation tests (this will come later, don’t worry).
Now what? How do we write good tests?
To make things easier, I’m going to start with what makes a bad test before moving onto some examples of good tests.
Sidenote: Examples
For the rest of this article we’re going to focus on testing the following simple reinsurance model using the library pytest,
from dataclasses import dataclass
def _validate_non_negative(name: str, value: float) -> None:
"""Helper to check for non-negative values"""
if value < 0:
raise ValueError(f"{name} must be non-negative")
def compute_recovery(
loss: float,
excess: float,
limit: float,
) -> float:
"""Return the recoverable amount under an excess-of-loss layer."""
_validate_non_negative("loss", loss)
_validate_non_negative("excess", excess)
_validate_non_negative("limit", limit)
return min(
max(loss - excess, 0),
limit
)
@dataclass(frozen=True)
class ReinsuranceResult:
"""Aggregate result from applying one excess-of-loss reinsurance layer."""
losses: list[float]
recoveries: list[float]
@property
def net_losses(self) -> list[float]:
return [
loss - recovery
for loss, recovery
in zip(
self.losses,
self.recoveries,
strict=True
)
]
def apply_reinsurance(
losses: list[float],
excess: float,
limit: float,
) -> ReinsuranceResult:
_validate_non_negative("excess", excess)
_validate_non_negative("limit", limit)
recoveries: list[float] = []
for loss in losses:
_validate_non_negative("loss", loss)
recovery = compute_recovery(loss, excess, limit)
recoveries.append(recovery)
return ReinsuranceResult(
losses=losses,
recoveries=recoveries
)
Tests in pytest are really easy to write: you write a function prepended by test_ and include assert statements in the body to define each test. You’ll see some examples of this below!
Bad Type I: Repeating Implementation
Observe,
def test_bad_example_repeats_the_implementation_do_not_copy():
"""
This… is checking that two 'copies' of each other give the same answer (pointless)
"""
loss = 120
excess = 100
limit = 50
expected = min(
max(
loss - excess,
0
),
limit
)
# uh, oh! Straight copy from the source code!
assert compute_recovery(
loss=loss,
excess=excess,
limit=limit
) == expected
This is bad because you’re not testing anything meaningful: all you’ve done is repeat the implementation in two places – a maintenance headache!
Bad Type II: Black Box Style
Observe,
def test_bad_example_black_box_do_not_copy():
"""
If this test breaks, it's harder for another developer / actuary to diagnose the root cause
"""
result = apply_reinsurance(
losses=[80, 120, 200],
excess=100,
limit=50,
)
assert result.total_net == 330
# err, why?!
Sure, you can work through the numbers yourself but if you have to do this in order to understand the test then it’s not a good test.
Bad Type III: Ignoring the Mathematical Contract
Observe,
def test_bad_example_barebones_do_not_copy():
"""
In isolation, this can easily be 'gamed'
"""
loss = 120
excess = 100
limit = 50
assert compute_recovery(
loss=loss,
excess=excess,
limit=limit
) <= limit
This is an example of an under-specified test which I alluded to earlier: it’s true that any computed recoveries should be less than or equal to the limit but if that’s the only thing we test, then somebody could easily game the model (e.g. by getting it to produce a random number between 0 and limit… yikes!).
Good Type I: Respecting the Mathematical Contract
Good tests start with respecting an expected mathematical contract. More concretely, consider the following series of unit tests,
def test_compute_recovery_no_recovery_below_excess():
assert compute_recovery(
loss=80,
excess=100,
limit=50
) == 0
def test_compute_recovery_partial_recovery_above_excess():
assert compute_recovery(
loss=120,
excess=100,
limit=50
) == 20
def test_compute_recovery_capped_at_limit():
assert compute_recovery(
loss=200,
excess=100,
limit=50
) == 50
What I like about these tests is as follows,
- Each test is comprehensible: the name of each test is self-documenting and expresses what it is trying to achieve
- Each test is meaningful from a domain perspective: for example, as actuaries it makes sense to us that we shouldn’t expect any recoveries below the provided excess
- We cover most feasible cases: i.e. below the excess, in the layer and above the limit. Note well that writing effective tests for an actuarial model is not necessarily about covering most of the codebase , it’s about covering feasible business scenarios which is what this does!
Good Type II: Property Tests
I mentioned earlier that one of the drawbacks of Bad Type III: Ignoring the Mathematical Contract is that it doesn’t test core behaviour well enough.
That doesn’t mean you should ignore properties altogether!
One slick approach you can use for testing a wide range of expected properties in Python is via the hypothesis library.
Observe the following,
from hypothesis import given
from hypothesis import strategies as st
@given
(
loss=st.integers(min_value=0, max_value=1_000_000_000),
excess=st.integers(min_value=0, max_value=1_000_000_000),
limit=st.integers(min_value=0, max_value=1_000_000_000),
)
def test_compute_recovery_properties
(loss, excess, limit):
recovery =
compute_recovery(
loss=loss,
excess=excess,
limit=limit
)
assert recovery >= 0
assert recovery <= limit
if loss <= excess:
assert recovery == 0
if loss >= excess + limit:
assert recovery == limit
This allows you to subject a given series of values to standard expected properties (e.g. the recovery should always be 0 if the excess is higher than the loss) without
having to write a whole bunch of tests manually.
Good Type III: Integration Tests
These tests – named integration tests because they cover the combined work of multiple units – are supposed to represent a move from a single calculation to a small hand-worked scenario with outputs an actuary can interrogate without trusting a black box.
In our working example, we may want to test something broader than compute_recovery(). For instance, how can we write tests for apply_reinsurance(), which leverages compute_recovery() in an interative fashion?
Remember the black-box example,
def test_bad_example_black_box_do_not_copy():
"""
If this test breaks, it's harder for another developer / actuary to diagnose the root cause
"""
result = apply_reinsurance(
losses=[80, 120, 200],
excess=100,
limit=50,
)
assert result.total_net == 330 # err, why?!
Here’s a better approach:
def test_apply_reinsurance_simple_hand_worked_scenario():
"""
Handworked example: actuaries can follow this!
"""
result = apply_reinsurance(
losses=[80, 120, 200],
excess=100,
limit=50,
)
assert result.recoveries == [0, 20, 50]
assert result.net_losses == [80, 100, 150]
assert result.total_gross == 400
assert result.total_recovery == 70
assert result.total_net == 330
I like this because it sort of mimics the way an actuary may step through the calculation themselves:
- First, let’s check the recoveries are correct… ah yes, 0 (doesn’t attach at all), 20 (20 above the attachment point) and 50 (we’ve reached the limit.
- Given the losses incurred and the recoveries, here’s what I think the net position per loss should look like
- That means, total net position is… let me see… 330
Good Type IV: End-to-end Tests
Finally, you may want a small number of canonical end-to-end tests. These are your tests which cover the entire model end-to-end.
Batch testing works towards this goal already but – for transparency – I often find it helpful to have one or two cases documented in a .yaml file which is then imported into a broader pytest routine at runtime.
For example, you might document your canonical end-to-end test like follows (in a .yaml file),
name: apply_reinsurance
description: >
Simple excess-of-loss test with three losses. Designed to check attachment,
partial recovery, and exhaustion in a small scenario an actuary can inspect.
model:
module: reinsurance
function: apply_reinsurance
inputs:
losses:
- 80
- 120
- 200
reinsurance:
excess: 100
limit: 50
expected:
loss_results:
- loss: 80
recovery: 0
net_loss: 80
- loss: 120
recovery: 20
net_loss: 100
- loss: 200
recovery: 50
net_loss: 150
totals:
gross_loss: 400
recovery: 70
net_loss: 330
And then use this in your test,
import yaml
from pathlib import Path
SCENARIO_DIR = Path(__file__).parent / "scenarios"
def load_scenario(filename: str) -> dict:
with (SCENARIO_DIR / filename).open() as scenario_file:
return yaml.safe_load(scenario_file)
def test_apply_reinsurance_complex_hand_worked_scenario():
"""
Hand-worked scenario: formal & reviewable
This becomes very useful for 'larger' routines or complicated classes
where regressions need to be managed carefully.
"""
scenario = load_scenario("regression_001.yaml")
layer = scenario["inputs"]["reinsurance"]
result = apply_reinsurance(
losses=scenario["inputs"]["losses"],
excess=layer["excess"],
limit=layer["limit"],
)
actual_loss_results = [
{
"loss": loss,
"recovery": recovery,
"net_loss": net_loss,
}
for loss, recovery, net_loss in zip(
result.losses,
result.recoveries,
result.net_losses,
strict=True,
)
]
assert actual_loss_results == scenario["expected"]["loss_results"]
assert result.total_gross == scenario["expected"]["totals"]["gross_loss"]
assert result.total_recovery == scenario["expected"]["totals"]["recovery"]
assert result.total_net == scenario["expected"]["totals"]["net_loss"]
Automated Testing
It would be remiss of me not to mention that part of the power of documenting tests (as above) using pytest – as opposed to running your own tests manually in ad-hoc throwaway scripts – is that you can automate these tests to run in an event-driven fashion.
If you’re tech-savvy enough to have access to a GitHub / GitLab platform, you should look into Actions and CI/CD workflows respectively.
These approaches allow you to create rules like Whenever somebody pushes some new code to the repository, automatically run all of the associated tests, allowing you to identify regressions in our models much more efficiently than doing so manually.
If you have written effective, well-documented tests then understanding the root cause of any regressions is as simple as reading the logs associated with your CI / CD implementation.
Conclusion
In conclusion, testing actuarial models might seem like a solved problem but I hope you’ve come away from this with an appreciation that solving this problem effectively requires:
- A good design approach from a software perspective
- An understanding of the different ‘types’ of unit tests and how they can be leveraged to your advantage.
- A solid understanding of the actuarial domain so that tests can reflect real scenarios likely to be experienced by the business
Johnny Breen
May 2026
