Dimensionality Reduction

Principal Component Analysis and its Alternatives

Principal Components Analysis (hereafter referred to as PCA) is a data analysis technique derived from linear algebra. It is used fairly regularly throughout the world of statistics and machine learning, such as modelling survival time and even reconstructing pictures of faces! Naturally, actuaries may be thinking “what does PCA do?” or “how can we apply this to our models?”. Indeed it is already used somewhat to model interest rate risk[1], though there are downsides and alternatives to it.

What is the purpose of PCA?

PCA is used to reduce the number of variables in a dataset into a few linear combinations of the original variables. For example, if I wanted to price a life insurance contract based on age, income and gender, I might find that age and income are quite correlated and so measure a combination of the two instead. This could make me want to change my model to only using 2 variables; gender and (age + income)/2 [2]. Perhaps the example is a bit simplistic, but when dealing with, say, 50 or more factors per observation, it can be computationally useful to crunch them down!

PCA uses principles of linear algebra to actively seek out which combinations of variables provides the most variance within the dataset. In geometric terms, it tells you which angle to look at the data to see the most variability within it. It then finds the next most variable angle to look at the data, subject to this second angle being perpendicular to the first and so on.

This is easily illustrated in 2 dimensions, and is quite similar (though not fully equivalent) to the least squares regression of one dimension onto the other. In 3 dimensions it is still possible to illustrate, and in 4 dimensions and above it is much harder.

How does it work?

Firstly, your data must be relatively standard. Each variable must have a marginal mean of 0 and variance of 1. This can simply be achieved by subtracting the sample mean off of each observation and dividing by the standard deviation.

Another, milder assumption is that the data follows a multivariate normal distribution, though this is not strictly necessary. In certain situations, such as claims frequency, it could very much be the case that your reconstructed values may have negative components. This clearly does not make sense, and so using PCA in this case may not be suitable.

This is where it gets a bit mathematical; the next step is to find the covariance matrix of the data, then take its eigenvalues and eigenvectors. That’s it. You now have the principal components of your data. Each eigenvalue gives the variance along the axis of its corresponding eigenvector, and so the first principal component is the eigenvector with the biggest eigenvalue.

These eigenvectors span the space of the observations, and so can be used to reconstruct the data in its original form. If more variance is distributed along fewer axis, then it will take fewer principal components to do a good enough job at reconstructing the data.

Further, these components are uncorrelated pairwise of each other, which is a decent help when working with their interactions with each other.

Case Study 1 – Risk-Free Rate Curves Using PCA

Risk-free rate curves, specifically spot yield curves, are integral to a lot of actuarial work. One particular aspect in which they are used is in capital modelling, where one of the aims is to find a 99.5^th percentile interest rate curve. Though generally you look for the change in interest rate curves as opposed to the full curve itself, here we will be looking at the full curve, as its easier to directly compare methods.

Spot yield curves can be thought of as vectors rather than continuous curves, with data for the spot yields at regular intervals such as yearly or half-yearly. These vectors both very long (30-50 elements) and highly correlated, especially at the tail. This makes it an ideal case for PCA analysis. The trick is to view each spot yield at time T as an observation, rather than taking each curve itself as the observation.

To illustrate this, I pulled some data from the US Treasury [3] to obtain the yield curves over the past 17 years. Each curve is made up of 11 data points ranging from 1 month to 30 years. Here’s what they look like when standardised:

As mentioned before, it’s good to check the marginal distributions of the variables to see if they look normal, however given the lack of data available to me, it is almost impossible to draw conclusions about the marginal distribution, and so this step is skipped. Now let’s look at the first 3 principal components:

From this analysis, it looks like the first principal component is not too far away from a uniform vector. This makes sense, as its effectively saying that the biggest differentiator of the yield curves shown before is how high they are, as opposed to how much they curve or bend. This should be quite clear from the graph before.

The second principal component looks like a typical yield curve itself! It appears that this one sets out the initial gradient of the curve, and also controls the direction, as in whether it’s bending up or down.

The last principal component is a bit more tricky to interpret, what this one shows is how flat or how much of a “hump” the curve has. As we see in a second, it is by far the least important of the 3.

Now we have the first 3 principal components, we can analyse how much of the total variance is captured by them. This can be displayed by plotting their eigenvalues on a graph and normalising by the sum of the eigenvalues.

This graph shows that in fact 90% of the difference between the curves are simply their vertical position on the graph, and all other details are only worth 10% of the variance. In fact, those 3 first principal components explain 99.7% of all variance in the data. This is certainly enough to reconstruct the curves. Here are some examples of reconstructed curves using only linear combinations of them 3 components:

As can be seen here, the principal components seem to smooth out the curves, only keeping their essential shape, but we have changed our number of free variables down from 11 to 3, and they’re pairwise uncorrelated! Hence, we can univariately find the 99.5^th percentile of each component by fitting a distribution to the data and finding the percentiles of that distribution.

Overall, this process makes the problem much simpler to compute, and easy to explain as the principal components have straight forward explanations, though this is very much not always the case [4]. Though it is true that the method does lose some information, it makes the problem much more manageable.

What is Non-Negative Matrix Factorisation?

Non-negative matrix factorisation (NMF) [5] is another data science technique used for reducing dimensionality in a different way than PCA. It assumes a Poisson of the elements of the original matrix, which isn’t unreasonable for non-negative data, however, also isn’t completely ideal for non-integer data.

Without going too far into the maths, suppose you have a \(N \times p\) matrix. NMF will find 2 matrices, W and H with dimensions \(N \times k\) and \(k \times p\) that when multiplied together give you an approximation of your original matrix. The value of k is similar to the number of principal components in PCA, but they don’t necessarily have variances associated with them.

The algorithm used to find these W and H matrices is an imperfect iterative one that converges on the solution. In fact, an analytical solution is likely impossible, as the problem has multiple valid solutions.

Case 2 – Risk-Free Rate Curves Using NMF

Now the drill is effectively the same, but with using a different method. Note how this time we don’t need to standardise the data, and we can use it in its raw form! Using Python’s Scikit Learn package, I can directly find the W and H matrices and examine the components. Here are the results:

This is really interesting! We can see that 2 of our components are almost identical in shape to the principal components computed before! However, we no longer get the “lift” component that we had before, and instead we get a steep slope downwards. Either way, let’s see how well it reconstructs the data.

These reconstructions look a lot more volatile than the PCA reconstructions, especially the one on the right. That being said, for the more regular curves, it matches almost perfectly. Now we can compare the two methods and see what fits better for certain applications.

NMF vs PCA

Before the comparison begins, it’s important to talk a bit about the framework in which these methods sit. This form of analysis comes under a section called “unsupervised learning”, meaning that there are very few objective measurements of success. There are no explicit labels on the data, unlike a classification or regression problem, so instead we need to use more implicit measures of success.

One of these measures is the error when reconstructing the curves to their original form. Both methods have an implicit objective to reduce these values, and so it would be a natural choice. I measure the error in 3 ways, squared distance, absolute distance, and the L2 norm per curve (Pythagorean distance). Here are the results:

So it seems that on all 3 fronts, NMF has less error than PCA. This is rather unexpected, as in most situations, PCA exactly minimises the L2 norm. To readers confused by this, remember that the PCA method required normalisation of data, then a reconstruction of that data. When it was normalised, the structure of the data was altered, and curves that were flatter in shape were inverted to slope downwards. It is entirely reasonable to think that there are losses that occur during this normalisation and reconstruction process that aren’t present for the NMF method.

Other Dimensionality Reduction Methods

These days, we can train incredibly complex non-linear methods to find patterns in data that PCA and NMF cannot. One such method is the use of autoencoders. Autoencoders use neural networks, more of which can be found here [6]. The method effectively creates a bottleneck in the neural network to map each data observation into a lower dimensional space (usually 2 or 3 dimensional).

This method is currently mainly used for advanced machine learning methods and data compression, rather than for unsupervised data analysis. This means it is not quite useful to us yet, however with the amount of research taking place with regards to neural networks, it is definitely something to watch.

Conclusion

PCA has long been the standard for dimensionality reduction (other than just simply deleting variables), but new methods are certainly worth considering when picking how to model your data. Each method has its upsides and downsides, from ease of interpretability to separation of components. It depends on your goals, preferences, and most importantly – what the regulator thinks!

Footnotes

[1] The Yield Curve and its Components | Thomas T. Bjerring

[2] This would require some standardisation, as for income to not dominate over age.

[3] Interest Rates Data CSV Archive | U.S. Department of the Treasury

[4] This is exceptionally well shown in the book “Elements of Statistical Learning”

[5] Elements of Statistical Learning

[6] Machine Learning Applications to Insurance Modelling

Liam Gettins

June 2025