Statistical Model-Based Composite Indicators for Tracking Coherent Policy Conclusions

Abstract

In the last years, with the data revolution and the use of new technologies, phenomena are frequently described by a huge quantity of information useful for making strategic decisions. A priority for policymakers is having simple statistical tools useful to synthesize data. Such tools are represented by composite indicators (CIs). According to the glossary of statistical terms of OECD (The OECD-JRC handbook on practices for developing composite indicators. Paper presented at the OECD Committee on Statistics, 2004), OECD-JRC (Handbook on constructing composite indicators. Methodology and user guide, OECD, Paris, 2008), a CI is formed when manifest (observed) indicators are compiled into a single index, on the basis of an underlying model of the multi-dimensional concept that is being measured, and weights commonly represent the relative importance of each indicator. CIs are increasingly used for bench-marking countries’ performances and the methodological challenges raise a series of technical issues that, if not adequately addressed, can lead to CIs being misinterpreted or manipulated. Yet doubts are often raised about the robustness of the resulting countries’ rankings and about the significance of the associated policy message. In this paper, we propose a model-based approach for the construction of CIs with a hierarchical structure where the CIs (first and second order) are estimated using the Hierarchical Disjoint Non-Negative Factor Analysis (Cavicchia and Vichi in Hierarchical disjoint non-negative factor analysis. Manuscript submitted for publication, 2020) in a LS framework. In order to assess the methodology of construction of a CI, a set of properties is proposed and applied. Some well-known CIs, such as the Human Development Index and the Multidimensional Poverty index, are taken into consideration to show the importance of those properties. Therefore, we include into our proposal the most frequently used approaches in the literature of CIs, and we evaluate the model to assess their performances.

Appendix: Examples

Example 1

Let us consider the vector $\mathbf{g}$ as a normalized normal random vector of 15 statistical units, the vector $\mathbf{c}$ as a vector of 3 weights equal to 1 and the matrix $\mathbf{B}$ as an identity matrix of order 10, that is with all weights equal to 1. So, we have 10 MIs explained by 3 SCIs with the constraint that each MI can be explained by one SCI, only.

Let us consider the matrix $\mathbf{V}'$ given by:

$$\begin{aligned} \begin{bmatrix} 1 &{} 1 &{} 1 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0\\ 0 &{} 0 &{} 0 &{} 1 &{} 1 &{} 1 &{} 1 &{} 0 &{} 0 &{} 0\\ 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 1 &{} 1 &{} 1\\ \end{bmatrix} \end{aligned}$$

which describes the relationships between MIs and SCIs.

We can generate the data starting from (5) and hypothesizing 3 levels of the error $\mathbf{E}$: small, medium and large. The corresponding matrices $\mathbf{X}$ are compared in Fig. 5. In Fig. 5 (left) the small error does not modify the columns of $\mathbf{X}$ that can be well distinguished. The same does not apply to the matrix $\mathbf{X}$ in Fig. 5 (right) with high error.

Fig. 5

Heatmap of 3 data matrices $\mathbf{X}$ with (left) small, (center) medium and (right) large errors

First of all, we can measure the goodness of fit of the model by $R_{GCI}^2$

Error	$R_{GCI}^2$
Small	0.974
Medium	0.622
Large	0.131

We can see how the smaller the error, the better the values of $R_{GCI}^2$ will be. Thus, we can compute the $R_{SCI_h}^2$ for each of the three SCIs in every situation of error:

Error	$R_{SCI_1}^2$	$R_{SCI_2}^2$	$R_{SCI_3}^2$
Small	0.988	0.988	0.989
Medium	0.778	0.837	0.855
Large	0.624	0.539	0.672

Here, we can see that the values of $R_{SCI_h}^2$ are greater than $R_{GCI}^2$ for each error and for each SCI.

Example 2

Let us consider, now, a different situation where the data matrix $\mathbf{X}$ is divided in three blocks: the first three variables are normal with mean equal to 0 and variance equal to 1, the variables from forth to seventh are normal with mean equal to 3 and variance equal to 1 and finally, the last three variables are normal with mean equal to 7 and variance equal to 1. This matrix $\mathbf{X}$ is generated considering a small error and its heatmap is reported in Fig. 6.

Fig. 6

Heatmap of matrix $\mathbf{X}$ formed by 3 blocks of variables

We can see that if we use the same weight system as before ($\mathbf{c} = \mathbf{1}_3$ and $\mathbf{B}=\mathbf{I}_{10}$) the fit of the model will be relatively bad $R_{GCI}^2=\frac{SS_{mod}}{SS_{tot}} =0.548$, because the arithmetic mean is not a good measure when variables are divided in blocks or are very different each other (i.e., the example with large error). However, we can see that the values of $R_{SCI_h}^2$ are very good

$R_{SCI_1}^2$	$R_{SCI_2}^2$	$R_{SCI_3}^2$
0.999	0.999	1

Therefore, in this situation the researcher should avoid using the GCI and consider the SCIs. The case of the dimensions of well-being: it is more appropriate to stop the analysis at the SCIs’ level such as Material Living Conditions and Quality of Life (Stiglitz et al. 2009).

Example 3

Let us consider the setting given in the Example 1 with the difference that the vector $\mathbf{c}$ has three values not equal to 1. Thus, we have three clusters of MIs with different weights (i.e., importance) for GCI but where into each cluster any MI has the same weight (i.e., equal to 1). Thus, the GCI is a weighted mean of the MIs and all MIs belonging to clusters corresponding to a high value of $\mathbf{c}$ (e.g., higher weight) are more important.

Let us consider $\mathbf{c}=[0.8 0.5 0.6]'$ and try to measure the goodness of fit of the model by $R_{GCI}^2$ and $R_{SCI_h}^2$ for all levels of error

Error	Small	Medium	Large
$R_{GCI}^2$	0.934	0.577	0.269
$R_{SCI_1}^2$	0.984	0.848	0.679
$R_{SCI_2}^2$	0.954	0.754	0.610
$R_{SCI_3}^2$	0.972	0.832	0.739

We can see that the values of $R_{SCI_h}^2$ are greater than $R_{GCI}^2$ for each error and for each SCIs also in this situation, that’s why the generated matrix of data is divided in three blocks and we have already seen arithmetic mean is not a good estimator when data are divided in different blocks of variables. Let us now consider $\mathbf{B}$ as a diagonal matrix (not equal to the identity matrix). Here, each MI has a different weight also into every single cluster. Thus, for example: $\mathbf{B}= {\text{dg}}([0.9 0.7 0.8 0.8 0.6 0.9 0.7 0.7 0.6 0.8]')$ and $\mathbf{c}=[0.7 0.6 0.5]'$; and let us try to measure the goodness of fit of the model by $R_{GCI}^2$ and $R_{SCI_h}^2$ for all levels of error

Error	Small	Medium	Large
$R_{GCI}^2$	0.909	0.431	0.196
$R_{SCI_1}^2$	0.968	0.699	0.599
$R_{SCI_2}^2$	0.936	0.722	0.554
$R_{SCI_3}^2$	0.934	0.554	0.513

Results highlights the same situation previously studied: the arithmetic mean is a good GCI only when the MIs are similar.

Example 4

Let us generate a sample of 5000 vectors (of 1000 units) from a standard normal distribution and let us compute the empirical distribution of variance for each type of normalizations presented previously. The average of the sample variance for each type of normalization is reported:

Raw data	Standardized	Min–max	Norm dispersion
1.04	1	0.02	24,229.95

It is useful to recall that for standardized data variance is constantly equal to one for all MIs thus keeping under control the effect of the variability while maintaining the data centered; that is why this is the most used method of normalization. The Min–max method keeps constant the range of values of the data, however, the value of the variance is reduced toward zero. This might be considered a problem because different MIs tend to be compressed reducing differences. The variance of data with Normalized dispersion method depends on the value of the mean of the raw data (i.e., they are inversely proportional), so if the mean is close to zero the value of the variance is very big (e.g., in the last example) otherwise the variance decreases consistently.

Let us generate a sample of 5000 vectors (of 1000 units) from a normal distribution with mean equal to 10 and variance equal to 1 and let us compute the empirical distribution of variance for each type of normalization previously presented. The average of the sample variance for each type of normalization is reported:

Raw data	Standardized	Min–max	Norm dispersion
0.95	1	0.03	0.01

Fig. 7

Comparison among empirical distribution of variances

In Fig. 7, it is possible to observe the difference among empirical distribution of variances according to different types of normalization via histograms.

Example 5

Suppose $a=0.8$ and $b=0.3$. In order to guarantee a valid correlation matrix, c must be into the range: $[-0.332364, 0.812364]$. In fact, for different values of c the eigenvalues are:

$c=0.9$	$c=0.8$	$c=-0.2$	$c=-0.4$
−  0.0651	0.0087	0.0658	−  0.0369
0.7024	0.7000	1.1274	1.2293
2.3627	2.2913	1.8069	1.8076

Hence, values of c chosen outside of the given range, e.g., $c =0.4$ or $c = 0.9$, give a matrix $\mathbf{A}$ that is not positive semi-definite, since it has at least one negative eigenvalue.

Example 6

Given a ($100 \times 3$) raw data matrix $\mathbf{X}$ with correlation matrix $\mathbf{R}$:

$$\begin{aligned} \mathbf{R} = \begin{bmatrix} 1 &{} 0.45 &{} -0.5\\ 0.45 &{} 1 &{} -0.8\\ -0.5 &{} -0.8 &{} 1\\ \end{bmatrix} \end{aligned}$$

After a normalization of $\mathbf{X}$, we can edit the third MI with the formula $\tilde{x}_{ij}=1 - x_{ij}$. In this way, we would obtain a new correlation matrix with all positive values:

$$\begin{aligned} \tilde{\mathbf{R}} = \begin{bmatrix} 1 &{} 0.45 &{} 0.5\\ 0.45 &{} 1 &{} 0.8\\ 0.5 &{} 0.8 &{} 1\\ \end{bmatrix} \end{aligned}$$

It is important to notice how this operation does not change the absolute value of correlations (i.e., intensity, level or magnitude).

Example 7

Let us suppose that data have the hierarchical structure presented in Fig. 8.

Fig. 8

Path diagram of two-level GCI with three SCIs

If we try to estimate this situation with a model with only 2 factors, it will be evident how two aspects are considered together into one single factor:

	Factor 1	Factor 2
Unidimensionality	2.737	0.556
Reliability	0.526	0.794

The Factor 1 explains the first two clusters of MIs, and its measure of unidimensionality is higher than 1. We can also see how its Cronbach’s $\alpha$ is not acceptable.

If we try to consider one more factor, the measures change essentially:

	Factor 1	Factor 2	Factor 3
Unidimensionality	0.400	0.618	0.556
Reliability	0.781	0.781	0.794

Here the values of Cronbach’s $\alpha$ are all good and the unidimensionality is verified per each factor (i.e., the values of the variance of the second component of the cluster are lower than 1).