# Exploratory factor analysis

December 18, 2019

## Factor analysis

Factor analysis can be performed to combine a large number of variables to smaller number of factors. This is done usually for the following reasons:
1. Find interrelationships among different kinds of variables
2. Identify common underlying dimension
3. Data reduction and removing duplicacy of columns

Among the many types of ways one can do factor analysis, two ways are popular. They are
1. Principle component analysis
2. Common factor analysis

PCA considers the total variance in the data while CFA only considers the common variance. In this blog, we are going to discuss about Principal component analysis.

## Principal component analysis

PCA is the most widely used exploratory factor analysis technique, It is developed by Pearson and Hotelling. The objective of PCA is to rigidly rotate the axes of p-dimensional space to new positions (principal axes) that have the following properties:
1. Ordered such that principal axis 1 has the highest variance, axis 2 has the next highest variance, …. , and axis p has the lowest variance
2. Covariance among each pair of the principal axes is zero (the principal axes are uncorrelated)

## Data and problem

This dataset contains 90 responses for 14 different variables that customers consider while purchasing car. The survey questions were framed using 5-point likert scale with 1 being very low and 5 being very high. The data can be downloaded from this link. The variables were the following:
1. Price
2. Safety
3. Exterior looks
4. Space and comfort
5. Technology
6. After sales service
7. Resale value
8. Fuel type
9. Fuel efficiency
10. Color
11. Maintenance
12. Test drive
13. Product reviews
14. Testimonials
A sample of the data is shown:
Car survey data
Price Safety Exterior_Looks Space_comfort Technology After_Sales_Service Resale_Value Fuel_Type Fuel_Efficiency Color Maintenance Test_drive Product_reviews Testimonials
4 4 3 4 5 5 5 4 5 4 5 4 4 3
4 3 4 3 4 4 5 4 4 4 5 5 5 4
5 4 3 4 3 5 5 5 5 5 4 2 5 3
4 5 2 4 5 4 3 4 4 4 4 4 4 4
4 4 5 4 3 4 3 4 4 5 4 4 4 5

### KMO Index

The Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy is an index used to examine the appropriateness of factor analysis. High values (between 0.5 and 1.0) indicate factor analysis is appropriate. Values below 0.5 imply that factor analysis may not be appropriate

## Kaiser-Meyer-Olkin factor adequacy
## Call: KMO(r = efa)
## Overall MSA =  0.61
## MSA for each item =
##               Price              Safety      Exterior_Looks
##                0.72                0.47                0.55
##       Space_comfort          Technology After_Sales_Service
##                0.61                0.65                0.62
##        Resale_Value           Fuel_Type     Fuel_Efficiency
##                0.63                0.68                0.62
##               Color         Maintenance          Test_drive
##                0.56                0.61                0.64
##     Product_reviews        Testimonials
##                0.69                0.50

As some variables have KMO close to and less than 5, more data should be captured for the variables for doing factor analysis. As this is just an example, we will continue to use the same data for analysis.

### Bartlett’s test of sphericity

Bartlett’s test of sphericity is a test statistic used to examine the hypothesis that the variables are uncorrelated in the population. In other words, the population correlation matrix is an identity matrix; each variable correlates perfectly with itself (r = 1) but has no correlation with the other variables (r = 0).
$H_0$: All non diagonal values of correlation matrix are zero
$H_1$: Not all diagonal values of correlation matrix are zero

##
##  Bartlett's test of sphericity
##
## data:  efa
## X-squared = 247.71, df = 91, p-value < 2.2e-16 From the above correlation matrix and the test we can observe that there is some dependence between the variables and factor analysis can therefore be performed.

In this example, the factors can be considered as the underlying thought process while each variable can be considered as the response to the question. While looking for comfort in a car, a person might look into aesthetics, functionality, economic value and credibility. These are factors while the survey questions are variables.

While replying to a question in a survey, for every variable, the respondent underlying thought process gives weightage to each factor as a function of that variable. This can be written as (for normalized variables):
$y_1 = \lambda_{11} f_1 + \lambda_{12} f_2 + \cdots + \lambda_{1m} f_m + \epsilon_1$ $y_2 = \lambda_{21} f_1 + \lambda_{22} f_2 + \cdots + \lambda_{2m} f_m + \epsilon_2$ $\vdots$ $y_p = \lambda_{p1} f_1 + \lambda_{p2} f_2 + \cdots + \lambda_{pm} f_m + \epsilon_p$ Where $f_1, f_2 \cdots$ are the factors and $y_1, y_2 \cdots$ are variables. $\lambda_{pm}$ are called factor loadings, or the correlation between variables and factors.

### Number of factors

The number of factors to decompose the dataset should be selected. There are multiple ways of doing it, the most popular ones are:
1. Number of eigenvalues greater than 1
2. Scree plot
3, Percentage of variation explained
Let us look at each one of them:

#### Eigen values

The eigenvalue represents the total variance explained by each factor. If each variable is normalized before the analysis, the maximum eigen value of all the factors combined should be equal to the number of variables (As normalized variables have variation as 1)

The eigen values(from the co-variance matrix) for this data set is:

##   2.1585694 1.6836760 1.0709449 0.9760982 0.7723481 0.6364842 0.5211956
##   0.4245587 0.3942269 0.3600300 0.2747809 0.2306904 0.1774646 0.1588822

Any factor which has eigen value less than 1 explains the variation less than the variation explained by a variable. So one way to identify the number of factors is the number of eigenvalues greater than 1. From the eigen values, the number of factors to consider is 3.

#### Scree plot

A scree plot is a line plot of the eigenvalues of factors or principal components in an analysis. A scree plot always displays the eigenvalues in a downward curve, ordering the eigenvalues from largest to smallest. According to the scree test, the “elbow” of the graph where the eigenvalues seem to level off is found and factors or components to the left of this point should be retained as significant. It is named after its resemblance to scree(broken rock fragments at the base of cliffs) after its elbow. ## Parallel analysis suggests that the number of factors =  4  and the number of components =  NA

From the scree plot, a significant slope change can be observed after the third or fourth factor. The number of factors to consider from scree plot is 3.

After identifying the number of factors, the next step in PCA is to create the factors without rotation. This is done in such a way to satisfy:
1. Principal axis-1 has the highest variance, axis-2 has the next highest variance, …. , and axis p has the lowest variance
2. Co-variance among each pair of the principal axes is zero (the principal axes are uncorrelated)

##
## Call:
## factanal(x = ~., factors = 3, data = efa, na.action = na.exclude,     rotation = "none", cutoff = 0.3)
##
## Uniquenesses:
##               Price              Safety      Exterior_Looks
##               0.721               0.896               0.849
##       Space_comfort          Technology After_Sales_Service
##               0.277               0.870               0.758
##        Resale_Value           Fuel_Type     Fuel_Efficiency
##               0.306               0.713               0.592
##               Color         Maintenance          Test_drive
##               0.258               0.567               0.904
##     Product_reviews        Testimonials
##               0.782               0.854
##
##                     Factor1 Factor2 Factor3
## Price                0.353   0.199   0.339
## Safety              -0.233   0.200
## Exterior_Looks      -0.281   0.127   0.238
## Space_comfort       -0.235   0.815
## Technology                   0.358
## After_Sales_Service  0.149   0.466
## Resale_Value         0.618  -0.116   0.547
## Fuel_Type                    0.524
## Fuel_Efficiency      0.538   0.341
## Color                0.678          -0.526
## Maintenance          0.608   0.208   0.141
## Test_drive           0.112   0.286
## Product_reviews      0.296   0.361
## Testimonials                 0.208  -0.318
##
##                Factor1 Factor2 Factor3
## Proportion Var   0.139   0.130   0.064
## Cumulative Var   0.139   0.268   0.333
##
## Test of the hypothesis that 3 factors are sufficient.
## The chi square statistic is 65.94 on 52 degrees of freedom.
## The p-value is 0.0925

The uniqueness of each variable is also shown. $Uniqueness=1−Communality$ where Communality is the SS of all the factor loadings for a given variable. If all the factors jointly explain a large percent of variance in a given variable, that variable has high Communality (and thus low uniqueness).

Our goal is to name the factors. Sometimes visualizations help. Plotting the factor loadings for the first two factors. It can be difficult to label factors when they are unrotated, since a description of one factor might overlap with a description of another factor. We can rotate the factors to obtain more straightforward interpretations. Rotations are of various types:
1. Varimax rotation: An orthogonal rotation method that minimizes the number of variables that have high loadings on each factor. This method simplifies the interpretation of the factors
2. Quartimax rotation: A rotation method that minimizes the number of factors needed to explain each variable. This method simplifies the interpretation of the observed variables
3. Equamax rotation: A rotation method that is a combination of the varimax method, which simplifies the factors, and the quartimax method which simplifies the variables
4. Direct Oblimin Method: A method for oblique (non-orthogonal) rotation
5. Promax rotation: An oblique rotation, which allows factors to be correlated ##
##                     MR1    MR2    MR3
## Maintenance          0.668
## Fuel_Efficiency      0.655
## Resale_Value         0.599
## Color                0.464
## Price                0.444
## Product_reviews      0.424
## Space_comfort               0.832
## Fuel_Type                   0.573
## After_Sales_Service         0.460
## Technology                  0.342
## Safety                      0.311
## Exterior_Looks
## Testimonials                       0.742
## Test_drive                         0.328
##
##                  MR1   MR2   MR3
## Proportion Var 0.144 0.115 0.069
## Cumulative Var 0.144 0.259 0.328

In this example, for any rotation, when we consider three factors, one variable is becoming insignificant and is not loading to any factor. In oblimin, the factor Exterior looks is not loading to any factor. If 4 factors were considered, then, ##
##                     MR1    MR2    MR4    MR3
## Resale_Value         0.729
## Maintenance          0.562
## Price                0.544
## Fuel_Efficiency      0.434
## Space_comfort               0.782
## Fuel_Type                   0.575
## After_Sales_Service         0.537
## Safety                      0.358
## Technology                  0.358
## Color                              0.731
## Exterior_Looks                    -0.548
## Testimonials                              0.685
## Test_drive                                0.365
## Product_reviews                           0.364
##
##                  MR1   MR2   MR4   MR3
## Proportion Var 0.117 0.117 0.075 0.069
## Cumulative Var 0.117 0.234 0.309 0.378

This model is single loaded model (simple structure). The factor mapping is as follows: ### Validation

The factors created can be validated by looking at error metrics, or TLI.

## Factor Analysis using method =  minres
## Call: fa(r = efa, nfactors = 4, rotate = "oblimin", fm = "minres")
##                       MR1   MR2   MR4   MR3   h2   u2 com
## Price                0.54  0.12 -0.08 -0.05 0.30 0.70 1.2
## Safety              -0.33  0.36  0.11 -0.25 0.23 0.77 3.0
## Exterior_Looks       0.12  0.07 -0.55  0.28 0.34 0.66 1.6
## Space_comfort       -0.04  0.78 -0.13  0.08 0.67 0.33 1.1
## Technology           0.01  0.36  0.06  0.02 0.13 0.87 1.1
## After_Sales_Service  0.10  0.54  0.18 -0.05 0.33 0.67 1.3
## Resale_Value         0.73 -0.14 -0.02 -0.16 0.57 0.43 1.2
## Fuel_Type            0.01  0.57 -0.01 -0.12 0.32 0.68 1.1
## Fuel_Efficiency      0.43  0.23  0.31  0.16 0.47 0.53 2.7
## Color                0.07 -0.06  0.73  0.13 0.61 0.39 1.1
## Maintenance          0.56  0.08  0.21  0.01 0.44 0.56 1.3
## Test_drive           0.11  0.16  0.01  0.36 0.20 0.80 1.6
## Product_reviews      0.34  0.15  0.05  0.36 0.32 0.68 2.4
## Testimonials        -0.19 -0.01  0.06  0.68 0.51 0.49 1.2
##
##                        MR1  MR2  MR4  MR3
## Proportion Var        0.12 0.12 0.08 0.07
## Cumulative Var        0.12 0.24 0.32 0.39
## Proportion Explained  0.31 0.30 0.20 0.18
## Cumulative Proportion 0.31 0.62 0.82 1.00
##
##  With factor correlations of
##      MR1   MR2   MR4  MR3
## MR1 1.00  0.05  0.27 0.00
## MR2 0.05  1.00 -0.02 0.21
## MR4 0.27 -0.02  1.00 0.12
## MR3 0.00  0.21  0.12 1.00
##
## Mean item complexity =  1.6
## Test of the hypothesis that 4 factors are sufficient.
##
## The degrees of freedom for the null model are  91  and the objective function was  2.97 with Chi Square of  247.71
## The degrees of freedom for the model are 41  and the objective function was  0.57
##
## The root mean square of the residuals (RMSR) is  0.05
## The df corrected root mean square of the residuals is  0.07
##
## The harmonic number of observations is  90 with the empirical chi square  38.46  with prob <  0.58
## The total number of observations was  90  with Likelihood Chi Square =  46.2  with prob <  0.27
##
## Tucker Lewis Index of factoring reliability =  0.922
## RMSEA index =  0.052  and the 90 % confidence intervals are  0 0.085
## BIC =  -138.29
## Fit based upon off diagonal values = 0.94
## Measures of factor score adequacy
##                                                    MR1  MR2  MR4  MR3
## Correlation of (regression) scores with factors   0.87 0.88 0.84 0.80
## Multiple R square of scores with factors          0.76 0.77 0.70 0.64
## Minimum correlation of possible factor scores     0.51 0.55 0.40 0.28

The root mean square of residuals (RMSR) is 0.05 and the RMSEA (root mean square error of approximation) index is 0.052. These are acceptable as these values should be closer to 0.
The Tucker-Lewis Index (TLI) is 0.93 – an acceptable value considering it’s over 0.9.
The correlation between the newly created factors is small

### Interpreting the Factors

After establishing the adequacy of the factors, it’s time for us to interpret the factors. This is the theoretical side of the analysis where we form the factors depending on the variable loadings. In this case, here is how the factors can be created:

### Factor 1 - Economic value:

Factor 1 contains resale value, maintenance, fuel efficiency and price. It is describing the Economic value of the car.

### Factor 2 - Functional benefits:

Factor 2 contains Space_comfort, Fuel_Type, After_Sales_Service, Safety and Technology. It is describing the functional benefits of the car

### Factor 3- Aesthetics

Factor 3 contains color and exterior looks. This factor is describing the Aesthetics of the car

### Factor 4 - Credibility

Factor 4 contains Test drive, product reviews and testimonials. It is describing the credibility of the car