# Factor Analysis PROOF

#### spunky

##### Doesn't actually exist
This question pertains a small result in Factor Analysis, under what is called the Orthogonal Factor Analytic Model, where the decomposition of the covariance matrix $$\bf{\Sigma}$$ is expressed as $$\bf{\Sigma}=\bf{\Lambda\Phi\Lambda '+\Psi}$$. In this case, since it's Orthogonal Factor Analysis, it is assumed that $$\bf{\Phi}=\bf{I}$$ and we will be using the sample analogues (so $$\bf{S}$$ for the sample covariance matrix, $$\bf{L}$$ for the matrix of loadings in a Principal Components Analysis sense (i.e. eigenvectors, etc.)

Anyway, here we go!

Let $$\bf{S}$$ be a covariance matrix with eigenvalue-eigenvector pairs ($$\lambda_1, \mathbf{e}_1$$), ($$\lambda_2, \mathbf{e}_2$$), ..., ($$\lambda_p, \mathbf{e}_p$$), where
$$\lambda_1 \ge \lambda_2 \ge ... \ge \lambda_p$$. Let $$m<p$$ and define:

$$\bf{L} = \{l_{ij}\} = \left[\sqrt{\lambda_1 }\mathbf{e}_1\ |\ \sqrt{\lambda_2} \mathbf{e}_2\ |\ ...\ |\ \sqrt{\lambda_m} \mathbf{e}_m \right]$$

and:

$$\ \mathbf\Psi = \left( \begin{array}{cccc} \psi_1 & 0 & ... & 0 \\ 0 & \psi_2 & ... & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & ... & \psi_p \\ \end{array} \right) \text{ with } \psi_i = s_{ii} - \sum_{j=1}^{m} l_{ij}^2$$

Then, PROVE:

$$\text{Sum of squared entries of } (\mathbf{S} - (\mathbf{LL'} + \mathbf{\Psi})) \le \lambda_{m+1}^2 + \cdots + \lambda_p^2$$

Spunky's attempt of a proof:

By definition of $$\psi_i$$, we know that the diagonal of $$(\mathbf{S} - (\mathbf{LL'} + \mathbf{\Psi}))$$ is all zeroes. Since
$$(\mathbf{S} - (\mathbf{LL'} + \mathbf{\Psi})))$$ and $$(\mathbf{S} - \mathbf{LL'})$$ have the same elements except on the diagonal, we know that

$$\text{(Sum of squared entries of } (\mathbf{S} - (\mathbf{LL'} + \mathbf{\Psi}))) \leq \text{ Sum of squared entries of } (\mathbf{S} - \mathbf{LL'})$$

Since $$\mathbf{S} = \lambda_1 \mathbf{e}_1 \mathbf{e}'_1 + \cdots + \lambda_p \mathbf{e}_p \mathbf{e}'_p$$
and $$\mathbf{LL'} = \lambda_1 \mathbf{e}_1 \mathbf{e}'_1 + \cdots + \lambda_m \mathbf{e}_m \mathbf{e}'_m$$, then it follows that
$$\mathbf{S} - \mathbf{LL'} = \lambda_{m+1} \mathbf{e}_{m+1} \mathbf{e}'_{m+1} + \cdots + \lambda_p \mathbf{e}_p \mathbf{e}'_p$$

Writing it in matrix form, this is saying $$\mathbf{S} - \mathbf{LL'} = \mathbf{P}_2 \mathbf{\Lambda}_2 \mathbf{P}'_2$$ where
$$\mathbf{P}_2 = [ \mathbf{e}_{m+1} | \cdots | \mathbf{e}_p ]$$ and $$\mathbf{\Lambda}_2 = Diag(\lambda_{m+1}, \cdots, \lambda_{p})$$

Then, the following is true:

$$\text{Sum of squared entries of }(\mathbf{S}- \mathbf{LL'})= \text{tr}((\mathbf{S} - \mathbf{LL'}) (\mathbf{S} - \mathbf{LL'})')=$$

$$\text{tr} (( \mathbf{P}_2 \mathbf{\Lambda}_2 \mathbf{P}'_2)( \mathbf{P}_2 \mathbf{\Lambda}_2 \mathbf{P}'_2)')=\text{tr}( \mathbf{P}_2 \mathbf{\Lambda}_2\mathbf{\Lambda}_2 \mathbf{P}'_2)$$

$$tr(\mathbf{\Lambda}_2\mathbf{\Lambda}_2)=\lambda_{m+1}^2 + \cdots + \lambda_p^2.$$

All the $$\bf{P}_2$$ disappear because by the definition of $$\bf{P}_2$$ we know that $$\bf{P}_2 '\bf{P}_2=\bf{I}$$

does this make sense? anything wrong someone may spot?

#### Dason

Looks good to me. A few of the steps could be slightly more explicit but I can't find anything glaringly wrong with it.

#### spunky

##### Doesn't actually exist
Thank you! Thank you! Thank you! Thank you! Thank you! Thank you! Thank you! Thank you! Thank you!