Factor Analysis PROOF

spunky

Doesn't actually exist
#1
This question pertains a small result in Factor Analysis, under what is called the Orthogonal Factor Analytic Model, where the decomposition of the covariance matrix \(\bf{\Sigma}\) is expressed as \(\bf{\Sigma}=\bf{\Lambda\Phi\Lambda '+\Psi}\). In this case, since it's Orthogonal Factor Analysis, it is assumed that \(\bf{\Phi}=\bf{I}\) and we will be using the sample analogues (so \( \bf{S}\) for the sample covariance matrix, \(\bf{L}\) for the matrix of loadings in a Principal Components Analysis sense (i.e. eigenvectors, etc.)

Anyway, here we go!


Let \(\bf{S}\) be a covariance matrix with eigenvalue-eigenvector pairs (\(\lambda_1, \mathbf{e}_1\)), (\(\lambda_2, \mathbf{e}_2\)), ..., (\(\lambda_p, \mathbf{e}_p\)), where
\(\lambda_1 \ge \lambda_2 \ge ... \ge \lambda_p\). Let \(m<p\) and define:

\(\bf{L} = \{l_{ij}\} = \left[\sqrt{\lambda_1 }\mathbf{e}_1\ |\ \sqrt{\lambda_2} \mathbf{e}_2\ |\ ...\ |\ \sqrt{\lambda_m} \mathbf{e}_m \right] \)

and:

\(\
\mathbf\Psi =
\left(
\begin{array}{cccc}
\psi_1 & 0 & ... & 0 \\
0 & \psi_2 & ... & 0 \\
\vdots & \vdots & \ddots & \vdots \\
0 & 0 & ... & \psi_p \\
\end{array}
\right)
\text{ with } \psi_i = s_{ii} - \sum_{j=1}^{m} l_{ij}^2\)

Then, PROVE:

\(
\text{Sum of squared entries of } (\mathbf{S} - (\mathbf{LL'} + \mathbf{\Psi})) \le \lambda_{m+1}^2 + \cdots + \lambda_p^2\)

Spunky's attempt of a proof:

By definition of \(\psi_i\), we know that the diagonal of \((\mathbf{S} - (\mathbf{LL'} + \mathbf{\Psi}))\) is all zeroes. Since
\((\mathbf{S} - (\mathbf{LL'} + \mathbf{\Psi})))\) and \((\mathbf{S} - \mathbf{LL'})\) have the same elements except on the diagonal, we know that

\(\text{(Sum of squared entries of } (\mathbf{S} - (\mathbf{LL'} + \mathbf{\Psi}))) \leq \text{ Sum of squared entries of } (\mathbf{S} - \mathbf{LL'}) \)

Since \(\mathbf{S} = \lambda_1 \mathbf{e}_1 \mathbf{e}'_1 + \cdots + \lambda_p \mathbf{e}_p \mathbf{e}'_p \)
and \(\mathbf{LL'} = \lambda_1 \mathbf{e}_1 \mathbf{e}'_1 + \cdots + \lambda_m \mathbf{e}_m \mathbf{e}'_m \), then it follows that
\(\mathbf{S} - \mathbf{LL'} = \lambda_{m+1} \mathbf{e}_{m+1} \mathbf{e}'_{m+1} + \cdots + \lambda_p \mathbf{e}_p \mathbf{e}'_p\)

Writing it in matrix form, this is saying \(\mathbf{S} - \mathbf{LL'} = \mathbf{P}_2 \mathbf{\Lambda}_2 \mathbf{P}'_2\) where
\(\mathbf{P}_2 = [ \mathbf{e}_{m+1} | \cdots | \mathbf{e}_p ]\) and \(\mathbf{\Lambda}_2 = Diag(\lambda_{m+1}, \cdots, \lambda_{p})\)

Then, the following is true:

\(\text{Sum of squared entries of }(\mathbf{S}- \mathbf{LL'})= \text{tr}((\mathbf{S} - \mathbf{LL'}) (\mathbf{S} - \mathbf{LL'})')=\)

\(\text{tr} (( \mathbf{P}_2 \mathbf{\Lambda}_2 \mathbf{P}'_2)( \mathbf{P}_2 \mathbf{\Lambda}_2 \mathbf{P}'_2)')=\text{tr}( \mathbf{P}_2 \mathbf{\Lambda}_2\mathbf{\Lambda}_2 \mathbf{P}'_2)\)

\(tr(\mathbf{\Lambda}_2\mathbf{\Lambda}_2)=\lambda_{m+1}^2 + \cdots + \lambda_p^2.\)

All the \(\bf{P}_2\) disappear because by the definition of \(\bf{P}_2\) we know that \(\bf{P}_2 '\bf{P}_2=\bf{I}\)


does this make sense? anything wrong someone may spot?
 

Dason

Ambassador to the humans
#2
Looks good to me. A few of the steps could be slightly more explicit but I can't find anything glaringly wrong with it.