DSC 140B
Problems tagged with variance

Problems tagged with "variance"

Problem #067

Tags: quiz-04, covariance, variance, lecture-06

Suppose \(C = \begin{pmatrix} 5 & -3\\ -3 & 6 \\ \end{pmatrix}\) is the empirical covariance matrix for a centered data set. What is the variance in the direction given by the unit vector \(\vec{u} = \frac{1}{\sqrt2}(-1, 1)^T\)?

Solution

\(17/2\).

The variance in the direction of a unit vector \(\vec{u}\) is given by \(\vec{u}^T C \vec{u}\).

$$\begin{align*}\vec{u}^T C \vec{u}&= \frac{1}{\sqrt{2}}(-1, 1)^T \begin{pmatrix} 5 & -3 \\ -3 & 6 \end{pmatrix}\frac{1}{\sqrt{2}}\begin{pmatrix} -1 \\ 1 \end{pmatrix}\\&= \frac{1}{2}(-1, 1)^T \begin{pmatrix} -5 - 3 \\ 3 + 6 \end{pmatrix}\\&= \frac{1}{2}(-1, 1)^T \begin{pmatrix} -8 \\ 9 \end{pmatrix}\\&= \frac{1}{2}(8 + 9) \\&= \frac{17}{2}\end{align*}$$

Problem #068

Tags: covariance, eigenvectors, quiz-04, variance, lecture-06

Let \(C\) be the sample covariance matrix of a centered data set, and suppose \(\vec{u}^{(1)}, \vec{u}^{(2)}, \vec{u}^{(3)}\) are normalized eigenvectors of \(C\) with eigenvalues \(\lambda_1 = 9, \lambda_2 = 3, \lambda_3=0\), respectively.

Suppose \(\vec x = \frac{1}{\sqrt 2}\vec{u}^{(1)} + \frac{1}{\sqrt 6}\vec{u}^{(2)} + \frac{1}{\sqrt 3}\vec{u}^{(3)}\). What is the variance in the direction of \(\vec x\)?

Solution

\(5\) Remember that for any unit vector \(\vec u\), the variance in the direction of \(\vec u\) is given by \(\vec u^T C \vec u\). So, for the vector \(\vec x\), we have

$$\begin{align*}\operatorname{Var}(\vec x) &= \vec x^T C \vec x \\&= \left(\frac{1}{\sqrt 2}\vec{u}^{(1)} + \frac{1}{\sqrt 6}\vec{u}^{(2)} + \frac{1}{\sqrt 3}\vec{u}^{(3)}\right)^T C \left(\frac{1}{\sqrt 2}\vec{u}^{(1)} + \frac{1}{\sqrt 6}\vec{u}^{(2)} + \frac{1}{\sqrt 3}\vec{u}^{(3)}\right)\end{align*}$$

Since \(\vec{u}^{(1)}, \vec{u}^{(2)}, \vec{u}^{(3)}\) are eigenvectors of \(C\), the matrix multiplication simplifies:

$$\begin{align*}\operatorname{Var}(\vec x) &= \left(\frac{1}{\sqrt 2}\vec{u}^{(1)} + \frac{1}{\sqrt 6}\vec{u}^{(2)} + \frac{1}{\sqrt 3}\vec{u}^{(3)}\right)^T \left(\frac{9}{\sqrt 2}\vec{u}^{(1)} + \frac{3}{\sqrt 6}\vec{u}^{(2)} + \frac{0}{\sqrt 3}\vec{u}^{(3)}\right)\\&= \frac{1}{2}\cdot 9 + \frac{1}{6}\cdot 3 + \frac{1}{3}\cdot 0 \\&= \frac{9}{2} + \frac{1}{2} + 0 \\&= 5 \end{align*}$$

Problem #072

Tags: quiz-04, covariance, variance, lecture-06

Suppose \(C = \begin{pmatrix} 4 & -2\\ -2 & 5 \\ \end{pmatrix}\) is the empirical covariance matrix for a centered data set. What is the variance in the direction given by the unit vector \(\vec{u} = \frac{1}{\sqrt2}(1, 1)^T\)?

Solution

\(5/2\) The variance in the direction of a unit vector \(\vec{u}\) is given by \(\vec{u}^T C \vec{u}\).

$$\begin{align*}\vec{u}^T C \vec{u}&= \frac{1}{\sqrt{2}}(1, 1) \begin{pmatrix} 4 & -2 \\ -2 & 5 \end{pmatrix}\frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ 1 \end{pmatrix}\\&= \frac{1}{2}(1, 1) \begin{pmatrix} 4 - 2 \\ -2 + 5 \end{pmatrix}\\&= \frac{1}{2}(1, 1) \begin{pmatrix} 2 \\ 3 \end{pmatrix}\\&= \frac{1}{2}(2 + 3) \\&= \frac{5}{2}\end{align*}$$

Problem #073

Tags: covariance, eigenvectors, quiz-04, variance, lecture-06

Let \(C\) be the sample covariance matrix of a centered data set, and suppose \(\vec{u}^{(1)}, \vec{u}^{(2)}, \vec{u}^{(3)}\) are normalized eigenvectors of \(C\) with eigenvalues \(\lambda_1 = 16, \lambda_2 = 12, \lambda_3=0\), respectively.

Suppose \(\vec x = \frac{1}{\sqrt 2}\vec{u}^{(1)} + \frac{1}{\sqrt 6}\vec{u}^{(2)} + \frac{1}{\sqrt 3}\vec{u}^{(3)}\). What is the variance in the direction of \(\vec x\)?

Solution

\(10\) Remember that for any unit vector \(\vec u\), the variance in the direction of \(\vec u\) is given by \(\vec u^T C \vec u\). So, for the vector \(\vec x\), we have

$$\begin{align*}\operatorname{Var}(\vec x) &= \vec x^T C \vec x \\&= \left(\frac{1}{\sqrt 2}\vec{u}^{(1)} + \frac{1}{\sqrt 6}\vec{u}^{(2)} + \frac{1}{\sqrt 3}\vec{u}^{(3)}\right)^T C \left(\frac{1}{\sqrt 2}\vec{u}^{(1)} + \frac{1}{\sqrt 6}\vec{u}^{(2)} + \frac{1}{\sqrt 3}\vec{u}^{(3)}\right)\end{align*}$$

Since \(\vec{u}^{(1)}, \vec{u}^{(2)}, \vec{u}^{(3)}\) are eigenvectors of \(C\), the matrix multiplication simplifies:

$$\begin{align*}\operatorname{Var}(\vec x) &= \left(\frac{1}{\sqrt 2}\vec{u}^{(1)} + \frac{1}{\sqrt 6}\vec{u}^{(2)} + \frac{1}{\sqrt 3}\vec{u}^{(3)}\right)^T \left(\frac{16}{\sqrt 2}\vec{u}^{(1)} + \frac{12}{\sqrt 6}\vec{u}^{(2)} + \frac{0}{\sqrt 3}\vec{u}^{(3)}\right)\\&= \frac{1}{2}\cdot 16 + \frac{1}{6}\cdot 12 + \frac{1}{3}\cdot 0 \\&= 8 + 2 + 0 \\&= 10 \end{align*}$$

Problem #088

Tags: lecture-07, eigenvalues, variance, pca

Let \(\mathcal X = \{\vec{x}^{(1)}, \ldots, \vec{x}^{(100)}\}\) be a set of 100 points in \(3\) dimensions. The variance in the direction of \(\hat{e}^{(1)}\) is 20, the variance in the direction of \(\hat{e}^{(2)}\) is 12, and the variance in the direction of \(\hat{e}^{(3)}\) is 10.

Suppose PCA is performed, but dimensionality is not reduced; that is, each point \(\vec{x}^{(i)}\) is transformed to the vector \(\vec{z}^{(i)} = U \vec{x}^{(i)}\), where \(U\) is a matrix whose \(3\) rows are the (orthonormal) eigenvectors of the data covariance matrix. Let \(\mathcal Z = \{\vec{z}^{(1)}, \ldots, \vec{z}^{(100)}\}\) be the resulting data set.

Suppose \(C\) is the covariance matrix of the new data set, \(\mathcal Z\), and that the top two eigenvalues of \(C\) are 25 and 15. What is the third eigenvalue?

Solution

\(2\).

Although it wasn't discussed in lecture, it turns out that the total variance is preserved under orthogonal transformations. The total variance of the original data is:

\[ 20 + 12 + 10 = 42 \]

After PCA, the covariance matrix \(C\) of \(\mathcal Z\) is diagonal with the eigenvalues on the diagonal. The sum of eigenvalues equals the total variance:

\[\lambda_1 + \lambda_2 + \lambda_3 = 42 \]
\[ 25 + 15 + \lambda_3 = 42 \]
\[\lambda_3 = 42 - 40 = 2 \]