Practice problems for topics on Quiz 08.
Tags in this problem set:
Tags: lecture-15, convolutional neural networks
A grayscale image of size \(32 \times 32 \times 1\) is convolved with a filter of size \(5 \times 5\). No padding is applied, and the stride is 1. What is the shape of the output response map?
\(28 \times 28 \times 1\).
With no padding and stride 1, the output height and width are each \(32 - 5 + 1 = 28\). The filter slides over each \(5 \times 5\) block of the image from left to right and top to bottom, producing one output value per position.
Tags: lecture-15, convolutional neural networks
An input \(5 \times 5\) grayscale image \((I)\) is represented by the matrix below.
Suppose you convolve \(I\) with the \(3 \times 3\) filter
to get the response map \(I'\)(with stride 1 and no padding). What is the value of \(I'_{11}\) -- the entry in the 1st row and 1st column of the output?
\(I'_{11} = 0.6\).
The \(3 \times 3\) patch at the top-left corner of \(I\) is:
Applying the filter element-wise and summing:
Tags: lecture-15, convolutional neural networks
An input image has shape \(80 \times 80 \times 11\), where \(11\) is the number of channels. We wish to convolve this image with a 3D filter of shape \(5 \times 5 \times k\). What must the value of \(k\) be for the convolution to work?
\(k = 11\).
A 3D convolution filter must have the same number of channels as the input. The filter slides spatially across the height and width of the image, but at each position it computes a dot product across all channels. Therefore the filter's third dimension must match the input's channel count.
Tags: lecture-15, convolutional neural networks
Consider a convolutional neural network with the following architecture. The input is a \(10 \times 10 \times 1\) grayscale image. It passes through Conv layer 1 (3 filters of size \(3 \times 3\), stride 1, no padding), producing an output of shape \(8 \times 8 \times 3\). Then \(2 \times 2\) max pooling is applied, producing an output of shape \(4 \times 4 \times 3\). Next is Conv layer 2 (5 filters of size \(3 \times 3 \times 3\), stride 1, no padding), producing an output of shape \(2 \times 2 \times 5\). This is flattened and fed into a fully connected layer with \(n\) nodes, followed by an output layer with 1 node.
What is the value of \(n\)?
\(n = 20\).
The output of Conv layer 2 is \(2 \times 2 \times 5\). Flattening this gives \(2 \times 2 \times 5 = 20\) values, so the fully connected layer has \(20\) nodes.
What is the total number of learnable parameters in the network, excluding biases?
\(182\).
Conv layer 1 has 3 filters of shape \(3 \times 3\), each with \(9\) weights, for \(3 \times 9 = 27\) parameters. Conv layer 2 has 5 filters of shape \(3 \times 3 \times 3\), each with \(27\) weights, for \(5 \times 27 = 135\) parameters. The fully connected layer connects to the output: \(20 \times 1 = 20\) parameters. The grand total is \(27 + 135 + 20 = 182\).
Note that max pooling has no learnable parameters.
Tags: lecture-15, convolutional neural networks
An input \(4 \times 4\) grayscale image \((I)\) is represented by the matrix below.
\(2 \times 2\) max pooling is applied to this image. What is the resulting output?
\(\begin{pmatrix} 0.7 & 0.8 \\ 0.8 & 0.9 \end{pmatrix}\).
With \(2 \times 2\) max pooling, we divide the \(4 \times 4\) image into four non-overlapping \(2 \times 2\) blocks and take the maximum of each. Top-left: \(\max\{0.7, 0.2, 0.3, 0.5\} = 0.7\). Top-right: \(\max\{0.1, 0.8, 0.4, 0.2\} = 0.8\). Bottom-left: \(\max\{0.6, 0.1, 0.2, 0.8\} = 0.8\). Bottom-right: \(\max\{0.9, 0.3, 0.5, 0.6\} = 0.9\).
Tags: multiple outputs, lecture-16, softmax
A neural network with 3 output nodes uses the softmax activation function. The pre-activation values (logits) at the output layer are \(\vec z = (0, 2, 0)\). Compute the softmax output \(\vec h = (h_1, h_2, h_3)\).
Leave your answer in terms of \(e\).
By the softmax formula, \(h_k = \frac{e^{z_k}}{\sum_{j=1}^{3} e^{z_j}}\). The denominator is:
Therefore:
Tags: multiple outputs, lecture-16, softmax
A neural network with 4 output nodes uses the softmax activation function. The pre-activation values (logits) at the output layer are \(\vec z = (1, 3, 1, 3)\). Compute the softmax output \(\vec h = (h_1, h_2, h_3, h_4)\).
Leave your answer in terms of \(e\).
By the softmax formula, \(h_k = \frac{e^{z_k}}{\sum_{j=1}^{4} e^{z_j}}\). The denominator is:
Therefore:
Tags: multiple outputs, lecture-16, softmax
A neural network with 3 output nodes uses the softmax activation function. The pre-activation values (logits) at the output layer are \(\vec z = (1, 2, 3)\). Compute the softmax output \(\vec h = (h_1, h_2, h_3)\).
Leave your answer in terms of \(e\).
By the softmax formula, \(h_k = \frac{e^{z_k}}{\sum_{j=1}^{3} e^{z_j}}\). The denominator is:
Therefore:
Tags: binary cross-entropy, lecture-16, multiple outputs
A multi-label classifier has 3 output nodes with sigmoid activations. The true labels are \(\vec y = (1, 0, 1)\) and the predicted probabilities are \(\vec h = (0.9, 0.2, 0.8)\).
Compute the binary cross-entropy loss. Leave your answer in terms of \(\log\).
\(-\log(0.9) - \log(0.8) - \log(0.8) = -\log(0.9) - 2\log(0.8)\).
By the binary cross-entropy formula:
Evaluating each term:
The total is \(-\log(0.9) - 2\log(0.8)\).
Tags: binary cross-entropy, lecture-16, multiple outputs
A multi-label classifier has 4 output nodes with sigmoid activations. The true labels are \(\vec y = (0, 1, 0, 1)\) and the predicted probabilities are \(\vec h = (0.3, 0.7, 0.1, 0.9)\).
Compute the binary cross-entropy loss. Leave your answer in terms of \(\log\).
\(-2\log(0.7) - 2\log(0.9)\).
By the binary cross-entropy formula:
Evaluating each term:
The total is \(-2\log(0.7) - 2\log(0.9)\).
Tags: binary cross-entropy, lecture-16, multiple outputs
A multi-label classifier has 3 output nodes with sigmoid activations. The true labels are \(\vec y = (1, 1, 0)\) and the predicted probabilities are \(\vec h = (0.8, 0.6, 0.4)\).
Compute the binary cross-entropy loss. Leave your answer in terms of \(\log\).
\(-\log(0.8) - 2\log(0.6)\).
By the binary cross-entropy formula:
Evaluating each term:
The total is \(-\log(0.8) - 2\log(0.6)\).
Tags: multiple outputs, lecture-16, categorical cross-entropy
A multi-class classifier has 4 output nodes with softmax activation. The true label is \(\vec y = (0, 0, 1, 0)\) and the softmax outputs are \(\vec h = (0.1, 0.2, 0.6, 0.1)\).
Compute the categorical cross-entropy loss. Leave your answer in terms of \(\log\).
\(-\log(0.6)\).
By the categorical cross-entropy formula:
Only \(y_3 = 1\) contributes, so the loss is \(-\log(h_3) = -\log(0.6)\).
Tags: multiple outputs, lecture-16, categorical cross-entropy
A multi-class classifier has 3 output nodes with softmax activation. The true label is \(\vec y = (0, 1, 0)\) and the softmax outputs are \(\vec h = (0.3, 0.5, 0.2)\).
Compute the categorical cross-entropy loss. Leave your answer in terms of \(\log\).
\(-\log(0.5) = \log 2\).
By the categorical cross-entropy formula:
Only \(y_2 = 1\) contributes, so the loss is \(-\log(h_2) = -\log(0.5) = \log 2\).
Tags: multiple outputs, lecture-16, categorical cross-entropy
A multi-class classifier has 4 output nodes with softmax activation. The true label is \(\vec y = (1, 0, 0, 0)\) and the softmax outputs are \(\vec h = (0.4, 0.3, 0.2, 0.1)\).
Compute the categorical cross-entropy loss. Leave your answer in terms of \(\log\).
\(-\log(0.4)\).
By the categorical cross-entropy formula:
Only \(y_1 = 1\) contributes, so the loss is \(-\log(h_1) = -\log(0.4)\).
Tags: multiple outputs, sigmoid, lecture-16, softmax
A neural network classifies images into 5 categories.
Suppose the categories are mutually exclusive (each image belongs to exactly one category). Which activation function should be used at the output layer: sigmoid or softmax?
Softmax.
Since the categories are mutually exclusive, we want the output probabilities to represent a single probability distribution over the 5 classes. Softmax enforces this by ensuring the outputs sum to 1.
True or False: with the activation from part (a), the 5 outputs must sum to 1.
True.
The softmax function produces outputs that sum to 1 by definition:
Now suppose an image can belong to multiple categories simultaneously. Which activation function should be used at the output layer: sigmoid or softmax?
Sigmoid.
Since the categories are not mutually exclusive, each output node independently predicts the probability that the image belongs to that category. Sigmoid is applied independently to each output.
True or False: with the activation from part (c), the 5 outputs must sum to 1.
False.
Sigmoid is applied independently to each output node, so there is no constraint that the outputs sum to 1. For example, if the image contains both a cat and a dog, the network might output high probabilities for both.
Tags: multiple outputs, lecture-16, regression
A neural network with 3 output nodes is trained to predict temperature, humidity, and wind speed simultaneously. The network uses the multi-target regression loss:
For a particular data point, the network's predictions are \(\vec h = (5, 3, 7)\) and the true values are \(\vec y = (3, 4, 5)\). Compute the loss.
\(9\).
Tags: lecture-16, autoencoders
An autoencoder is trained on data in \(\mathbb{R}^{8}\) with a bottleneck (hidden) layer of dimension 3.
What are the input and output dimensions of the encoder?
The encoder maps \(\mathbb{R}^8 \to\mathbb{R}^3\). Its input dimension is 8 and its output dimension is 3.
What are the input and output dimensions of the decoder?
The decoder maps \(\mathbb{R}^3 \to\mathbb{R}^8\). Its input dimension is 3 and its output dimension is 8.
What is the output dimension of the full autoencoder, \(H(\vec x) = \operatorname{decode}(\operatorname{encode}(\vec x))\)?
\(8\), the same as the input dimension.
The autoencoder maps \(\mathbb{R}^8 \to\mathbb{R}^8\). Its goal is to reconstruct the input, so the output must have the same dimensionality.
True or False: training this autoencoder is a supervised learning problem.
False.
Training an autoencoder is an unsupervised learning problem. The network is trained to reconstruct its own input --- there are no separate labels. The loss function is the reconstruction error \(\sum_{i=1}^n \|\vec{x}^{(i)} - H(\vec{x}^{(i)})\|^2\).
Tags: lecture-16, autoencoders
True or False: if an autoencoder has 10 input nodes, 10 hidden nodes, and 10 output nodes, the smallest reconstruction error it can possibly achieve is zero.
True.
Since the hidden layer has the same dimensionality as the input, the autoencoder can learn the identity function, mapping each input to itself exactly. This results in zero reconstruction error.