DSC 140B
Problems tagged with softmax

Problems tagged with "softmax"

Problem #166

Tags: multiple outputs, lecture-16, softmax

A neural network with 3 output nodes uses the softmax activation function. The pre-activation values (logits) at the output layer are \(\vec z = (0, 2, 0)\). Compute the softmax output \(\vec h = (h_1, h_2, h_3)\).

Leave your answer in terms of \(e\).

Solution
\[\vec h = \left(\frac{1}{2 + e^2},\;\frac{e^2}{2 + e^2},\;\frac{1}{2 + e^2}\right)\]

By the softmax formula, \(h_k = \frac{e^{z_k}}{\sum_{j=1}^{3} e^{z_j}}\). The denominator is:

\[ e^{0} + e^{2} + e^{0} = 1 + e^2 + 1 = 2 + e^2 \]

Therefore:

$$\begin{align*} h_1 &= \frac{e^0}{2 + e^2} = \frac{1}{2 + e^2}\\ h_2 &= \frac{e^2}{2 + e^2}\\ h_3 &= \frac{e^0}{2 + e^2} = \frac{1}{2 + e^2}\end{align*}$$

Problem #167

Tags: multiple outputs, lecture-16, softmax

A neural network with 4 output nodes uses the softmax activation function. The pre-activation values (logits) at the output layer are \(\vec z = (1, 3, 1, 3)\). Compute the softmax output \(\vec h = (h_1, h_2, h_3, h_4)\).

Leave your answer in terms of \(e\).

Solution
\[\vec h = \left(\frac{1}{2(1 + e^2)},\;\frac{e^2}{2(1 + e^2)},\;\frac{1}{2(1 + e^2)},\;\frac{e^2}{2(1 + e^2)}\right)\]

By the softmax formula, \(h_k = \frac{e^{z_k}}{\sum_{j=1}^{4} e^{z_j}}\). The denominator is:

\[ e^{1} + e^{3} + e^{1} + e^{3} = 2e + 2e^3 = 2e(1 + e^2) \]

Therefore:

$$\begin{align*} h_1 = h_3 &= \frac{e}{2e(1 + e^2)} = \frac{1}{2(1 + e^2)}\\ h_2 = h_4 &= \frac{e^3}{2e(1 + e^2)} = \frac{e^2}{2(1 + e^2)}\end{align*}$$

Problem #168

Tags: multiple outputs, lecture-16, softmax

A neural network with 3 output nodes uses the softmax activation function. The pre-activation values (logits) at the output layer are \(\vec z = (1, 2, 3)\). Compute the softmax output \(\vec h = (h_1, h_2, h_3)\).

Leave your answer in terms of \(e\).

Solution
\[\vec h = \left(\frac{1}{1 + e + e^2},\;\frac{e}{1 + e + e^2},\;\frac{e^2}{1 + e + e^2}\right)\]

By the softmax formula, \(h_k = \frac{e^{z_k}}{\sum_{j=1}^{3} e^{z_j}}\). The denominator is:

\[ e^{1} + e^{2} + e^{3} = e(1 + e + e^2) \]

Therefore:

$$\begin{align*} h_1 &= \frac{e}{e(1 + e + e^2)} = \frac{1}{1 + e + e^2}\\ h_2 &= \frac{e^2}{e(1 + e + e^2)} = \frac{e}{1 + e + e^2}\\ h_3 &= \frac{e^3}{e(1 + e + e^2)} = \frac{e^2}{1 + e + e^2}\end{align*}$$

Problem #175

Tags: multiple outputs, sigmoid, lecture-16, softmax

A neural network classifies images into 5 categories.

Part 1)

Suppose the categories are mutually exclusive (each image belongs to exactly one category). Which activation function should be used at the output layer: sigmoid or softmax?

Solution

Softmax.

Since the categories are mutually exclusive, we want the output probabilities to represent a single probability distribution over the 5 classes. Softmax enforces this by ensuring the outputs sum to 1.

Part 2)

True or False: with the activation from part (a), the 5 outputs must sum to 1.

True False
Solution

True.

The softmax function produces outputs that sum to 1 by definition:

\[\sum_{k=1}^{K} h_k = \sum_{k=1}^{K}\frac{e^{z_k}}{\sum_{j} e^{z_j}} = 1 \]

Part 3)

Now suppose an image can belong to multiple categories simultaneously. Which activation function should be used at the output layer: sigmoid or softmax?

Solution

Sigmoid.

Since the categories are not mutually exclusive, each output node independently predicts the probability that the image belongs to that category. Sigmoid is applied independently to each output.

Part 4)

True or False: with the activation from part (c), the 5 outputs must sum to 1.

True False
Solution

False.

Sigmoid is applied independently to each output node, so there is no constraint that the outputs sum to 1. For example, if the image contains both a cat and a dog, the network might output high probabilities for both.