Neural Networks

Initial learning rate 0.1 Weight decay 0.0001 Momentum 0.9 Batch size 1024

optimizer = optim.SGD(model.parameters(), lr=0.1, momentum=0.9,weight_decay=0.0001)
 ...
trainLoader = torch.utils.data.DataLoader(
    datasets.LARGE('../data', train=True, download=True, 7 transform=transforms.Compose([
    transforms.ToTensor(),
])),
batch_size=1024, shuffle=True)

In your opinion, what could possibly go wrong with this training pipeline?

from sklearn.model_selection import train_test_split
dataset = datasets.load_iris()
X_train, X_test, y_train, y_test =
train_test_split(dataset.data, dataset.target, test_size=0.2) 
clf = LogisticRegression(data_norm=12)
clf.fit(X_train, y_train)

He then evaluated the performance of the trained model on the Xtest set.

1. Explain why his methodology is far from perfect.
2. Help him resolve the problem by utilizing a difference splitting methodology.
3. Your friend now amends the code an uses:

clf = GridSearchCV(method, params, scoring='roc_auc', cv=5) clf.fit(train_X, train_y)

Explain why his new approach may work better?

1. Draw a graph number of training epochs vs training error for when the learning rate is:
   1. too high
   2. too low
   3. acceptable.
2. What’s learning rate warmup? Why do we need it?

1. What’s the motivation?
   1. What might be the exceptions?

elements. During inference, you notice that the performance is not that good. A friend tells you that in computer vision faces are gathered in various poses and perspectives. He there- fore suggests that during inference you would augment the incoming face five times, run inference on each augmented image and then fuse the output probability distributions by averaging.

1. Name the method he is suggesting.
2. Provide several examples of augmentation that you might use during inference.

1. What are some sources of randomness in a neural network?
2. Sometimes stochasticity is desirable when training neural networks. Why is that?

1. What happens to your model training when you decrease the batch size to 1?
2. What happens when you use the entire training data in a batch?
3. How should we adjust the learning rate as we increase or decrease the batch size?

1. How do we know that gradients are exploding? How do we prevent it?
2. Why are RNNs especially susceptible to vanishing and exploding gradients?

1. What can you say about the ability to converge and generalize of Adam vs. SGD?
2. What else can you say about the difference between these two optimizers?

known to suffer from the phenomenon of saturated units.

1. Explain the phenomenon.
2. You switch to using the tanh activation instead of the logistic activation, in your opinion does the phenomenon still exists?
3. In your opinion, is using the tanh function makes the SGD operation to converge better?

1. In stochastic gradient descent we first calculate the gradient and only then adjust weights for each data point in the training set.
2. In stochastic gradient descent, the gradient for a single sample is not so different from the actual gradient, so this gives a more stable value, and converges faster.
3. SGD usually avoids the trap of poor local minima.
4. SGD usually requires more memory.

1. Which norm does the following equation represent? $|x1 − x2| + |y1 − y2|$
2. Which formulae does the following equation represent? $\sqrt{\sum_{i=1}^n(x_i - y_i)^2}$
3. When your read that someone penalized the L2 norm, was the euclidean or the Manhattan distance involved?
4. Compute both the Euclidean and Manhattan distance of the vectors: $x1$= [6,1,4,5] and$x2$ = [2,8,3,−1].

function (8.51):

from scipy import spatial
x1=[6,1,4,5]
x2=[2,8,3,-1]
cityblock = spatial.distance.cityblock(x1, x2) 5 print("Manhattan:", cityblock)

In many cases, and for large vectors in particular, it is better to use a GPU for imple- menting numerical computations. PyTorch has full support for GPU’s (and its my favourite DL library ... ), use it to implement the Manhattan distance function on a GPU.

1. What’s a dead neuron?
2. How do we detect them in our neural network?
3. How to prevent them?

1. Pruning is a popular technique where certain weights of a neural network are set to 0. Why is it desirable?
2. How do you choose what to prune from a neural network?

2. The normal distribution is defined as follows: $P(X) = \frac{1}{\sigma\sqrt{2\pi}}e^{\frac{-(x-\mu)^2}{2\sigma^2}}$ Generally i.i.d. $X ∼ N (μ, σ2)$ however BN uses the standard normal distribution. What mean and variance does the standard normal distribution have?
3. What is the mathematical process of normalization?
4. Describe, how normalization works in BN.