Neural Networks

1. Write in plain NumPy the forward and backward pass for a two-layer feed-forward neural network with a ReLU layer in between.
2. Implement vanilla dropout for the forward and backward pass in NumPy.

$output = \begin{cases} 0, & \text{if } \sum_j(w_jx_j + b) \leq 0, \\ 1, & \text{if } \sum_j(w_jx_j + b) > 0 \end{cases}$ Where weights are denoted by wj and biases are denoted by b. Answer the following questions:

1. True or False: If such a perceptron is trained using a labelled corpus, for each participating neuron the values $w_j$ and $b$ are learned automatically.
2. True or False: If we instead use a new perceptron (sigmoidal) defined as follows: $\sigma(wx + b)$ where $\sigma$ is the sigmoid function $\sigma(z) = \frac{1}{1+e^{-z}}$ Then the new perceptron can process inputs ranging between 0 and 1 and emit output ranging between 0 and 1.

$\nabla{C} = \frac{1}{n}\sum_x\nabla{C_x}$ Where: $C_x = \frac{1}{2}\|y(x) - a(x,w,b)\|^2$

1. Further to the above, the ReLU non-linear activation function $g(z) = max{0, z}$ is applied (8.15) to the output of the linear transformation. What is the value of the output (out2) now?

1. Draw the graphs for sigmoid, tanh, ReLU, and leaky ReLU.
2. Pros and cons of each activation function.
3. Is ReLU differentiable? What to do when it’s not differentiable?
4. Derive derivatives for sigmoid function when is a vector.

tion functions in a MLP. Which ones can never be back-propagated and why?

1. $f(x) = |x|$2.$f(x) = f(x)$3.$f(x)= \begin{cases} 0, & \text{if } x = 0, \\ x \sin\left(\frac{1}{x}\right), & \text{if } x \neq 0. \end{cases}$4.$f(x)= \begin{cases} 0 & \text{if } x = 0 \\ -x & \text{if } x < 0 \\ x^2 & \text{if } x > 0 \end{cases}$

classified samples divided by the total number of incorrectly classified samples.

ImageNet Classification with Deep Convolutional Neural Networks”

$f_{tanh}(x) = \frac{sinh(x)}{cosh(x)} = \frac{e^x - e^-x}{e^x + e^-x}$

  import torch
  nn001 = nn.Sequential(
  nn.Linear(200, 512),
  nn.Tanh(),
  nn.Linear(512, 512),
  nn.Tanh(),
  nn.Linear(512, 10),
  nn.LogSoftmax(dim=1)
  )

1. What type of a neural network does nn001 in 8.24 represent?
2. How many hidden layers does the layer entitles nn001 have?

1. Using the normal distribution, with parameter initialization method as suggested by Kaiming [14].
2. Using the uniform distribution, with parameter initialization method as suggested by Xavier Glorot [9].
3. Initialize all parameters to a constant zero value.

$f_{ReLU} = max(0, x)$

1. In what sense is the ReLU better than traditional sigmoidal activation functions?

$f(x) = \begin{cases} x & \text{if } x > 0 \\ ax & \text{otherwise} \end{cases}$

1. In, researchers employed an automatic pipeline for searching what exactly?
2. What types of functions did the researchers include in their search space?
3. What were the main findings of their research and why were the results surprising? 4. Write the formulae for the Swish activation function.
4. Plot the Swish activation function.

  import torchvision
  import torch
  def main():
  vgg11 = torchvision.models.vgg11(pretrained=True)
  vgg_layers = vgg11.features
  for param in vgg_layers.parameters():
  param.requires_grad = False
  example = [torch.rand(1, 3, 224, 224),
  torch.rand(1, 3, 512, 512),
  torch.rand(1, 3, 704, 1024)]
  vgg11.eval()
  for e in example:
    out=vgg_layers(e)
    print(out.shape)
  if __name__ == "__main__":
    main()^^I^^I

1. In each case for the input variable example , determine the dimensions of the tensor which is the output of applying the VGG11 CNN to the respective input.
2. Choose the correct option. The last layer of the VGG11 architecture is:
   1. Conv2d
   2. MaxPool2d
   3. ReLU

1. What type of block is now represented by the new line? Print it using PyTorch. 2. In each case for the input variable example , determine the dimensions of the tensor which is the output of applying the block: vgg_layers=vgg11.features[:3] to the respective input.

tensor after the application of the convolutional layer?

import torch
from torch import nn
class MaxPool001(nn.Module):
    def __init__(self): 
        super(MaxPool001, self).__init__() 
        self.math = torch.nn.Sequential(
        torch.nn.Conv2d(3, 32, kernel_size=7, padding=2),
        torch.nn.BatchNorm2d(32),
        torch.nn.MaxPool2d(2, 2),
        torch.nn.MaxPool2d(2, 2),
        )
def forward(self, x):
     print (x.data.shape)
    x = self.math(x)
    print (x.data.shape)
    x = x.view(x.size(0), -1)
    print ("Final shape:{}",x.data.shape)
    return x
model = MaxPool001()
model.eval()
x = torch.rand(1, 3, 224, 224)
out=model.forward(x)

The architecture is presented in 9.2:

import scipy
scipy.stats.norm.pdf(x, mu, sigma)

1. Without using Scipy, implement the normal distribution from scratch in Python.
2. Assume, you want to back propagate on the normal distribution, and therefore you need the derivative. Using Scipy write a function for the derivative.

1. Pooling layers
2. Convolutional layers
3. Normalization layers
4. Non-linear activation function
5. Linear activation function

1. CONV→act→BN→Dropout→...
2. CONV→act→Dropout→BN→...
3. CONV→BN→act→Dropout→...
4. BN→CONV→act→Dropout→...
5. CONV→Dropout→BN→act→...
6. Dropout→CONV→BN→act→...

$OUT = A⊙B$ Where: $(A⊙B)i,j :=(A)i,j(B)i,j$ A scientist, constructs a Dropout layer using the following algorithm:

1. Assign a probability of p for zeroing the output of any neuron.
2. Accept an input tensor T , having a shape S
3. Generate a new tensor T ‘ ∈ {0, 1}^S
4. Assign each element in T‘a randomly and independently sampled value from a Bernoulli distribution: $T‘i ∼ B(1,p)$
5. Calculate the OUT tensor as follows:$OUT = T‘⊙T$You are surprised to find out that his last step is to multiply the output of a dropout layer with: $\frac{1}{1-p}$ Explain what is the purpose of multiplying by the term $\frac{1}{1-p}$

1. Mathematically, the residual block may be represented by: $y = x + F(x)$ What is the function F?
2. In one sentence, what was the main idea behind deep residual networks (ResNets) as introduced in the original paper?

Initial learning rate 0.1 Weight decay 0.0001 Momentum 0.9 Batch size 1024

optimizer = optim.SGD(model.parameters(), lr=0.1, momentum=0.9,weight_decay=0.0001)
 ...
trainLoader = torch.utils.data.DataLoader(
    datasets.LARGE('../data', train=True, download=True, 7 transform=transforms.Compose([
    transforms.ToTensor(),
])),
batch_size=1024, shuffle=True)

In your opinion, what could possibly go wrong with this training pipeline?

from sklearn.model_selection import train_test_split
dataset = datasets.load_iris()
X_train, X_test, y_train, y_test =
train_test_split(dataset.data, dataset.target, test_size=0.2) 
clf = LogisticRegression(data_norm=12)
clf.fit(X_train, y_train)

He then evaluated the performance of the trained model on the Xtest set.

1. Explain why his methodology is far from perfect.
2. Help him resolve the problem by utilizing a difference splitting methodology.
3. Your friend now amends the code an uses:

clf = GridSearchCV(method, params, scoring='roc_auc', cv=5) clf.fit(train_X, train_y)

Explain why his new approach may work better?

1. Draw a graph number of training epochs vs training error for when the learning rate is:
   1. too high
   2. too low
   3. acceptable.
2. What’s learning rate warmup? Why do we need it?

1. What’s the motivation?
   1. What might be the exceptions?

elements. During inference, you notice that the performance is not that good. A friend tells you that in computer vision faces are gathered in various poses and perspectives. He there- fore suggests that during inference you would augment the incoming face five times, run inference on each augmented image and then fuse the output probability distributions by averaging.

1. Name the method he is suggesting.
2. Provide several examples of augmentation that you might use during inference.

1. What are some sources of randomness in a neural network?
2. Sometimes stochasticity is desirable when training neural networks. Why is that?

1. What happens to your model training when you decrease the batch size to 1?
2. What happens when you use the entire training data in a batch?
3. How should we adjust the learning rate as we increase or decrease the batch size?

1. How do we know that gradients are exploding? How do we prevent it?
2. Why are RNNs especially susceptible to vanishing and exploding gradients?

1. What can you say about the ability to converge and generalize of Adam vs. SGD?
2. What else can you say about the difference between these two optimizers?

known to suffer from the phenomenon of saturated units.

1. Explain the phenomenon.
2. You switch to using the tanh activation instead of the logistic activation, in your opinion does the phenomenon still exists?
3. In your opinion, is using the tanh function makes the SGD operation to converge better?

1. In stochastic gradient descent we first calculate the gradient and only then adjust weights for each data point in the training set.
2. In stochastic gradient descent, the gradient for a single sample is not so different from the actual gradient, so this gives a more stable value, and converges faster.
3. SGD usually avoids the trap of poor local minima.
4. SGD usually requires more memory.

1. Which norm does the following equation represent? $|x1 − x2| + |y1 − y2|$
2. Which formulae does the following equation represent? $\sqrt{\sum_{i=1}^n(x_i - y_i)^2}$
3. When your read that someone penalized the L2 norm, was the euclidean or the Manhattan distance involved?
4. Compute both the Euclidean and Manhattan distance of the vectors: $x1$= [6,1,4,5] and$x2$ = [2,8,3,−1].

function (8.51):

from scipy import spatial
x1=[6,1,4,5]
x2=[2,8,3,-1]
cityblock = spatial.distance.cityblock(x1, x2) 5 print("Manhattan:", cityblock)

In many cases, and for large vectors in particular, it is better to use a GPU for imple- menting numerical computations. PyTorch has full support for GPU’s (and its my favourite DL library ... ), use it to implement the Manhattan distance function on a GPU.

1. What’s a dead neuron?
2. How do we detect them in our neural network?
3. How to prevent them?

1. Pruning is a popular technique where certain weights of a neural network are set to 0. Why is it desirable?
2. How do you choose what to prune from a neural network?

2. The normal distribution is defined as follows: $P(X) = \frac{1}{\sigma\sqrt{2\pi}}e^{\frac{-(x-\mu)^2}{2\sigma^2}}$ Generally i.i.d. $X ∼ N (μ, σ2)$ however BN uses the standard normal distribution. What mean and variance does the standard normal distribution have?
3. What is the mathematical process of normalization?
4. Describe, how normalization works in BN.