Neural Networks

1. Write in plain NumPy the forward and backward pass for a two-layer feed-forward neural network with a ReLU layer in between.
2. Implement vanilla dropout for the forward and backward pass in NumPy.

$output = \begin{cases} 0, & \text{if } \sum_j(w_jx_j + b) \leq 0, \\ 1, & \text{if } \sum_j(w_jx_j + b) > 0 \end{cases}$ Where weights are denoted by wj and biases are denoted by b. Answer the following questions:

1. True or False: If such a perceptron is trained using a labelled corpus, for each participating neuron the values $w_j$ and $b$ are learned automatically.
2. True or False: If we instead use a new perceptron (sigmoidal) defined as follows: $\sigma(wx + b)$ where $\sigma$ is the sigmoid function $\sigma(z) = \frac{1}{1+e^{-z}}$ Then the new perceptron can process inputs ranging between 0 and 1 and emit output ranging between 0 and 1.

$\nabla{C} = \frac{1}{n}\sum_x\nabla{C_x}$ Where: $C_x = \frac{1}{2}\|y(x) - a(x,w,b)\|^2$

1. Further to the above, the ReLU non-linear activation function $g(z) = max{0, z}$ is applied (8.15) to the output of the linear transformation. What is the value of the output (out2) now?

1. Draw the graphs for sigmoid, tanh, ReLU, and leaky ReLU.
2. Pros and cons of each activation function.
3. Is ReLU differentiable? What to do when it’s not differentiable?
4. Derive derivatives for sigmoid function when is a vector.

tion functions in a MLP. Which ones can never be back-propagated and why?

1. $f(x) = |x|$2.$f(x) = f(x)$3.$f(x)= \begin{cases} 0, & \text{if } x = 0, \\ x \sin\left(\frac{1}{x}\right), & \text{if } x \neq 0. \end{cases}$4.$f(x)= \begin{cases} 0 & \text{if } x = 0 \\ -x & \text{if } x < 0 \\ x^2 & \text{if } x > 0 \end{cases}$

classified samples divided by the total number of incorrectly classified samples.

ImageNet Classification with Deep Convolutional Neural Networks”

$f_{tanh}(x) = \frac{sinh(x)}{cosh(x)} = \frac{e^x - e^-x}{e^x + e^-x}$

  import torch
  nn001 = nn.Sequential(
  nn.Linear(200, 512),
  nn.Tanh(),
  nn.Linear(512, 512),
  nn.Tanh(),
  nn.Linear(512, 10),
  nn.LogSoftmax(dim=1)
  )

1. What type of a neural network does nn001 in 8.24 represent?
2. How many hidden layers does the layer entitles nn001 have?

1. Using the normal distribution, with parameter initialization method as suggested by Kaiming [14].
2. Using the uniform distribution, with parameter initialization method as suggested by Xavier Glorot [9].
3. Initialize all parameters to a constant zero value.

$f_{ReLU} = max(0, x)$

1. In what sense is the ReLU better than traditional sigmoidal activation functions?

$f(x) = \begin{cases} x & \text{if } x > 0 \\ ax & \text{otherwise} \end{cases}$

1. In, researchers employed an automatic pipeline for searching what exactly?
2. What types of functions did the researchers include in their search space?
3. What were the main findings of their research and why were the results surprising? 4. Write the formulae for the Swish activation function.
4. Plot the Swish activation function.

  import torchvision
  import torch
  def main():
  vgg11 = torchvision.models.vgg11(pretrained=True)
  vgg_layers = vgg11.features
  for param in vgg_layers.parameters():
  param.requires_grad = False
  example = [torch.rand(1, 3, 224, 224),
  torch.rand(1, 3, 512, 512),
  torch.rand(1, 3, 704, 1024)]
  vgg11.eval()
  for e in example:
    out=vgg_layers(e)
    print(out.shape)
  if __name__ == "__main__":
    main()^^I^^I

1. In each case for the input variable example , determine the dimensions of the tensor which is the output of applying the VGG11 CNN to the respective input.
2. Choose the correct option. The last layer of the VGG11 architecture is:
   1. Conv2d
   2. MaxPool2d
   3. ReLU

1. What type of block is now represented by the new line? Print it using PyTorch. 2. In each case for the input variable example , determine the dimensions of the tensor which is the output of applying the block: vgg_layers=vgg11.features[:3] to the respective input.

tensor after the application of the convolutional layer?

import torch
from torch import nn
class MaxPool001(nn.Module):
    def __init__(self): 
        super(MaxPool001, self).__init__() 
        self.math = torch.nn.Sequential(
        torch.nn.Conv2d(3, 32, kernel_size=7, padding=2),
        torch.nn.BatchNorm2d(32),
        torch.nn.MaxPool2d(2, 2),
        torch.nn.MaxPool2d(2, 2),
        )
def forward(self, x):
     print (x.data.shape)
    x = self.math(x)
    print (x.data.shape)
    x = x.view(x.size(0), -1)
    print ("Final shape:{}",x.data.shape)
    return x
model = MaxPool001()
model.eval()
x = torch.rand(1, 3, 224, 224)
out=model.forward(x)

The architecture is presented in 9.2:

import scipy
scipy.stats.norm.pdf(x, mu, sigma)

1. Without using Scipy, implement the normal distribution from scratch in Python.
2. Assume, you want to back propagate on the normal distribution, and therefore you need the derivative. Using Scipy write a function for the derivative.

1. Pooling layers
2. Convolutional layers
3. Normalization layers
4. Non-linear activation function
5. Linear activation function

1. CONV→act→BN→Dropout→...
2. CONV→act→Dropout→BN→...
3. CONV→BN→act→Dropout→...
4. BN→CONV→act→Dropout→...
5. CONV→Dropout→BN→act→...
6. Dropout→CONV→BN→act→...

$OUT = A⊙B$ Where: $(A⊙B)i,j :=(A)i,j(B)i,j$ A scientist, constructs a Dropout layer using the following algorithm:

1. Assign a probability of p for zeroing the output of any neuron.
2. Accept an input tensor T , having a shape S
3. Generate a new tensor T ‘ ∈ {0, 1}^S
4. Assign each element in T‘a randomly and independently sampled value from a Bernoulli distribution: $T‘i ∼ B(1,p)$
5. Calculate the OUT tensor as follows:$OUT = T‘⊙T$You are surprised to find out that his last step is to multiply the output of a dropout layer with: $\frac{1}{1-p}$ Explain what is the purpose of multiplying by the term $\frac{1}{1-p}$

1. Mathematically, the residual block may be represented by: $y = x + F(x)$ What is the function F?
2. In one sentence, what was the main idea behind deep residual networks (ResNets) as introduced in the original paper?