Calculus and Algorithmic Differentiation

1. What does it mean when a function is differentiable?
2. Give an example of when a function doesn’t have a derivative at a point.
3. Give an example of non-differentiable functions that are frequently used in machine learning. How do we do backpropagation if those functions aren’t differentiable?

1. What does it mean for a function to be convex or concave? Draw it.
2. Why is convexity desirable in an optimization problem?
3. Show that the cross-entropy loss function is convex.

2. What is the main problem with this formulae?
3. Indicate one problem with software tools which utilize numerical differentiation and successive operations on floating point numbers.

2. What other commonly used alternative name does the instantaneous rate of change have?
3. Given a function $f (x)$ and a point a, define the tangent line of $f (x)$ at a.

2. With respect to the DAG depicted in 5.3:

2. Elaborate how the chain rule is utilized in the context of neural networks.

1. $\frac{1}{1-x}$2.$e^x$3.$sin(x)$4.$cos(x)$

$\log(x)$

$f(x) = 5x^2 − 11x + 1$

$f(x) = cos(x)$

$f (x) = ln 3x.$

1. $\lim_{{x \to 3}}\frac{e^{x^3} - e^{27}}{3x - 9}$2. $\lim_{{x \to 0}}\frac{e^{x^2} - x - 1}{3\cos x-x - 3}$3. $\lim_{{x \to \inf}}\frac{x - ln x}{x^{1/100} + 4}$

2. Given $g(x, y)$, find its partial derivative with respect to x: $g(x,y) = x^2y + yx + 8y$

$∇f (x, y) = \frac{∂f}{∂x} i + \frac{∂f}{∂y}j$

1. Find the gradient of the function:$f(x, y) = xy^2 - y^2 + x^3$2. Given the function: $g(x, y) = xy^2 − y − 1$ evaluate it at $(−1, 0)$, directed at $(1, 1)$.

$f(x, y) = 3 sin^2(x − y)$

$z = 2 sin(x) sin(y)$

2. Where is $f(x)$ increasing and decreasing? 3. Where is $f(x)$ reaching minimum and maximum values.

2. Derive once again and discuss the concavity of the function $f (x)$.

$f(x,y)=2x^2 −xy + y^2$ and find maximum, minimum, and saddle points.

Stationary points are required in order to minimize a convex function. A very simple approach for finding stationary points is to start at an arbitrary point, and move along the gradient at that point towards the next point, and repeat until converging to a stationary point.

1. What is the term used to describe the vector of all partial derivatives for a function $f(x)$?
2. Complete the sentence: when searching for a minima, if the derivative is positive, the function is increasing/decreasing.
3. The function $x^2$ as depicted in 5.5, has a derivative of $f′(x) = 2x$. Evaluated at$x = −1$, the derivative equals$f′(x = −1) = −2$.$At x = −1$, the function is decreasing as$x$gets larger. We will happen if we wish to find a minima using gradient descent, and increase (decrease)$x$ by the size of the gradient, and then again repeatedly keep jumping?

$p(y=1|x)=σ(w^Tx)$ where the sigmoid function is given by: $σ(z)=(1+exp(−z))^{−1}$ The logistic loss for a training sample $x_i$ with class label $y_i$ is given by $L(yi,xi;w)=−logp(y_i|x_i)$1. Show that$p(y=−1|x)=σ(−w^Tx)$.

1. Show that $Δ_wL(y_i,x_i;w)=−y_i(1−p(y_i|x_i))x_i$.
2. Show that $Δ_wL(y_i,x_i;w)$ is convex.

1. How can we use second-order derivatives for training models?
2. Pros and cons of second-order optimization.
3. Why don’t we see more second-order optimization in practice?

$dZ = \frac{dJ}{dσ(x)}\frac{dσ(x)}{dx} = dA·σ(x).(1−σ(x))$ 2. Code snippet 5.6 provides a pure Python-based (e.g. not using Autograd) implementation of the forward pass for the sigmoid function. Complete the backward pass that directly computes the analytical gradients. python class Sigmoid: def forward(self,x): self.x = x return 1/(1+np.exp(-x)) def backward(self, grad): grad_input = [???] return grad_input

2. True or false: In Autograd, multiple calls to backward will sum up previously computed gradients if they are not zeroed.

$\tanh^{-1}(x) = \frac{1}{2}[\ln(1+x) - ln(1-x)]$ On the other hand, the $artanh$ function, which returns the inverse hyperbolic tangent of its argument x, is implemented in numpy as $arctanh()$. Its derivative is given by: $(arctanh(x))′ = \frac{1}{1 − x^2}$ Your friend, a veteran of the DL community wants to implement a custom activation function for the $arctanh$ function using Autograd. Help him in realize the method.

1. Use this numpy array as an input $[[0.37, 0.192, 0.571]]$ and evaluate the result using pure Python.
2. Use the PyTorch based $torch.autograd.Function$ class to implement a custom Function that implements the forward pass for the arctanh function in Python.
3. Use the PyTorch based $torch.autograd.Function$ class to implement a custom Function that implements the backward pass for the arctanh function in Python.
4. Name the class ArtanhFunction,and using the grad check method from $torch.autograd$, verify that your numerical values equate the analytical values calculated by gradcheck. Remember you must implement a method entitled$.apply(x)$ so that the function can be invoked by Autograd.

2. Explain the notion of DN as introduced by ([2]).
3. What arithmetic operations are possible on DN?.
4. Explain the relationship between a Taylor series and DN.

$\sin(x + \dot{x}d)$ 2. With respect to the expression graph depicted in 5.9:

1. Find the function, $g(A, B, C)$ represented by the expression graph in 5.11.

M outputs.

1. True or False? (a) Forward and reverse mode AD always yield the same result. (b) In reverse mode AD there are fewer operations (time) and less space for intermediates (memory). (c) The cost for forward mode grows with N. (d) The cost for reverse mode grows with M.

$f (x1, x2) = x1x2 + ln (x1)$ and the graph 5.1, annotate each vertex (edge) of the graph with the partial derivatives that would be propagated in forward mode AD. 2. Transform the graph into a table that computes the function: $g(x1, x2)$ evaluated at $(x1; x2) = (e^2; π)$ using forward-mode AD. 3. Write and run a Python code snippet to prove your results are correct. 4. Describe the role of seed values in forward-mode AD. 5. Transform the graph into a table that computes the derivative of $g(x1,x2)$ evaluated at $(x1; x2) = (e^2; π)$ using forward-mode AD for $x1$ as the chosen independent variable. 6. Write and run a Python code snippet to prove your results are correct.

1. Which differentiation method is inherently prone to rounding errors?
2. Define the term symbolic differentiation.

1. Implement the sigmoid function $σ(x) = 1$ symbolically using a Python based 1+e−x SymPy program.
2. Differentiate the sigmoid function using SymPy and compare it with the analytical derivation $σ′(x) = σ(x)(1 − σ(x))$ .
3. Using SymPy, evaluate the gradient of the sigmoid function at $x = 0$.
4. Using SymPy, plot the resulting gradient of the sigmoid function.

We discussed the Beta-Binomial model extensively in chapter 3. Recall that the Beta- Binomial distribution is frequently used in Bayesian statistics to model the number of suc- cesses in n trials. We now employ SymPy to do the same; demonstrate computationally how a prior distribution is updated to develop into a posterior distribution after observing the data via the relationship of the Beta-Binomial distribution. Provided the probability of success, the number of successes after n trials follows a binomial distribution. Note that the beta distribution is a conjugate prior for the parameter of the binomial distribution. In this case, the likelihood function is binomial, and a beta prior distribution yields a beta posterior distribution. Recall that for the Beta-Binomial distribution the following relationships exist: (5.27)