Tree Based Methods in Machine Learning

1. Majority Voting
2. Using K-identical base-learning algorithms
3. Using K-different base-learning algorithms
4. Using K-different data-folds
5. Using K-different random number seeds
6. A combination of all the above approaches

1. Stacking since each classier is trained on all of the available data.
2. Bagging since we can combine as many classifiers as we want by training each on a different sub-set of the training corpus.

In your opinion, which approach is being utilized?

2. True or false: The generalization accuracy of an ensemble increases with the number of well-trained models it consists of.
3. True or false: Bootstrap aggregation (or bagging), refers to a process wherein a CNN ensemble is being trained using a random subset of the training corpus.
4. True or false: Bagging assumes that if the single predictor shave independent errors, then a majority vote of their outputs should be better than the individual predictions.

1. A mixture of experts
2. Bagging
3. Ensemble averaging
4. Boosting

1. Do deep ensembles achieve a better performance on out-of-distribution uncertainty benchmarks compared with Monte-Carlo (MC)-dropout?

the number of adjustable parameters is large compared to cardinality (i.e., size of the set) of the training corpus.

2. Provide examples of ensemble methods in classical machine-learning.
3. True or false: Ensemble methods usually have stronger generalization ability.
4. Complete the sentence: Bagging is variance/bias reduction scheme while boosting reduced variance/bias.

  import torchvision.models as models ...
  models = ['resnext']
  for m in models: 
      train ...
      compute VAL loss ... 
      amend LR ...
      if (val_acc > 90.0):
          saveModel()

1. What type of ensembling can be used with this approach? Explain in detail.
2. What is the main advantage of snapshot ensembling? What are the disadvantages, if any?

  import torchvision.models as models
  import random
  import np
  ...
  models = ['resnext']
  for m in models:
      train ...
      compute loss ...
      amend LR ...
      manualSeed= draw a new random number
      random.seed(manualSeed)
      np.random.seed(manualSeed)
      torch.manual_seed(manualSeed)
      if (val_acc > 90.0):
          saveModel()

Explain in detail what would be the possible effects of adding lines 10-13.

    import torchvision.models as models 
    ...
    models = ['resnext','vgg','dense']
    for m in models: 
        train ...
        compute loss/acc 
        ... 
        if (val_acc > 90.0):
            saveModel()
    ```
    What type of ensembling is being utilized in this approach? Explain in detail.
2. Name one method by which NN models may be combined to yield a single prediction.

1. What is gradient boosting?
2. What problems is gradient boosting good for?