- Leaf-wise vs Level-wise
- LGBM splits the tree leaf-wise (others do level-wise)
- The difference between the leaf wise and the level wise is in the order in which the trees are expanded.
- Leaf-wise chooses splits based on their contribution to the global loss and not just the loss along a particular branch.
- Leaf wise often learn lower-error trees faster than level-wise.
- For a given number of leaves, leaf-wise will probably outperform level-wise.
- Leaf-wise can lead to overfitting, but can be minimized by defining the depth for splitting
- Boosting vs Bagging
- Bagging and Boosting are both ensemble learning methonds in machine learning
- To use Bagging or Boosting, you must select a base learner algorithm. e.g. if we choose a classification tree, bagging and boosting would consist of a pool of trees as big as we want.
1) Ensemble Learning- The basic idea behind ensemble learning is to learn a set of classifiers(experts) and to allow them to vote.
- The main causes of error in learning are due to noise, bias, and variance. Ensemble helps to minimize these factors: Bagging helps to decrease the model's variance, Boosting helps to decrease the model's bias.
2) Bootstrapping - Bootstrapping is a sampling technique in which we create subsets of observations from the original dataset with replacement. The size of the subsets is the same as the size of the original set.
3) Bagging - Parallel process
- The idea behind Bagging (or Bootstrap Aggregation) is combining the results of multiple models to get a generalized result. - uses the subsets(bags) to get a fair idea of the distribution (complete set). The size of subsets created for bagging may be less than the original set.
4) Boosting - Sequential process
- each subsequent model attempts to correct the errors of the previous model
- The succeeding models are dependent on the previous model
- learners are learned sequentially with early learners fitting simple models to the data and then analyzing data for errors.
- we fit consecutive trees (random sample) and at every step, the goal is to solve for net error from the prior tree
- The basic idea behind ensemble learning is to learn a set of classifiers(experts) and to allow them to vote.
😎