Deep Learning

The ReLU function is a fundamental activation function used in the field of machine learning, particularly within neural networks.

Leaky ReLU is an activation function commonly used in deep learning models, particularly in neural networks. It represents improved version of ReLU.

The ELU is a nonlinear activation function used in neural networks, introduced as a means to enhance model learning and convergence speed.

The hyperbolic tangent function, commonly referred to as tanh, is a widely used activation function in neural networks.

The sigmoid activation function, also known as the logistic function, is a significant activation function in the field of neural network.

The perceptron is a type of artificial neuron or simplest form of a neural network. It is the foundational building block of more complex neural net.

Perceptron regression extends the concept of the perceptron from classification to regression tasks.

Stochastic Gradient Descent (SGD) is a fundamental optimization technique used widely in machine learning and deep learning for minimizing the loss.

The Momentum Optimizer is an advanced variant of the classical SGD algorithm, designed to accelerate the convergence of gradient-based optimization.

RMSProp is an adaptive learning rate optimization algorithm designed to address some of the drawbacks of traditional SGD methods.

Adam is a popular optimization algorithm widely used in the training of deep neural networks. Introduced by Kingma and Ba in their 2014 paper.

AdaGrad is a gradient-based optimization algorithm that adjusts the learning rate to the parameters, performing smaller updates.

AdaDelta is an optimization algorithm designed to address the rapidly diminishing learning rates encountered in AdaGrad.

Adamax is a variant of the Adam optimization algorithm, which is itself an extension of the stochastic gradient descent method incorporating momentum

AdamW is an extension of the Adam optimization algorithm, specifically designed to better handle weight decay in the training of deep learning models.

NAdam, or Nesterov-accelerated Adaptive Moment Estimation, is an optimization algorithm that combines the techniques of Adam and Nesterov momentum.

Kaiming Initialization, also known as He Initialization, is a technique used to initialize the weights of deep neural networks, particularly with ReLU

Xavier Initialization, also known as Glorot Initialization, is a strategy for weight initialization in neural networks.

The convolution operation is a cornerstone in many fields such as signal processing, image processing, and machine learning.

A pooling layer is a common component in convolutional neural networks (CNNs) that reduces the spatial dimensions of the input.

The dense layer, or FC layer, is an essential building block in many neural network architectures used for a broad range of machine learning.

A Dropout Layer is a regularization technique used in neural networks to prevent overfitting.

The Flatten layer is a crucial component in the architecture of many deep learning models, particularly those dealing with image and video processing.

The softmax activation function is a crucial component in the field of machine learning, particularly in the context of multi classification problems.

The Scaled Exponential Linear Unit (SELU) is an activation function used in neural networks that automatically induces self-normalizing properties .

The Softplus activation function is a smooth, nonlinear function used in neural networks as an alternative to the commonly used ReLU.

The Swish activation function is a relatively recent addition to the repertoire of activation functions used in deep learning.

Cross-entropy loss, also known as log loss, measures the performance of a classification model whose output is a probability value between 0 and 1.

Binary cross-entropy loss is a specific instance of cross-entropy loss used primarily for binary classification tasks.

A Multilayer Perceptron (MLP) is a class of feedforward artificial neural network (ANN) that consists of at least three layers of nodes.

Convolutional Neural Networks (CNNs) are a class of deep neural networks that are primarily used to analyze visual imagery.