Intro to Energy-based-models

Introduction

Energy-Based Models capture dependencies by associating a scalar energy to each configuration of the variables.
Inference (making a prediction or decision) consists in setting the value of observed variables and finding values of the remaining variables that minimize the energy.
Learning consists in finding an energy function that associates low energies to correct values of the remaining variables, and higher energies to incorrect values.
Loss Function is used to measure the quality of the available energy functions.
Model is an energy function which measures the "goodness" of each possible configuration of X and Y. Therefore, the output number of $E(X,Y)$ can be interpreted as the degree of compatibility between the values of $X$ and $Y$.

• High Energy values correspond to highly incompatible configurations of the variables.
• Low Energy values correspond to highly compatible configurations of the variables.
• Note that Energy Function $\cancel =$ Loss Function.
  - Energy function is minimized by the inference process.
  • Loss functional is minimized by the learning process.

Example 1)

Let $X$ be an input (observed from the world), and the model produces the answer $Y$ that is most compatible with the observed $X$.

If the size of the set $\mathcal Y$ is small, we can compute $E(Y,X)$ for all possible values of $Y \in \mathcal Y$, and pick the smallest.

If $Y$ is a collection of discrete variables,

Energy Function Architectures

(a) Regressor: The energy is the discrepancy between the output of the regression function $G_W(X)$ and the answer $Y$. The best is $Y^*=G_W(X)$.
(b) Simple two-class classifier: The set of possible answers is ${-1, +1}$. The best inference is $Y^*=sign(G_W(X))$.
(c) Multiclass classifier: The discriminant function produces one value for each of the three categories. The answer, which can take three values, controls the position of a "switch", which connects one output of the discriminant function to the energy function. The best inference is the index of the smallest output component of $G_W(X)$.

1. Regression
Energy function is defined to be the following.

where $G_W(X)$ is the regression function.

Energy loss, perceptron loss and negative log-likelihood loss are all equivalent because the contrastive term of the perceptron loss is zero, and that of the NLL loss is constant (Gaussian integral with a constant variance).

Now, linear regression occurs when $G$ is a linear function of the parameters: