Formal Description

Detection vs Estimation

Detection(Classification)

: Discrete set of hypotheses(right or wrong, Classification etc.)

Estimation(Regression)

: Continuous set of hypotheses(almost always wrong - minimize error instead)

Sort of Detection & Estimation

Classical : Fixed, non-random Hypotheses/parameters

Bayesian : Random, non-fixed Hypotheses/parameters

Bayesian problem assume priors or priori distributions

Mathematical Estimation Problem

Parameter estimation

• From discrete-time waveform or data set, i.e., N-point data set depending on $\theta$

  • Fixed(Deterministic) $\theta$ $\rightarrow$ Classical Estimation

    $\{x[0], x[1], \cdots, x[N-1]\} \rightarrow \hat{\theta} = g(x[0], x[1], \cdots, x[N-1])$

• Mathematically model the data $\rightarrow$ probability density funcion(PDF) due to the randomness

  • $\cdots, x[N-1]; \;\theta)$ : ; means it's deterministic

    $p(x[0], x[1], \cdots, x[N-1]; \theta), \quad x[i] = \theta + n[i], \quad n[i] \sim N(0, \sigma^2):white\;gaussian\;noise$

• Example assume that it's Gaussian : $N=1, \;\theta$ denotes the mean, $x[0]\sim N(\theta, {\sigma}^2)$ (N: Gaussian Normal Distribution)

  $p(x[0];\theta)=\frac{1}{\sqrt{2\pi\sigma^2}}exp[-\frac{1}{2\sigma^2}(x[0]-\theta^2)]$

• Infer the value of $\theta$ from the observed value of $x[0]$

  • If observed value equal to 20 than probaility of observation 20 maximized when $\theta$ is 20.
    

• In actual problems, PDF is not given but chosen

  • Consistent with constraints and prior knowledge
  • Mathematically tractable

    We don't know the actual distribution $\rightarrow$ The thing only we can do is Guess the distribution using every background, knowledge etc.

• Example : Dow-Jones industrial average

  • $x[n] = A + Bn + w[n]$
  • $w[n]$ : White Guassian noise(WGN) $\sim N(0, \sigma ^ 2)$
    • White : i.i.d $\rightarrow$ independent and identically distributed
  • $\theta = [A, B]^T, \;x= [x[0]\;x[1]\;\cdots\;x[N-1]]^T$
    $p(x;\theta)\;=\;\frac{1}{(2\pi\sigma^2)^{\frac{N}{2}}}exp\left[ \begin{array}{ll} -\frac{1}{2\sigma^2} \displaystyle \sum_{n=0}^{N-1} (x[n] -A-Bn)^2 \end{array} \right]\\ =\;p(x[0], x[1], \cdots,x[n-1];\theta)\\ = p(x[0];\theta) \times \cdots \times P(x[n-1];\theta)$

      - It can be just Product of marginal Probs because it'd White(iid)

    Why Gaussian?

    Widely used and convience of calculation

    CLT(Central Limit Theorem)

    Add enough observations $\rightarrow$ Follow Gaussian Distribution
    For Guassian When WSS(Wide Sense Stationary) $\rightarrow$ SSS(Stric Sence Stationary)

Types of estimation

• Classical Estimation : Parameters of interest are assumed to be Deterministic
• Baysian Estimation : Parameters are assumed to be Random Variables to exploit any prior knowledge
  • Example : Average of Dow-Jones industrial average is in [2800, 3200] $\leftarrow$ background knowledge(prior)

Estimator and Estimate

• Estimator : A rule(Function) that assigns a value to $\theta$ for each realization of $x$

  Function of Random Variable $x\quad\rightarrow$ Random Variable
  $\hat \theta = g(x)\;\rightarrow$ Function $g(x)$ is Estimator

• Estimate : The value of $\theta$ obtained for a given realization of $x$

  $\hat \theta = g(x)\;\rightarrow$ Getting value of $\theta$ into $\hat\theta$$ is Estimation

Accessing Estimator Performance

Better Estimator? (Sample mean vs First sample value)

• Example of the DC level in noise
  • $x[n]=A+w[n]$
  • $A$ : unknown DC level target
  • $w[n]$ : zero mean Gaussian process $\sim N(0, \sigma^2)$
  • $N$ observations : $\{x[0], x[1], \cdots,x[N-1]\}$
• Two candidate estimators : Sample Mean vs First sample value
  $\hat A = \frac{1}{N}\displaystyle\sum_{n=0}^{N-1}x[n], \quad \check A=x[0]$
• Statiscal Analysis
  • Mean
    $x[n] = A+w[n]\\E[x[n]] = A + E[w[n]] = A,\quad E[w[n]] =0\\[1cm] E(\hat A) = E(\frac{1}{N}\displaystyle\sum_{n=0}^{N-1}x[n])=\frac{1}{N}\displaystyle\sum_{n=0}^{N-1}E(x[n]) = \textcolor{red}A\\ E(\check A) = E(x[0]) = \textcolor{red}A\\$

    We can switch $E$ and $\sum$

  • Variance
    $Var(x[n]) = E[(x[n]-E[x[\mu]])^2]\\ =E[w^2[n]]\\ =Var(w[n]) + E[w^2[n]],\quad E[w^2[n]] = 0\\ \therefore Var(x[n]) = \sigma ^2\\[1cm] Var(\hat A) = Var\left(\frac{1}{N} \displaystyle \sum_{n=0}^{N-1}x[n]\right) = \frac{1}{N^2}\displaystyle\sum_{n=0}^{N-1}Var(x[n])\\ =\frac{1}{N^2}N\sigma^2 = \textcolor{red}{\frac{\sigma^2}{N}}\\[0.5cm] Var(\check A) = Var(x[0] ) = \textcolor{red}{\sigma^2}$
  • What's better? $\rightarrow$ Sample mean is better estimator than First value

    Estimator	Mean	Variance
    Sample mean	$A$	$\frac{\sigma^2}{N}$
    First value	$A$	$\sigma^2$

Mathematical Detection Problem

Binary Hypothesis Test

• Noise only hypothesis vs. signal present hypothesis(Deterministic signals)
  $H_0 :x[n]=w[n]\\H_1:x[n]=s[n]+w[n]$
• Example of the DC level in noise
  $s[n] = A = 1, \;w[n]$: zero mean Gaussian process $\sim N(0, \sigma^2)$
  $p(x[0];\;H_0)=\frac{1}{\sqrt{2\pi\sigma^2}}exp\left[-\frac{1}{2\sigma^2}x[0]^2\right]\\[0.3cm] p(x[0];\;H_1) = \frac{1}{\sqrt{2\pi\sigma^2}}exp\left[-\frac{1}{2\sigma^2}(x[0]-1)^2\right]$
  
• Which Detector is better?
  $H_1\quad: x[0] > 1/2$ vs $H_0 \quad :otherwise$
• Receiver Operating Characteristic(ROC) curves
  • Probability of false alarm($P_{FA}=P($decide $H_1$ when $H_0$ is true$)$ : Correct
  • Probability of detection($P_D=P($decide $H_1$ when $H_1$ is true$)$ : Wrong
    
    

All Content has been written based on lecture of Prof. eui-seok.Hwang in GIST(Detection and Estimation)