Inference is a compiled probability model.

A class label, a prediction with uncertainty, a hidden-state marginal, or the most probable path: one joint model answers all four, with different computations.

UofT ECE368 · Winter 2021 notes · 6 slides · about 2.5 minutes

The program

Full explainer

Slide 2 of 6 · Q: What are the four stages?

Factor. Observe. Eliminate. Decide.

The probabilistic program

$$\underbrace{p(z,x)}_{\text{factor}} \;\to\; \underbrace{x \text{ observed}}_{\text{evidence}} \;\to\; \underbrace{\textstyle\sum_z \;/\; \max_z}_{\text{eliminate}} \;\to\; \text{decision}$$

Factorization preserves the probability while changing the cost of computing it.

Three compiled forms

Full explainer

Slide 3 of 6 · Q: What does the spine compile into?

Same spine. Three model families.

Classification · 0–1 loss

$$\hat c = \arg\max_c\; \log p(c) + \log p(x\mid c)$$

Bayesian regression · known noise

$$\Sigma_N^{-1} = \Sigma_0^{-1} + \sigma^{-2} X^{\top}X \qquad \operatorname{Var}(y_*) = \sigma^2 + \varphi_*^{\top}\Sigma_N\varphi_*$$

Hidden Markov model

$$p(z_{1:T}, x_{1:T}) = p(z_1)\prod_{t} p(z_t\mid z_{t-1})\, p(x_t\mid z_t)$$

The operator fork

Full explainer

Slide 4 of 6 · Q: Sum or max?

Same factors. Different question. Different algebra.

Sum-product · marginals

$$p(z_t\mid x_{1:T}) = \sum_{z_{\setminus t}} \prod \text{factors}$$

Exact on trees; on cycles, not generically exact or convergent.

Max-product · Viterbi, joint MAP

$$z^{*}_{1:T} = \arg\max_{z_{1:T}} \; p(z_{1:T}\mid x_{1:T})$$

One globally compatible path via stored backpointers.

CarefulPer-time marginal modes can form a different, even impossible, sequence. Viterbi does not promise minimum error at every time.

Engineering discipline

Full explainer

Slide 5 of 6 · Q: Where does inference fail?

Inference fails in the implementation details.

Numerics · take logs, solve don't invert

$$\prod_i p_i \;\to\; \sum_i \log p_i \quad\text{(log-sum-exp for sums)}$$

Modeling · missing vs impossible evidence

$$\text{missing} \mapsto 1 \qquad \text{impossible} \mapsto 0$$

Make the state sufficient for the dynamics.

Verification

$$\text{engine} \;\overset{?}{=}\; \text{brute-force enumeration on a tiny model}$$

Provenance

Full explainer

Slide 6 of 6 · Q: What is this source?

A historical map, corrected in the margins.

Winter 2021 UofT ECE368 notes, preserved unchanged. Material errors are corrected in a page-located ledger, never silently.

SourceAman Bhargava's Jan–Apr 2021 notes; course taught by Saeideh Parsaei Fard and Foad Sohrabi. The 21-row errata ledger gives exact pages.