EngSci Abridged visual companion · ECE368

Inference is a compiled probability model.

Factor the joint law. Insert evidence. Eliminate what is hidden. Then answer the question you actually asked.

Why can one model produce a class label, a prediction with uncertainty, a hidden-state marginal, or a globally most likely path?

Historical-source notice. The linked PDF is Aman Bhargava's unchanged January-April 2021 ECE368 condensation. It is a useful map, not a canonical or error-free textbook. This companion corrects material errors in expectation/MMSE, Gaussian algebra, regression, Markov-chain convergence, message passing, HMM recurrences, and EM. Read the page-located ledger.
01 · The probabilistic program

The structure stays. The factors and operators change.

A joint distribution is a declarative model. Inference compiles it into a computation by exploiting factorization and choosing what to eliminate.

Choose a question to see the four-stage computation.

Generative classification

A prior and class-conditional feature model become a posterior decision rule.

local discrete decision
  1. 1 · Model

    Factor the joint

    p(c,x) = p(c)p(x|c)

    Naive Bayes further factors the feature likelihood conditionally on the class.

  2. 2 · Observe

    Insert features

    x = observed feature vector

    The evidence is fixed; its common normalizer is not needed to rank classes.

  3. 3 · Eliminate

    Score each class

    s(c) = log p(c) + log p(x|c)

    Log space turns products into sums and prevents numerical underflow.

  4. 4 · Decide

    Minimize loss

    ĉ = arg maxc p(c|x)

    MAP is the Bayes rule for 0-1 loss. Other costs change the threshold.

Key distinction: changing a likelihood-ratio threshold changes the decision rule; it does not retrain the class-conditional model.

Bayesian linear regression

Gaussian conjugacy carries uncertainty from weights into predictions.

continuous posterior
  1. 1 · Model

    Prior plus likelihood

    w ~ N(μ₀,Σ₀)
    y|w ~ N(Xw,σ²I)

    Known observation noise and a Gaussian prior produce a Gaussian posterior.

  2. 2 · Observe

    Condition on data

    D = (X,y)

    The design and responses contribute information about the unknown weights.

  3. 3 · Eliminate

    Integrate weights

    ΣN-1 = Σ₀-1 + σ-2XᵀX

    Solve the linear systems; do not form explicit inverses in production code.

  4. 4 · Predict

    Return a distribution

    Var(y*) = σ² + φ*ᵀΣNφ*

    Fresh observation noise and remaining parameter uncertainty both survive.

Key distinction: a posterior mean is a point estimate; the posterior and predictive distributions carry uncertainty that the point alone discards.

Hidden Markov inference

A chain factorization supports two different dynamic programs.

structured latent state
  1. 1 · Model

    Factor the chain

    p(z₁) ∏ p(zt|zt-1) ∏ p(xt|zt)

    The state must contain enough history for the transition to be Markov.

  2. 2 · Observe

    Insert emissions

    x₁:T = observed sequence

    A missing observation contributes factor one for every possible state.

  3. 3 · Eliminate

    Choose the operator

    Σ paths → marginals
    max paths → joint MAP

    Forward-backward sums. Viterbi maximizes and stores predecessors.

  4. 4 · Return

    Answer one question

    p(zt|x₁:T) or z*₁:T

    Marginal modes and the globally most likely path are different objectives.

Key distinction: independently selecting the most likely state at each time need not produce a valid or jointly most probable trajectory.

Classification

Model
p(c)p(x|c)
Evidence
features x
Eliminate
score or normalize over classes
Output
posterior or cost-sensitive decision

Bayesian regression

Model
Gaussian prior and linear-Gaussian likelihood
Evidence
design X and response y
Eliminate
integrate weights
Output
predictive distribution

Hidden Markov model

Model
initial, transition, and emission factors
Evidence
observations x₁:T
Eliminate
sum or maximize hidden paths
Output
state marginals or joint MAP path
02 · The operator fork

Same factors. Different question. Different algebra.

Distributivity lets both computations reuse local messages. The elimination operator determines what the answer means.

Sum-product
Σhidden ∏ factors

Preserve posterior mass

  • Returns evidence, marginals, or normalized posteriors.
  • Forward-backward combines past and future evidence.
  • Exact on trees with the standard message equations.
Max-product / max-sum
maxhidden Σ log factors

Preserve one best configuration

  • Returns a joint MAP assignment, not local uncertainty.
  • Viterbi retains one maximizing predecessor per state.
  • Backtracking reconstructs a globally compatible path.

Do not swap the objective after the fact. Per-time marginal modes can have lower pointwise error yet fail to be the joint MAP path. Viterbi optimizes joint path probability; it does not promise minimum error at every time step.

03 · The engineering discipline

Inference fails in the implementation details.

These guardrails are reconstructed lessons, not claims that the historical lab code passed them.

Take logs.

Long products underflow. Add log factors and use log-sum-exp where sums remain.

Solve, do not invert.

Use Cholesky or stable linear solves for Gaussian updates and discriminants.

Make state sufficient.

If the future depends on finite history, include that history in the state.

Distinguish missing from impossible.

Missing evidence contributes one; impossible evidence contributes zero.

Enumerate tiny worlds.

Brute-force a small model to oracle-check marginals, normalization, and MAP paths.