Probability Theory: From Events to Numbers

Why It Matters to the Lab

Cities, data streams, and neural networks all share a common feature: uncertainty. Probability theory gives us the language to quantify it.

For Second Street Labs, this matters when we interpret noisy sensor data, design algorithms with randomness, or ensure robustness in the face of incomplete information.

The Probability Space

A probability space is a triple $(S, \mathcal{F}, P)$ where:

• $S$: the sample space (all possible outcomes).
• $\mathcal{F}$: a $\sigma$-algebra of subsets of $S$ (the measurable events).
• $P$: a probability measure that assigns numbers to events.

Together, these ensure that every “legal” event gets a consistent probability.

Kolmogorov’s Axioms

Probability is built on three axioms:

1. Non-negativity: $P(A) \geq 0$ for all $A \in \mathcal{F}$.
2. Normalization: $P(S) = 1$.
3. Countable Additivity: If $A_1, A_2, \dots$ are disjoint, $$P\Big(\bigcup_{i=1}^\infty A_i\Big) = \sum_{i=1}^\infty P(A_i).$$

From these, we can derive continuity, complements, and conditional probability.

Examples

• Coin Tosses: $P(\{H\}) = 0.5$, $P(\{T\}) = 0.5$.
• Twins Example: Assign probabilities to “identical twins,” “fraternal twins,” and “female twins” using set intersections and unions.
• Continuous Example: $S = \{x \in \mathbb{R}: x > 0\}$, with $P$ defined by an exponential distribution.

Why Machine Learning Cares

• Bayesian inference: Posterior updates are probability measures over parameter sets.
• Generalization bounds: Expressed in terms of probability of error over unseen samples.
• Stochastic optimization: Algorithms like SGD rely on treating gradients as random variables drawn from a probability space.

Whether explicit or implicit, probability is the glue that holds machine learning theory together.

Key Takeaways

Further Reading

• Casella & Berger, Statistical Inference (Chapter 1, probability axioms)
• Kolmogorov, Foundations of the Theory of Probability