Quantifying Uncertainty – Entropy | Eyal Kazin

Life is like a box of chocolate. Generated using DALL-E

My momma always said “Life was like a box of chocolates. You never know what you’re gonna get.”

— F. Gump (fictional philosopher and entrepreneur)

This is the second article in a series on information quantification — an essential framework for data scientists. Learning to measure information unlocks powerful tools for improving statistical analyses and refining decision criteria in machine learning.

In this article we focus on entropy — a fundamental concept that quantifies “on average, how surprising is an outcome?” As a measure of uncertainty, it bridges probability theory and real-world applications, offering insights into applications from data diversity to decision-making.

We’ll start with intuitive examples, like coin tosses and roles of dice , to build a solid foundation. From there, we’ll explore entropy’s diverse applications, such as evaluating decision tree splits and quantifying DNA diversity . Finally, we’ll dive into fun puzzles like the Monty Hall problem and I’ll refer to a tutorial for optimisation of the addictive WORDLE game .

_{Images are for reference only.Images and contents gathered automatic from google or 3rd party sources.All rights on the images and contents are with their legal original owners.}