Entropy
A mathematical measure of randomness or unpredictability in a probability distribution, frequently used in machine learning to quantify information content.
Think of It Like This
Like the difference between a loaded die that always rolls a six and a fair die where you have absolutely no idea what number will come up next.
In information theory, entropy represents the expected amount of information in an event. Machine learning models often use cross-entropy as a loss function to minimize the difference between predicted probabilities and actual outcomes. Lower entropy indicates higher confidence and less randomness in the model's predictions.