Multiclass Brier Score

Measure to compare true observed labels with predicted probabilities in multiclass classification tasks.

Usage

mbrier(truth, prob, ...)

Arguments

truth: (factor())
True (observed) labels. Must have the same levels and length as response.
prob: (matrix())
Matrix of predicted probabilities, each column is a vector of probabilities for a specific class label. Columns must be named with levels of truth.
...: (any)
Additional arguments. Currently ignored.

Value

Performance value as numeric(1).

Details

Brier score for multi-class classification problems with $k$ labels defined as $\frac{1}{n} \sum_{i = 1}^{n} \sum_{j = 1}^{k} (I_{i j} - p_{i j})^{2} .$ $I_{i j}$ is 1 if observation $x_{i}$ has true label $j$ , and 0 otherwise. $p_{i j}$ is the probability that observation $x_{i}$ belongs to class $j$ .

Note that there also is the more common definition of the Brier score for binary classification problems in bbrier().

Meta Information

Type: "classif"
Range: $[0, 2]$
Minimize: TRUE
Required prediction: prob

References

Brier GW (1950). “Verification of forecasts expressed in terms of probability.” Monthly Weather Review, 78(1), 1–3. doi:10.1175/1520-0493(1950)078<0001:vofeit>2.0.co;2 .

Examples

set.seed(1)
lvls = c("a", "b", "c")
truth = factor(sample(lvls, 10, replace = TRUE), levels = lvls)
prob = matrix(runif(3 * 10), ncol = 3)
colnames(prob) = levels(truth)
mbrier(truth, prob)
#> [1] 1.084326