Skip to contents

Multiclass Brier Score

Source: R/classif_mbrier.R
mbrier.Rd

Measure to compare true observed labels with predicted probabilities in multiclass classification tasks.

Usage

mbrier(truth, prob, sample_weights = NULL, ...)

Arguments

truth

(factor())
True (observed) labels. Must have the same levels and length as response.

prob

(matrix())
Matrix of predicted probabilities, each column is a vector of probabilities for a specific class label. Columns must be named with levels of truth.

sample_weights

(numeric())
Vector of non-negative and finite sample weights. Must have the same length as truth. The vector gets automatically normalized to sum to one. Defaults to equal sample weights.

...

(any)
Additional arguments. Currently ignored.

Value

Performance value as numeric(1).

Details

Brier score for multi-class classification problems with \(k\) labels defined as $$ \frac{1}{n} \sum_{i=1}^n \sum_{j=1}^k (I_{ij} - p_{ij})^2. $$ \(I_{ij}\) is 1 if observation \(x_i\) has true label \(j\), and 0 otherwise. \(p_{ij}\) is the probability that observation \(x_i\) belongs to class \(j\).

Note that there also is the more common definition of the Brier score for binary classification problems in bbrier().

Meta Information

  • Type: "classif"

  • Range: \([0, 2]\)

  • Minimize: TRUE

  • Required prediction: prob

References

Brier GW (1950). “Verification of forecasts expressed in terms of probability.” Monthly Weather Review, 78(1), 1–3. doi:10.1175/1520-0493(1950)078<0001:vofeit>2.0.co;2 .

See also

Other Classification Measures: acc(), bacc(), ce(), logloss(), mauc_aunu(), mcc(), zero_one()

Examples

set.seed(1)
lvls = c("a", "b", "c")
truth = factor(sample(lvls, 10, replace = TRUE), levels = lvls)
prob = matrix(runif(3 * 10), ncol = 3)
colnames(prob) = levels(truth)
mbrier(truth, prob)
#> [1] 1.084326