+ Reply to Thread
Results 1 to 2 of 2

Thread: Brier score calculation: 2 methods should yield same result

  1. #1
    Points: 8, Level: 1
    Level completed: 15%, Points required for next Level: 42

    Posts
    2
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Brier score calculation: 2 methods should yield same result




    I have a set of 234 predictions of tennis match outcomes and 5 different prediction models. I use the two different methods for calculating the Brier score described, for instance, here.

    The first method is:

    BS=\frac{1}N\sum_{t=1}^N(f_{t}-o_{t})^2


    Where N is the number of forecasting instances, f_{t} is the forecast probability of the t-th instance, and o_{t} is the outcome (either 0 or 1).

    The second method decomposes the brier score into *Resolution*, *Reliability*, and *Uncertainty*:

    BS=\frac{1}{N}\sum\limits _{k=1}^{K}{n_{k}(\mathbf{f_{k}}-\mathbf{\bar{o}}_{\mathbf{k}})}^{2}-\frac{1}{N}\sum\limits _{k=1}^{K}{n_{k}(\mathbf{\bar{o}_{k}}-\bar{\mathbf{o}})}^{2}+\mathbf{\bar{o}}\left({1-\mathbf{\bar{o}}}\right)

    With \textstyle N being the total number of forecasts issued, \textstyle K the number of unique forecasts issued, \mathbf{\bar{o}}={\sum_{t=1}^{N}}\mathbf{{o_t}}/N the observed base rate for the event to occur, n_{k} the number of forecasts with the same probability category and \mathbf{\overline{o}}_{\mathbf{k}} the observed frequency, given forecasts of probability \mathbf{f_{k}}.

    For more details see Wikipedia.

    So for my calculations I would expect that both methods yield the same results. However, I can't achieve this with my data, hence, there must be an error. I provide a minimal working example with my data (google spreadsheet) where I show how I calculate the brier score in both ways.

    Here is the link to the minimal "working" example: https://goo.gl/uwUoLU
    (nb: the *Resolution* in my example is 0, because in this example the data set is not split into bins, hence the *bin base rate* (aka *observed frequency*) equals the *overall observed base rate*)

    I would greatly appreciate, if

    - you could point me to errors in my calculation
    - explain, what I am doing wrong
    - provide the correct calculations for both ways of calculating the Brier score

    I already spent a few hours on getting this right, but did not succeed. As pointed out above, I think I probably have a slight error when applying the decomposed brier score formula (second method). However, I could not identify, what I am doing wrong exactly. One thing I noticed is, that the second formula speaks about *total number of forecasts* and *number of unique of forecasts*. Since my example (for simplicity) only uses 1 bin, I wonder if *total number of forecasts* and *number of unique of forecasts* are the same, or do they mean something different?

  2. #2
    Points: 8, Level: 1
    Level completed: 15%, Points required for next Level: 42

    Posts
    2
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Brier score calculation: 2 methods should yield same result


    thread can be deleted. i made a subtle mistake in my calculations...

+ Reply to Thread

           




Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats