Skip to content

Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers

Research output: Chapter in Book/Report/Conference proceedingConference contribution

  • Meelis Kull
  • Telmo De Menezes E Silva Filho
Original languageEnglish
Title of host publicationProceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS 2017)
Publisher or commissioning bodyJournal of Machine Learning Research
Number of pages9
StatePublished - 1 Apr 2017
Event20th International Conference on Artificial Intelligence and Statistics (AISTATS 2017) - Fort Lauderdale, Florida, United States

Publication series

NameJMLR Workshop and Conference Proceedings
PublisherJournal of Machine Learning Research
Volume54
ISSN (Print)1938-7228

Conference

Conference20th International Conference on Artificial Intelligence and Statistics (AISTATS 2017)
Abbreviated titleAISTATS
CountryUnited States
CityFort Lauderdale, Florida
Period20/04/1722/04/17
Internet address

Abstract

For optimal decision making under variable class distributions and misclassification costs a classifier needs to produce well-calibrated estimates of the posterior probability. Isotonic calibration is a powerful non-parametric method that is however prone to overfitting on smaller datasets; hence a parametric method based on the logistic curve is commonly used. While logistic calibration is designed for normally distributed per-class scores, we demonstrate experimentally that many classifiers including Naive Bayes and Adaboost suffer from a particular distortion where these score distributions are heavily skewed. In such cases logistic calibration can easily yield probability estimates that are worse than the original scores. Moreover, the logistic curve family does not include the identity function, and hence logistic calibration can easily uncalibrate a perfectly calibrated classifier.
In this paper we solve all these problems with a richer class of calibration maps based on the beta distribution. We derive the method from first principles and show that fitting it is as easy as fitting a logistic curve. Extensive experiments show that beta calibration is superior to logistic calibration for Naive Bayes and Adaboost.

Event

20th International Conference on Artificial Intelligence and Statistics (AISTATS 2017)

Abbreviated TitleAISTATS
Conference number20
Duration20 Apr 201722 Apr 2017
CityFort Lauderdale, Florida
CountryUnited States
Web address (URL)
Degree of recognitionInternational event

Event: Conference

Download statistics

No data available

Documents

Documents

  • Full-text PDF (accepted author manuscript)

    Rights statement: This is the author accepted manuscript (AAM). The final published version (version of record) is available online via PMLR at http://proceedings.mlr.press/v54/kull17a.html . Please refer to any applicable terms of use of the publisher.

    Accepted author manuscript, 380 KB, PDF-document

Links

View research connections

Related faculties, schools or groups