How to Teach Machine Learning Models a Sense of Uncertainty | HackerNoon

Table of Links

Abstract and 1. Introduction

2. Motivation and design goals

3. Related Work

4. Conformal prediction

   4.1. Mondrian conformal prediction (MCP)

   4.2. Evaluation metrics

5. Mondrian conformal prediction for Disk Scrubbing: our approach

   5.1. System and Storage statistics

   5.2. Which disk to scrub: Drive health predictor

   5.3. When to scrub: Workload predictor

6. Experimental setting and 6.1. Open-source Baidu dataset

   6.2. Experimental results

7. Discussion

   7.1. Optimal scheduling aspect

   7.2. Performance metrics and 7.3. Power saving from selective scrubbing

8. Conclusion and References

4. Conformal prediction

Conformal prediction (Shafer and Vovk, 2008) is a powerful framework in machine learning that allows for prediction with uncertainty. Unlike traditional point prediction methods used in classification tasks, conformal prediction provides a set prediction. This means that instead of outputting a single predicted label, conformal prediction provides a range of possible labels that are likely to be correct, along with confidence and credibility measures in the correctness of these predictions. This is particularly useful in cases where the prediction task may be uncertain or when the model is dealing with previously unseen data, especially when the application is of high risk (Luo et al., 2022).

Conformal prediction is not limited to classification tasks alone, but it is also valid for regression tasks by providing prediction intervals instead of a single-point prediction. These prediction intervals represent a range of possible values for the target variable, along with a measure of confidence in the correctness of these intervals. This allows for more nuanced and interpretable predictions in regression tasks, where the goal is to estimate a continuous value rather than a discrete class label.

One of the key advantages of conformal prediction is its agnostic nature, which means it can be used with any machine learning algorithm. This flexibility allows for the integration of conformal prediction into various machine learning pipelines without being constrained by the choice of a specific algorithm. This makes conformal prediction a versatile tool that can be applied to various tasks and domains (Messoudi et al., 2020).

A great resource for learning more about conformal prediction is the book ”Algorithmic Learning in a Random World” (Vovk et al., 2022), which provides a detailed overview of the theory and applications of conformal prediction. Additionally, a repository of conformal prediction implementations and resources is maintained on GitHub[1] (Manokhin, 2022), providing practical tools and examples for applying conformal prediction in various machine learning settings. This makes it easier for researchers and practitioners to implement and experiment with conformal prediction in their own work.

The overall approach of conformal prediction in the inductive setting is presented in algorithm 1.

4.1. Mondrian conformal prediction (MCP)

Mondrian conformal prediction (MCP) is a variant of the conformal prediction framework that provides a guarantee on a subset of the dataset, or on specific categories of the dataset. This variant is originally established for a classification problem by creating class-conditional or attribute-conditional categories (Vovk et al., 2003). However, a Mondrian version exists for regressors (Bostr¨om and Johansson, 2020).

In imbalanced datasets, the minority class (i.e., the class with fewer instances) may lead to biased predictions and inaccurate confidence measures. Mondrian conformal prediction is a powerful tool to handle this issue by maintaining the same error rate for both the majority and minority classes, ensuring that the predictions are not biased toward the majority class.

Mondrian conformal prediction has been implemented for various domains in many real-life use cases for academia as well as industry. For instance, (Messoudi et al., 2021) apply MCP for tenant debt prediction in real estate, (Alvarsson et al., 2021) use it for modeling ABC transporters in drug discovery, and (Vishwakarma and Liu, 2021) leverage conformal predictors for detecting persistent storage failure analysis in the enterprise storage domain.

4.2. Evaluation metrics

To evaluate the performance of conformal prediction models, several metrics can be employed. In our study, we will focus only on two:

• Confidence: reflects the certainty of the model that a prediction is a singleton, or a unique outcome. Confidence is based on the concept of p-values, which are used to assess the probability of obtaining an outcome as extreme as the one observed, assuming that the null hypothesis is true. A higher confidence value suggests that the model is more confident about the accuracy of its prediction and that the predicted label is likely to be correct. Conversely, a lower confidence value implies that there may be alternative labels that are equally likely. This metric is defined as:

• Credibility quantifies the likelihood that a sample comes from the training set, as determined by the minimal significance level that would result in an empty prediction region. In other words, credibility is expressed as the largest p-value, which serves as the lower bound for the value of the significance level ϵ that would result in an empty prediction. A higher credibility value indicates a higher likelihood that the sample is consistent with the training set, while a lower credibility value indicates a higher likelihood of the sample being inconsistent with the training set. This metric is defined as:

:::info
This paper is available on arxiv under CC BY-NC-ND 4.0 Deed (Attribution-Noncommercial-Noderivs 4.0 International) license.

:::

[1] https://github.com/valeman/awesome-conformal-prediction

:::info
Authors:

(1) Rahul Vishwakarma, California State University Long Beach, 1250 Bellflower Blvd, Long Beach, CA 90840, United States ([email protected]);

(2) Jinha Hwang, California State University Long Beach, 1250 Bellflower Blvd, Long Beach, CA 90840, United States ([email protected]);

(3) Soundouss Messoudi, HEUDIASYC – UMR CNRS 7253, Universit´e de Technologie de Compiegne, 57 avenue de Landshut, 60203 Compiegne Cedex – France ([email protected]);

(4) Ava Hedayatipour, California State University Long Beach, 1250 Bellflower Blvd, Long Beach, CA 90840, United States ([email protected]).

:::