climb.tool.impl.data_suite.third_party.uq360.utils.calibrators package

Submodules

climb.tool.impl.data_suite.third_party.uq360.utils.calibrators.calibrator module

class climb.tool.impl.data_suite.third_party.uq360.utils.calibrators.calibrator.Calibrator

Bases: Base

Base class for calibrators for binary classification problems. Calibrators produce monotonic shifts in model confidences which preserve the order of confidence scores. Given as input the confidence score for class “1” in a binary classification problem, they transform this score into an estimated probability that class “1” was a correct prediction.

fit()

classmethod instance(subtype_name=None, **params)

property json_registry

load(input_location=None)

property pkl_registry

predict()

register_json_object(obj, name)

register_pkl_object(obj, name)

save(output_location=None)

climb.tool.impl.data_suite.third_party.uq360.utils.calibrators.confidence_binning module

class climb.tool.impl.data_suite.third_party.uq360.utils.calibrators.confidence_binning.ConfidenceBinsCalibrator

Bases: Calibrator

Calibrator based on histogram of model confidence scores. Recalibrates based on the sampling distribution from the (calibrator) training set. The (calibrator) train set accuracy for each set of samples defined by a confidence histogram bin is used as the recalibrated confidence value at inference time for any sample falling into that bin.

fit(probs, ground_truth)

get_confidence_dictionary(probs, ground_truth)

load(input_location=None)

classmethod name()

Name of this subtype, used for lookup purposes. To be implemented by subclasses. This method can either return a single name, or a list/tuple of names.

predict(preds)

save(output_location=None)

climb.tool.impl.data_suite.third_party.uq360.utils.calibrators.isotonic_regression module

class climb.tool.impl.data_suite.third_party.uq360.utils.calibrators.isotonic_regression.IsotonicRegressionCalibrator

Bases: Calibrator

Calibrator based on isotonic regression procedure. This calibrator finds the best piecewise-constant, monotonic function of the confidences to recalibrate to represent the probability of a correct classification.

fit(predicted_confidences, labels)

load(input_location=None)

classmethod name()

Name of this subtype, used for lookup purposes. To be implemented by subclasses. This method can either return a single name, or a list/tuple of names.

predict(predicted_confidences)

save(output_location=None)

climb.tool.impl.data_suite.third_party.uq360.utils.calibrators.linear_extrapolation module

class climb.tool.impl.data_suite.third_party.uq360.utils.calibrators.linear_extrapolation.LinearExtrapolationCalibrator

Bases: Calibrator

Calibrator based on a fitted linear transformation of the confidence scores.

fit(predicted_confidences, labels)

lin_equ(l1, l2)

classmethod name()

Name of this subtype, used for lookup purposes. To be implemented by subclasses. This method can either return a single name, or a list/tuple of names.

predict(predicted_confidences)

climb.tool.impl.data_suite.third_party.uq360.utils.calibrators.shift_calibrator module

class climb.tool.impl.data_suite.third_party.uq360.utils.calibrators.shift_calibrator.ShiftCalibrator

Bases: Calibrator

Calibrator based on a fitted constant shift.

fit(predicted_confidences, labels)

load(input_location=None)

classmethod name()

Name of this subtype, used for lookup purposes. To be implemented by subclasses. This method can either return a single name, or a list/tuple of names.

predict(predicted_confidences)

save(output_location=None)

Module contents