| Title: | Cumulative Calibration Assessment for Prediction Models |
| Version: | 0.0.1 |
| Description: | Tools for visualization of, and inference on, the calibration of prediction models on the cumulative domain. This provides a method for evaluating calibration of risk prediction models without having to group the data or use tuning parameters (e.g., loess bandwidth). This package implements the methodology described in Sadatsafavi and Patkau (2024) <doi:10.1002/sim.10138>. The core of the package is cumulcalib(), which takes in vectors of binary responses and predicted risks. The plot() and summary() methods are implemented for the results returned by cumulcalib(). |
| License: | MIT + file LICENSE |
| URL: | https://github.com/resplab/cumulcalib |
| Imports: | graphics, stats |
| Suggests: | knitr, predtools, rmarkdown, markdown, spelling, testthat (≥ 3.0.0) |
| VignetteBuilder: | knitr |
| Config/testthat/edition: | 3 |
| Encoding: | UTF-8 |
| RoxygenNote: | 7.3.1 |
| Language: | en-US |
| NeedsCompilation: | no |
| Packaged: | 2024-06-13 12:51:31 UTC; msafavi |
| Author: | Mohsen Sadatsafavi
 [aut, cre] |
| Maintainer: | Mohsen Sadatsafavi <mohsen.sadatsafavi@ubc.ca> |
| Repository: | CRAN |
| Date/Publication: | 2024-06-13 18:10:06 UTC |
cumulcalib: Cumulative Calibration Assessment for Prediction Models
Description
Tools for visualization of, and inference on, the calibration of prediction models on the cumulative domain. This provides a method for evaluating calibration of risk prediction models without having to group the data or use tuning parameters (e.g., loess bandwidth). This package implements the methodology described in Sadatsafavi and Patkau (2024) doi:10.1002/sim.10138. The core of the package is cumulcalib(), which takes in vectors of binary responses and predicted risks. The plot() and summary() methods are implemented for the results returned by cumulcalib().
Author(s)
Maintainer: Mohsen Sadatsafavi mohsen.sadatsafavi@ubc.ca (ORCID)
See Also
Useful links:
Cumulative calibration assessment
Description
This is the core function for performing cumulative calibration assessment
Usage
cumulcalib(y, p, method = c("BB", "BM"), ordered = FALSE, n_sim = 0)
Arguments
y |
vector of binary responses |
p |
vector of predicted probabilities. |
method |
string with either BB (Brownian bridge test, default method), BM (Brownian motion test), BM2p (two-part BM test - experimental), BB1p (one-part BB test wit only the 'bridge' component). Multiple methods can be specified. The first one will be the 'main' method (e.g., when submitting the resulting object to plot()). Default is c("BB","BM") |
ordered |
if TRUE, y and p are already ordered based on ascending values of p. This is to speed up simulations. |
n_sim |
if >0, indicates a simulation-based test is requested for inference. |
Value
an objective of class cumulcalib that can be printed or plotted
Examples
pi <- rbeta(1000,1,2)
Y <- rbinom(length(pi),1,pi)
res <- cumulcalib(Y, pi, method="BB")
summary(res)
plot(res)
CDF of the Kolmogorov distribution
Description
CDF of the Kolmogorov distribution
Usage
pKolmogorov(q, summands = ceiling(q * sqrt(72) + 3/2))
Arguments
q |
the quantity at which CDF will be evaluated. Currently accepts only a scalar |
summands |
maximum number of terms to be evaluated in the infinite series (default=ceiling(q*sqrt(72)+3/2)) |
Value
a scalar value
CDF of the distribution of the maximum absolute deviation of Brownian motion in [0,1] interval
Description
CDF of the distribution of the maximum absolute deviation of Brownian motion in [0,1] interval
Usage
pMAD_BM(q, summands = 100)
Arguments
q |
the quantity at which CDF will be evaluated. Currently accepts only a scalar |
summands |
maximum number of terms to be evaluated in the infinite series (default=100) |
Value
a scalar value
CDF of the distribution of the maximum absolute deviation of Brownian motion in [0,1] interval, conditional on its terminal value
Description
CDF of the distribution of the maximum absolute deviation of Brownian motion in [0,1] interval, conditional on its terminal value
Usage
pMAD_BM_c(
q,
w1,
method = 1,
exp_tolerance = -30,
summands = ceiling(q * sqrt(72) + 3/2)
)
Arguments
q |
the quantity at which CDF will be evaluated. Currently accepts only a scalar |
w1 |
the terminal value |
method |
different infinite series to use (1,2,3) |
exp_tolerance |
numerical tolerance as the stopping rule when evaluating the infinite sum (default -30 on the exponential scale) |
summands |
number of terms to evaluate (default is ceiling(q * sqrt(72) + 3/2)) |
Value
a scalar value
Generates cumulative calibration plots
Description
Generates cumulative calibration plots
Usage
## S3 method for class 'cumulcalib'
plot(
x,
method = NULL,
draw_stat = TRUE,
stat_col = c("blue", "red"),
draw_sig = TRUE,
sig_level = c(0.95, 0.95),
x2axis = TRUE,
y2axis = TRUE,
...
)
## S3 method for class 'cumulcalib'
plot(
x,
method = NULL,
draw_stat = TRUE,
stat_col = c("blue", "red"),
draw_sig = TRUE,
sig_level = c(0.95, 0.95),
x2axis = TRUE,
y2axis = TRUE,
...
)
Arguments
x |
An object of class cumulcalib generated by cumulcalib() |
method |
Which method to use. Options are BB (Brownian bridge test), BM (Brownian motion test), BB1p (1-part Brownian bridge test), and BM2p (2-part Brownian bridge test). If unspecified, returns the default method used in the cumulcalib() call |
draw_stat |
Should the statistic (terminal value an/or maximum drift, depending on method) be drawn? Default is TRUE |
stat_col |
The color(s) to draw the stat. Default is c('blue','red'). For single-part tests (BM and BB1p) only the first element is used |
draw_sig |
Whether significance lines should be drawn. Default is T. Colors will be the same as stat_col |
sig_level |
If to draw significance lines, at what level? Default is c(0.95,0.95). For single-part tests (BM and BB1p) only the first element is used |
x2axis |
If true, draws a second x-axis (on top) showing predicted risks |
y2axis |
If true, draws a second y-axis (on right) showing scaled partial sums |
... |
Parameters to be passed to plot() |
Value
None
None
Quantile function of the Kolmogorov distribution
Description
Quantile function of the Kolmogorov distribution
Usage
qKolmogorov(p)
Arguments
p |
the quantity at which the quantile function will be evaluated. Currently accepts only a scalar |
Value
a scalar value
Quantile function of the distribution of the maximum absolute deviation of Brownian motion in [0,1] interval
Description
Quantile function of the distribution of the maximum absolute deviation of Brownian motion in [0,1] interval
Usage
qMAD_BM(p)
Arguments
p |
the quantity at which the quantile function will be evaluated. Currently accepts only a scalar |
Value
a scalar value
Quantile function of the distribution of the maximum absolute deviation of Brownian motion in [0,1] interval, conditional on its terminal value
Description
Quantile function of the distribution of the maximum absolute deviation of Brownian motion in [0,1] interval, conditional on its terminal value
Usage
qMAD_BM_c(p, w1)
Arguments
p |
the quantity at which the quantile function will be evaluated. Currently accepts only a scalar |
w1 |
the terminal value |
Value
a scalar value
Summarizes a cumulcalib object
Description
Summarizes a cumulcalib object
Summarizes a cumulcalib object
Usage
## S3 method for class 'cumulcalib'
summary(object, method = NULL, ...)
## S3 method for class 'cumulcalib'
summary(object, method = NULL, ...)
Arguments
object |
An object of class cumulcalib generated by cumulcalib() |
method |
Which method to use. Options are BB (Brownian bridge test), BM (Brownian motion test), BB1p (1-part Brownian bridge test), and BM2p (2-part Brownian bridge test). If unspecified, returns the default method used in the cumulcalib() call |
... |
Other parameters passed to summary() |
Value
None
None