Title: Fast Functions for Liu Regression with Regularization Parameter and Statistics
Version: 1.0
Date: 2024-02-02
Maintainer: Murat Genç <muratgenc@tarsus.edu.tr>
Description: Efficient computation of the Liu regression coefficient paths, Liu-related statistics and information criteria for a grid of the regularization parameter. The computations are based on the 'C++' library 'Armadillo' through the 'R' package 'Rcpp'.
License: GPL (≥ 3)
Encoding: UTF-8
NeedsCompilation: yes
RoxygenNote: 7.2.3
LinkingTo: Rcpp, RcppArmadillo
Imports: graphics, Rcpp, stats
Depends: R (≥ 3.6)
Author: Murat Genç ORCID iD [aut, cre], Ömer Özbilen [aut]
LazyData: true
Packaged: 2024-02-04 19:11:27 UTC; murat
Repository: CRAN
Date/Publication: 2024-02-05 20:50:11 UTC

fastliu: Fast Functions for Liu Regression with Regularization Parameter and Statistics

Description

Efficient computation of the Liu regression coefficient paths, Liu-related statistics and information criteria for a grid of the regularization parameter. The computations are based on the 'C++' library 'Armadillo' through the 'R' package 'Rcpp'.

Author(s)

Maintainer: Murat Gen<U+00E7> muratgenc@tarsus.edu.tr (ORCID)

Authors:

Hitters

Description

Major League Baseball Data from the 1986 and 1987 seasons.

Format

A data frame with 322 observations of major league players on the following 20 variables with explanations.

AtBat Number of times at bat in 1986
Hits Number of hits in 1986
HmRun Number of home runs in 1986
Runs Number of runs in 1986
RBI Number of runs batted in 1986
Walks Number of walks in 1986
Years Number of years in the major leagues
CAtBat Number of times at bat during his career
CHits Number of hits during his career
CHmRun Number of home runs during his career
CRuns Number of runs during his career
CRBI Number of runs batted in during his career
CWalks Number of walks during his career
League A factor with levels A and N indicating player's league at the end of 1986
Division A factor with levels E and W indicating player's division at the end of 1986
PutOuts Number of put outs in 1986
Assists Number of assists in 1986
Errors Number of errors in 1986
Salary 1987 annual salary on opening day in thousands of dollars
NewLeague A factor with levels A and N indicating player's league at the beginning of 1987

Source

The dataset was retrieved from the StatLib library maintained at Carnegie Mellon University. This is part of the data used in the 1988 ASA Graphics Section Poster Session. The dataset is available in the R package ISLR2 (James et al., 2022). For more details, see the book, An Introduction to Statistical Learning with applications in R by James et al. (2013).

References

James, G., Witten, D., Hastie, T., and Tibshirani, R. (2013). An Introduction to Statistical Learning with applications in R, https://www.statlearning.com, Springer-Verlag, New York.

James G, Witten D, Hastie T, Tibshirani R (2022). ISLR2: Introduction to Statistical Learning, Second Edition. R package version 1.3-2, https://CRAN.R-project.org/package=ISLR2.

Extract coefficient estimates from a liureg object

Description

Prints coefficient estimates from a fitted liureg object.

Usage

## S3 method for class 'liureg'
coef(object, ...)

Arguments

object

A liureg object.

...

Not used in this implementation.

Value

The returned object is a data.frame containing the coefficients path.

Author(s)

Murat Genç

See Also

liureg(), predict(), summary(), pressliu(), residuals()

Examples

data("Hitters")
Hitters <- na.omit(Hitters)
X <- model.matrix(Salary ~ ., Hitters)[, -1]
y <- Hitters$Salary
lam <- seq(0, 1, 0.01)
liu.mod <- liureg(X, y, lam)
coef(liu.mod)

Covariance matrix based on a fitted liureg object.

Description

For a scalar or vector tuning parameter lambda, the covliureg computes the covariance matrix for the estimates of a Liu regression model.

Usage

covliu(obj)

Arguments

obj

A liureg object.

Value

The returned object is a list of the matrix of estimated covariances.

Author(s)

Murat Genç and Ömer Özbilen

See Also

liureg(), coef(), predict(), summary(), pressliu(), residuals()

Examples

data("Hitters")
Hitters <- na.omit(Hitters)
X <- model.matrix(Salary ~ ., Hitters)[, -1]
y <- Hitters$Salary
lam <- seq(0, 1, 0.01)
liu.mod <- liureg(X, y, lam)
# List of covariance matrices for 101 lambda values
cov.mat <- covliu(liu.mod)
print(cov.mat$lam1)

Diagonal Elements of the Hat Matrix

Description

For each value of the regularization parameter lambda, diagHliu returns the diagonal elements of the hat matrix. Unlike the hatliu function, only the diagonal elements of the hat matrix are calculated, thus the computation of diagonal elements is faster than hatliu.

Usage

diagHliu(obj)

Arguments

obj

A liureg object

Value

The returned object is a matrix whose columns are the diagonal elements of the hat matrix for each value of the lambda regularization parameter.

Author(s)

Murat Genç

See Also

liureg(), summary(), pressliu(), residuals()

Examples

data("Hitters")
Hitters <- na.omit(Hitters)
X <- model.matrix(Salary ~ ., Hitters)[, -1]
y <- Hitters$Salary
lam <- seq(0, 1, 0.01)
liu.mod <- liureg(X, y, lam)
diagHliu(liu.mod)

Hat matrix of Liu Regression

Description

For each value of the regularization parameter lambda, hatliu returns the hat matrix of Liu regression. The hat matrix for Liu regression is computed using the formula \mathbf{H}=\mathbf{X}\left(\mathbf{X}^{T}\mathbf{X}+\mathbf{I}_{p}\right)^{-1} \left(\mathbf{X}^{T}\mathbf{X}+\lambda\mathbf{I}_{p}\right)\left(\mathbf{X}^{T} \mathbf{X}\right)^{-1}\mathbf{X}^{T}.

Usage

hatliu(obj)

Arguments

obj

A liureg object.

Value

The returned object is a list of matrices whose elements are the hat matrices for the values of the lambda regularization parameter.

Author(s)

Murat Genç

See Also

liureg(), summary(), pressliu(), residuals()

Examples

data("Hitters")
Hitters <- na.omit(Hitters)
X <- model.matrix(Salary ~ ., Hitters)[, -1]
y <- Hitters$Salary
lam <- seq(0, 1, 0.01)
liu.mod <- liureg(X, y, lam)
# Hat matrix list
hatlist <- hatliu(liu.mod)
# Hat matrix for third regularization parameter
hatlist[[3]]

Information Criteria for Liu Regression

Description

For each value of lambda, infoliu calculates the values of the AIC and BIC model selection criteria. Model selection criteria are based on the degrees of the freedom, \texttt{df}=\mathrm{trace}\left(\mathbf{H}_\lambda\right) of the Liu regression model where \mathbf{H} is the hat matrix of Liu regression model.

Usage

infoliu(obj)

Arguments

obj

A liureg object

Value

infoliu returns the matrix of information criteria for each value of the regularization parameter lambda.

Author(s)

Murat Genç

References

Akaike, H. (1974). A new look at the statistical model identification. IEEE Transaction on Automatic Control, 9(6), 716-723. doi:10.1109/TAC.1974.1100705.

Liu, K. (1993). A new class of blased estimate in linear regression. Communications in Statistics-Theory and Methods, 22(2), 393-402. doi:10.1080/03610929308831027.

Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6(2), 461-464. doi:10.1214/aos/1176344136.

See Also

predict(), summary()

Examples

data("Hitters")
Hitters <- na.omit(Hitters)
X <- model.matrix(Salary ~ ., Hitters)[, -1]
y <- Hitters$Salary
lam <- seq(0, 1, 0.01)
liu.mod <- liureg(X, y, lam)
infoliu(liu.mod)

Computation of Liu Tuning Parameter

Description

lamest computes the Liu tuning parameters provided in the literature. The tuning parameter estimates are based on

Usage

lamest(obj, ...)

Arguments

obj

An object of class liureg.

...

Not used in this implemetation.

Details

The lamest function computes the following tuning parameter estimates available in the literature.

lam.mm (Liu, 1993) \displaystyle{1-\hat{\sigma}^{2}\left(\frac{\sum\limits _{j=1}^{p}\frac{1}{\lambda_{j}\left(1+\lambda_{j}\right)}}{\sum\limits _{j=1}^{p}\frac{\hat{\alpha}_{j}^{2}}{\left(1+\lambda_{j}\right)^{2}}}\right)}
lam.CL (Liu, 1993) \displaystyle{1-\hat{\sigma}^{2}\left(\frac{\sum\limits _{j=1}^{p}\frac{1}{\left(1+\lambda_{j}\right)}}{\sum\limits _{j=1}^{p}\frac{\lambda_{j}\hat{\alpha}_{j}^{2}}{\left(1+\lambda_{j}\right)^{2}}}\right)}
lam.opt (Liu, 1993) \displaystyle{\frac{\sum\limits _{j=1}^{p}\left(\frac{\alpha_{j}^{2}-\sigma^{2}}{\left(1+\lambda_{j}\right)^{2}}\right)}{\sum\limits _{j=1}^{p}\left(\frac{\sigma^{2}+\lambda_{j}\alpha_{j}^{2}}{\lambda_{j}\left(1+\lambda_{j}\right)^{2}}\right)}}
lam.OK (Ozkale and Kaciranlar, 2007; Liu, 2011) \displaystyle{\frac{\sum\limits _{i=1}^{n}\frac{\tilde{e}_{i}}{1-g_{ii}}\left(\frac{\tilde{e}_{i}}{1-h_{1-ii}}-\frac{\hat{e}_{i}}{1-h_{ii}}\right)}{\sum\limits _{i=1}^{n}\left(\frac{\tilde{e}_{i}}{1-g_{ii}}-\frac{\hat{e}_{i}}{1-h_{ii}}\right)^{2}}} with \hat{e}_{i}=y_{i}-\mathbf{x}_{i}^{T}\left(\mathbf{X}^{T}\mathbf{X}-\mathbf{x}_{i}\mathbf{x}_{i}^{T}\right)^{-1}\left(\mathbf{X}^{T}\mathbf{y}-\mathbf{x}_{i}y_{i}\right) and \tilde{e}_{i}=y_{i}-\mathbf{x}_{i}^{T}\left(\mathbf{X}^{T}\mathbf{X}+\mathbf{I}-\mathbf{x}_{i}\mathbf{x}_{i}^{T}\right)^{-1}\left(\mathbf{X}^{T}\mathbf{y}-\mathbf{x}_{i}y_{i}\right) where g_{ii} and h_{ii} are the ith diagonal elements of \mathbf{G}=\mathbf{X}\left(\mathbf{X}^{T}\mathbf{X}+\mathbf{I}\right)^{-1}\mathbf{X}^{T} and \mathbf{H=}\mathbf{X}\left(\mathbf{X}^{T}\mathbf{X}\right)^{-1}\mathbf{X}^{T}, respectively.
lam.GCV This is the \lambda value corresponding to the minimum of the generalized cross-validition (GCV) values. The GCV is computed by \frac{\mathrm{SSR}_{\lambda}}{n-1-\mathrm{trace}\left(\mathbf{H}_{\lambda}\right)} where \mathrm{SSR}_{\lambda} is the residual sum of squares and \mathrm{trace}\left(\mathbf{H}_{\lambda}\right) is the trace of the hat matrix at corresponding value of \lambda from Liu regression.

Value

The return object is the Liu tuning parameter estimates based on the literature.

Author(s)

Murat Genç and Ömer Özbilen

References

Liu, K. (1993). A new class of blased estimate in linear regression. Communications in Statistics-Theory and Methods, 22(2), 393-402. doi:10.1080/03610929308831027.

Liu, X. Q. (2011). Improved Liu estimator in a linear regression model. Journal of Statistical Planning and Inference, 141(1), 189-196. doi:10.1016/j.jspi.2010.05.030.

Ozkale, M. R. and Kaciranlar, S. (2007). A prediction-oriented criterion for choosing the biasing parameter in Liu estimation. Communications in Statistics-Theory and Methods, 36(10), 1889-1903. doi:10.1080/03610920601126522. Imdadullah, M., Aslam, M., and Altaf, S., (2017). liureg: A Comprehensive R Package for the Liu Estimation of Linear Regression Model with Collinear Regressors. The R Journal, 9(2), 232-247.

See Also

liureg(), predict(), summary(), pressliu(), residuals()

Examples

Hitters <- na.omit(Hitters)
X <- model.matrix(Salary ~ ., Hitters)[, -1]
y <- Hitters$Salary
lam <- seq(0, 1, 0.01)
liu.mod <- liureg(X, y, lam)
lamest(liu.mod)

Fit a Liu Regression Coefficients Path

Description

liureg fits coefficients paths for Liu regression models over a grid of values for the regularization (biasing) parameter lambda. The returned object is of class liureg.

Usage

liureg(X, y, lambda = 1, scale = c("ulength", "unormal", "none"), ...)

Arguments

X

The design matrix of features. fastliu standardizes the data and includes an intercept term by default.

y

The response vector.

lambda

User-specified values of lambda. The default value is 1, which corresponds to the least squares estimator. A lambda sequence can be entered to generate multiple models.

scale

Scaling type of the design matrix. "ulength" corresponds to unit-length scaling. In this scaling the scaled design matrix is in the form of a correlation matrix. "unormal" scales the features to have unit variance (using 1/n rather than 1/(n-1) formula). "none" does not make scaling and computations are done on centered features.

...

Not used in this implementation.

Details

The sequence of Liu regression models indexed by the tuning parameter. \lambda are obtained by

\hat{\boldsymbol{\beta}}^{liu}\left(\lambda\right)= \left(\mathbf{X}^{T}\mathbf{X}+\mathbf{I}_{p}\right)^{-1} \left(\mathbf{X}^{T}\mathbf{y}+\lambda\hat{\boldsymbol{\beta}}^{ls}\right),

where \hat{\boldsymbol{\beta}}^{ls} is the ordinary least squares estimator.To obtain the models, the singular value decomposition (SVD) of the matrix \mathbf{X} is used. This SVD is done once and is used to generate all models.

Explanatory variables in the design matrix are always centered before fitting a model in the fastliu package. For scaling, two options are possible: unit-length and unit-normal scaling. In unit-length scaling, the matrix of explanatory variables has correlation form. In unit-normal scaling, the explanatory variables have zero mean and unit variance. Both Coefficient estimates based on the scaled data and in original scale are presented. The intercept of the model is not penalized and computed by \bar{y}-\bar{X}\boldsymbol{\hat{\beta}}_1, where \bar{X} is the row vector of the explanatory variables and \boldsymbol{\hat{\beta}}_1 is computed based on centered design matrix.

The returned liureg object is used for statistical testing of Liu coefficients, plotting method and computing the Liu regression related statistics.

Value

Fitted Liu regression object with the class of liureg

Author(s)

Murat Genç and Ömer Özbilen

See Also

coef(), predict(), summary(), pressliu(), residuals()

Examples

data("Hitters")
Hitters <- na.omit(Hitters)
X <- model.matrix(Salary ~ ., Hitters)[, -1]
y <- Hitters$Salary
lam <- seq(0, 1, 0.05)
liu.mod <- liureg(X, y, lam)

Plot method for liureg objects

Description

Plot method for liureg objects

Usage

## S3 method for class 'liureg'
plot(x, type = c("coefpath", "biasvar", "info"), ...)

Arguments

x

A liureg object.

type

What to plot on the vertical axis. coefpath plots the coefficient path of the Liu regression; biasvar generates a bias-variaance plot, info plots the information criteria corresponding the regularization parameter values.

...

Other graphical parameters to plot.

Value

No return value.

Author(s)

Murat Genç

See Also

liureg(), predict(), summary()

Examples

Hitters <- na.omit(Hitters)
X <- model.matrix(Salary ~ ., Hitters)[, -1]
y <- Hitters$Salary
liu.mod <- liureg(X, y, seq(0, 1, 0.01))

# Liu coefficient paths
plot(liu.mod)

# Bias-variance trade-off
plot(liu.mod, type="biasvar")

Predict method for liureg objects

Description

Predict method for liureg objects

Usage

## S3 method for class 'liureg'
predict(object, newdata, ...)

Arguments

object

A liureg object.

newdata

A data frame of new values for X at which predictions are to be made. Can be a data.frame or a matrix.

...

Not used in this implementation.

Value

Depending on whether the lambda is a scalar or a vector, the predict.liureg function returns a vector or matrix of predictions, respectively.

Author(s)

Murat Genç

See Also

liureg(), predict(), summary(), pressliu(), residuals()

Examples

data("Hitters")
Hitters <- na.omit(Hitters)
X <- model.matrix(Salary ~ ., Hitters)[, -1]
y <- Hitters$Salary
lam <- seq(0, 1, 0.01)
liu.mod <- liureg(X, y, lam)
# Predictions based on original X matrix.
predict(liu.mod)
# Predictions based on newdata. newdata can be a matrix or a data.frame.
predict(liu.mod, newdata=X[1:5, ])

Predicted Residual Sum of Squares (PRESS)

Description

pressliu computes the predicted residual sum of squares (PRESS) based on a Liu regression model.

Usage

pressliu(obj, digits = 5L, ...)

Arguments

obj

A liureg object.

digits

Decimal places in the columns of data frame of PRESS values. Can be an integer of vector of integers.

...

Not used in this implementation.

Details

The PRESS statistic is based on the predicted leave-one-out residual sum of squares. The statistic is computed as {\sum\limits _{i=1}^{n}\left(\frac{\hat{e}_{\lambda i}}{1-h_{1-ii}}-\frac{e_{i}\left(h_{1-ii}-\tilde{\mathbf{H}}_{\lambda-ii}\right)}{\left(1-h_{1-ii}\right)\left(1-h_{ii}\right)}\right)^{2}} where h_{ii} is the ith diagonal element of the hat matrix corresponding to the least squares estimator, h_{1-ii} is the ith diagonal element of the hat matrix of the Liu estimator and e_{\lambda i} is the residual at the specific value of \lambda.

Value

The returned object is a vector of PRESS values computed for each lambda..

Author(s)

Murat Genç, Ömer Özbilen

References

Liu, K. (1993). A new class of blased estimate in linear regression. Communications in Statistics-Theory and Methods, 22(2), 393-402. doi:10.1080/03610929308831027.

Ozkale, M. R. and Kaciranlar, S. (2007). A prediction-oriented criterion for choosing the biasing parameter in Liu estimation. Communications in Statistics-Theory and Methods, 36(10), 1889-1903. doi:10.1080/03610920601126522.

See Also

liureg(), pressliu(), residuals()

Examples

data("Hitters")
Hitters <- na.omit(Hitters)
X <- model.matrix(Salary ~ ., Hitters)[, -1]
y <- Hitters$Salary
lam <- seq(0, 1, 0.01)
liu.mod <- liureg(X, y, lam)
pressliu(liu.mod)

Print method for liureg objects

Description

Prints coefficients paths for Liu regression models over a grid of values for the regularization (biasing) parameter lambda.

Usage

## S3 method for class 'liureg'
print(x, digits = max(3, getOption("digits") - 3), ...)

Arguments

x

An object of class liureg.

digits

Number of decimal places in the coefficients data.frame.

...

Not used in this implementation.

Value

The returned object is a data.frame showing the coefficients path.

Author(s)

Murat Genç

See Also

liureg(), summary(), pressliu(), residuals()

Examples

Hitters <- na.omit(Hitters)
X <- model.matrix(Salary ~ ., Hitters)[, -1]
y <- Hitters$Salary
lam <- seq(0, 1, 0.01)
liu.mod <- liureg(X, y, lam)
print(liu.mod)

Print Method for Liu Regression Statistics

Description

statliu computes the statistics related to the Liu regression.

Usage

## S3 method for class 'statliu'
print(x, digits = 5, ...)

Arguments

x

A statliu object

digits

Number of decimal places in the data frame of Liu regression statistics.

...

Other parameters related to print.

Value

The return object is the statistics relatec to the Liu regression.

Author(s)

Murat Genç

See Also

liureg(), summary(), pressliu(), residuals()

Examples

Hitters <- na.omit(Hitters)
X <- model.matrix(Salary ~ ., Hitters)[, -1]
y <- Hitters$Salary
lam <- seq(0, 1, 0.01)
liu.mod <- liureg(X, y, lam)
stats <- statliu(liu.mod)
print(stats)

Liu Regression Residuals

Description

Liu Regression Residuals

Usage

## S3 method for class 'liureg'
residuals(object, ...)

Arguments

object

An object of class liureg.

...

Not used in this implementation.

Value

The returned object is a vector or matrix whose columns are Liu residuals for each lambda.

Author(s)

Murat Genç

See Also

liureg(), pressliu(), residuals()

Examples

Hitters <- na.omit(Hitters)
X <- model.matrix(Salary ~ ., Hitters)[, -1]
y <- Hitters$Salary
lam <- seq(0, 1, 0.01)
liu.mod <- liureg(X, y, lam)
residuals(liu.mod)

Liu Regression Statistics

Description

statliu computes the statistics related to the Liu regression.

Usage

statliu(obj)

Arguments

obj

An object of class liureg.

Details

EDF (Liu, 1993; Hastie et al., 2009) Effective degrees of freedom, n-\mathrm{trace}\left(2\mathbf{H}_\lambda\right)-\mathbf{H}_\lambda\mathbf{H}_\lambda^T for each \lambda where n is the number of the observations in the design matrix and \mathbf{H}_\lambda is the hat matrix of Liu regression at \lambda.
sigma2 Computed \hat{\sigma}^2 from the Liu regression for each \lambda.
VAR Variance from the Liu regression for each \lambda.
BIAS2 Squared-bias from the Liu regression for each \lambda.
MSE Mean squared error (MSE) from the Liu regression for each \lambda.
FVal F-statistics value from the Liu regression for each \lambda.
GCV Generalized cross-validation (GCV) from the Liu regression for each \lambda. The GCV is computed by \frac{\mathrm{SSR}_{\lambda}}{n-1-\mathrm{trace}\left(\mathbf{H}_{\lambda}\right)} where \mathrm{SSR}_{\lambda} is the residual sum of squares and \mathrm{trace}\left(\mathbf{H}_{\lambda}\right) is the trace of the hat matrix at corresponding value of \lambda from Liu regression.
R2 R-squared from the Liu regression for each \lambda.
AdjR2 Adjusted R-squared from the Liu regression for each \lambda.

Value

The return object is the statistics related to the Liu regression.

Author(s)

Murat Genç

References

Liu, K. (1993). A new class of blased estimate in linear regression. Communications in Statistics-Theory and Methods, 22(2), 393-402. doi:10.1080/03610929308831027.

Hastie, T., Tibshirani, R., Friedman, J. H., Friedman, J. H. (2009). The elements of statistical learning: data mining, inference, and prediction (Vol. 2, pp. 1-758). New York: Springer.

See Also

liureg(), summary(), pressliu(), residuals()

Examples

Hitters <- na.omit(Hitters)
X <- model.matrix(Salary ~ ., Hitters)[, -1]
y <- Hitters$Salary
lam <- seq(0, 1, 0.01)
liu.mod <- liureg(X, y, lam)
stats <- statliu(liu.mod)
print(stats)

Summarizing Liu Regression Fit

Description

summary method for liureg objects.

Usage

## S3 method for class 'liureg'
summary(object, digits, ...)

Arguments

object

An object of class liureg.

digits

Number of decimal places in the data frame of Liu regression statistics.

...

Not used in this implemetation.

Details

summary.liureg produces an object with S3 class summary.liureg. The function returns a list of summary statistics of the Liu regression fit for the grid of regularization parameter \lambda values. Each element of the output list includes:

coefficients A p\times 5 matrix with columns coefficient estimates, scaled coefficient estimates, scaled standard errors, scaled t-values with corresponding p-value.
Statistics Liu related statistics R^2, \textrm{adjusted}-R^2, F-statistics, AIC, BIC and MSE values.

Value

The returned object is a list whose elements are Liu regression coefficient estimates and statistics related to Liu regression.

Author(s)

Murat Genç

See Also

liureg(), coef(), predict(), residuals()

Examples

Hitters <- na.omit(Hitters)
X <- model.matrix(Salary ~ ., Hitters)[, -1]
y <- Hitters$Salary
lam <- seq(0, 1, 0.01)
liu.mod <- liureg(X, y, lam)
summary(liu.mod)