CrossValidation (JMSL Numerical Library)

java.lang.Object
- com.imsl.datamining.CrossValidation

All Implemented Interfaces:

Serializable, Cloneable
```
public class CrossValidation
extends Object
implements Serializable, Cloneable
```
Performs V-Fold cross-validation for predictive models. In V-fold cross validation, the training data is partitioned randomly into V approximately equally sized sub-samples. The model is then trained V different times with each of the sub-samples removed in turn. The cross-validated estimate of the risk function is given by

$R^{CV}(d_k)=frac{1}{N}sum^{V}_{v=1} sum_{(x_n,y_n)in{eta_v}}{L(y_n,d^{v}_k(x_n))}$

is the loss incurred when the prediction is for the actual y. The inner summation is over the examples in the test sample held out for each fold.

If the predictive model is an instance of a decision tree, cross-validation is performed on each optimal sub-tree determined by cost-complexity pruning. Let the symbol denote the full training data set, the $v^{th}$ sub-sample. Then use $d^{v}_k$ to indicate the set of predictions corresponding to the k^th optimal sub-tree fitted on the training sample ${eta-eta_v}$ . To select one sub-tree from among the configurations, two criteria are the minimum
$k^* = text{argmin}_k R^{CV}(d_k)$
and the least complicated (smallest sub-tree) which satisfies:
$k^{**} = text{max}(k): R^{CV}(d_k) le R^{CV}(d_{k^*}) + SE(R^{CV}(d_{k}))$
The standard error is approximated by
$SE(R^{CV}(d_{k})) approx sqrt{frac{s^2}{n}}$
with
$s^2 = frac{1}{N} sum^{V}_{v=1}sum_{(x_n,y_n)in{eta_v}}(L(Y_n,d_k^{v}(x_n)- R^{CV}(d_{k}) ))^2$
The summation is over each fold and each learning sub-sample within each fold.

See Also:
Example 1, Serialized Form

Constructor Summary

Constructors
Constructor and Description

CrossValidation(PredictiveModel pm)
Creates a CrossValidation object.

Constructors
Constructor and Description
`CrossValidation(PredictiveModel pm)` Creates a `CrossValidation` object.

Method Summary

Methods
Modifier and Type	Method and Description
`void`	`crossValidate()` Performs V-Fold cross-validation.
`double`	`getCrossValidatedError()` Returns the cross-validated error.
`int`	`getNumberOfSampleFolds()` Returns the number of folds set for the cross validation selection.
`int`	`getNumberOfThreads()` Returns the maximum number of `java.lang.Thread` instances that may be used for parallel processing.
`double[]`	`getRiskStandardErrors()` Returns the estimated standard errors for the risk values.
`double[]`	`getRiskValues()` Returns the vector of risk values.
`void`	`setNumberOfSampleFolds(int nFolds)` Sets the number of folds to use in cross validation selection.
`void`	`setNumberOfThreads(int numberOfThreads)` Sets the maximum number of `java.lang.Thread` instances that may be used for parallel processing.
`void`	`setRandomObject(Random r)` Sets the random object to be used in the permutation of observation data.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - CrossValidation
```
public CrossValidation(PredictiveModel pm)
                throws PredictiveModel.PredictiveModelException
```
    Creates a CrossValidation object.
    
    Parameters:
    pm - an object of a class that extends PredictiveModel.
    
    Throws:
    
    PredictiveModel.PredictiveModelException
- Method Detail
  - crossValidate
```
public void crossValidate()
                   throws PredictiveModel.PredictiveModelException,
                          NoSuchMethodException,
                          InstantiationException,
                          IllegalAccessException,
                          InvocationTargetException
```
    Performs V-Fold cross-validation.
    
    Throws:
    
    PredictiveModel.PredictiveModelException - an exception has occurred in the common PredictiveModel programming interface methods or an exception class that has extended the PredictiveModelException class.
    
    NoSuchMethodException - the PredictiveModel subclass is missing a constructor with the expected signature (see com.imsl.datamining.PredictiveModel.PredictiveModel).
    
    InstantiationException - the object has failed to instantiate. This maybe have occurred if your PredictiveModel subclass is not concrete.
    
    IllegalAccessException - the currently executing method does not have access to the definition of the specified class, field, method or constructor.
    
    InvocationTargetException - an exception has occurred within one of the method. Use java.lang.Throwable.getCause() to extract the initiating exception.
  - getCrossValidatedError
```
public double getCrossValidatedError()
                              throws PredictiveModel.StateChangeException
```
    Returns the cross-validated error. If the response variable is categorical, the error is the misclassification rate, weighted by the misclassification costs and prior probabilities, attributes of the PredictiveModel object. If the response variable is quantitative/continuous, the error is the mean squared prediction error, also weighted if weights are set in the PredictiveModel object.
    
    Returns:
    a double equal to the cross-validated prediction error.
    
    Throws:
    
    PredictiveModel.StateChangeException - an input parameter in the PredictiveModel changed that might affect the model estimates or predictions.
  - getNumberOfSampleFolds
```
public int getNumberOfSampleFolds()
```
    Returns the number of folds set for the cross validation selection.
    
    Returns:
    an int that indicates the number of folds.
  - getNumberOfThreads
```
public int getNumberOfThreads()
```
    Returns the maximum number of java.lang.Thread instances that may be used for parallel processing.
    
    Returns:
    an int containing the maximum number of java.lang.Thread instances that may be used for parallel processing.
    The actual number of threads used in parallel processing will be the lesser of numberOfThreads and nFolds, the number of folds set for cross-validation. This assessment is made to optimize use of resources.
  - getRiskStandardErrors
```
public double[] getRiskStandardErrors()
```
    Returns the estimated standard errors for the risk values.
    In most cases the length is 1. For DecisionTree, CrossValidation returns an array of length >= 1.
    
    Returns:
    a double array containing the estimated standard errors for the risk values.
  - getRiskValues
```
public double[] getRiskValues()
```
    Returns the vector of risk values.
    In most cases the length is 1. For DecisionTree, CrossValidation returns an array of length >= 1.
    
    Returns:
    a double array containing the estimated risk values.
  - setNumberOfSampleFolds
```
public void setNumberOfSampleFolds(int nFolds)
```
    Sets the number of folds to use in cross validation selection.
    
    Parameters:
    nFolds - an int that specifies the number of folds to use in cross validation selection.
    nFolds must be between 1 and the number of observations (xy.length), inclusive. If nFolds = 1 the full data set is used once to generate the PredictiveModel. In other words, no cross-validation is performed. If 1 < xy.length/nFolds 3, leave-one-out cross validation is performed.
    
    Default: nFolds = 10.
  - setNumberOfThreads
```
public void setNumberOfThreads(int numberOfThreads)
```
    Sets the maximum number of java.lang.Thread instances that may be used for parallel processing.
    
    Parameters:
    numberOfThreads - an int specifying the maximum number of java.lang.Thread instances that may be used for parallel processing.
    The actual number of threads used in parallel processing will be the lesser of numberOfThreads and nFolds, the number of folds set for cross-validation. This assessment is made to optimize use of resources.
    
    Default: numberOfThreads = 1.
  - setRandomObject
```
public void setRandomObject(Random r)
```
    Sets the random object to be used in the permutation of observation data.
    
    Parameters:
    r - a Random object to be used in random permutation of observation data.
    Specifying a seed for the Random object can produce repeatable/deterministic output.

Class CrossValidation

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

CrossValidation

Method Detail

crossValidate

getCrossValidatedError

getNumberOfSampleFolds

getNumberOfThreads

getRiskStandardErrors

getRiskValues

setNumberOfSampleFolds

setNumberOfThreads

setRandomObject