scaleFilter¶

Scales or unscales continuous data prior to its use in neural network training, testing, or forecasting.

Synopsis¶

scaleFilter (x, method)

Required Arguments¶

float x[] (Input): An array of length nPatterns. The values in x are either the scaled or unscaled values of a continuous variable. Missing values are allowed, and are indicated by placing a NaN (not a number) in x. See machine(6).
int method (Input): The scaling method to apply to each variable. The association of the value in method and the scaling algorithm is summarized in the table below. The sign of method determines whether the values in x are scaled or unscaled. If method is positive then values in x are scaled. If method is negative then values in x are unscaled.

`method`	Algorithm
0	No scaling.
±1	Bounded scaling and unscaling.
±2	Unbounded z-score scaling using the mean and standard deviation.
±3	Unbounded z-score scaling using the median and mean absolute difference.
±4	Bounded z-score scaling using the mean and standard deviation.
±5	Bounded z-score scaling using the median mean absolute difference.

Return Value¶

An array of length nPatterns containing either the scaled or unscaled value of x, depending upon whether method is positive or negative, respectively. If errors are encountered, None is returned.

Optional Arguments¶

scaleLimits, float realMin, float realMax, float targetMin, float targetMax (Input): The real and target limits for x. This optional argument is required when bounded scaling is performed, i.e., method=±1, ±4, or ±5. realMin is the lowest value expected for each input variable in x. realMax is the largest value expected. targetMin is lowest value allowed for the output variable, z. targetMax is the largest value allowed for the output variable.
supplyCenterSpread, float center, float spread (Input): The values center and spread are only used for z-score scaling or unscaling of x, that is, when method is one of ±2, ±3, ±4, and ±5. The value of center is either the mean or median, and the value of spread is either the standard deviation or mean absolute difference. When method is positive, this optional argument can be used to supply a user-defined center and spread rather than allowing scaleFilter to compute the center and spread from the data in x. When method is one of -2, -3, -4, or -5, this optional argument must be used to supply the center and spread used during scaling.
returnCenterSpread, center, spread (Output): Pointers to scalars containing the computed center and spread of x. The values center and spread are only used for z-score scaling or unscaling of x. These methods, ±2, ±3, ±4, and ±5, require two numbers, either the mean or median, and either the standard deviation, or mean absolute difference. The value of center is either the mean or median for x. The value of spread is either the standard deviation or mean absolute difference.

Description¶

The function scaleFilter is designed to either scale or unscale a continuous variable using one of four methods prior to their use as neural network input or output.

The specific encoding computations employed are specified by argument method. Scaling limits are supplied with the optional argument scaleLimits, and are required for the bounded scaling methods, i.e., method=±1, ±4, or ±5. Bounded scaling ensures that the scaled values in the returned array fall between a lower and upper bound.

If method=1 then the bounded method of scaling and unscaling is applied to x using the scaling limits in scaleLimits.

If method=±2, ±3, ±4, or ±5, then the z-score method of scaling is used. These calculations are based upon the following scaling calculation:

\[\mathtt{z}[i] = \frac{(\mathtt{x}[i] - a)}{b}\]

where a is a measure of center for x, and b is a measure of the spread of x.

If method=±2 or ±4, then by default a and b are the arithmetic average and sample standard deviation of the training data. These values can be overridden using the optional argument supplyCenterSpread.

If method=±3 or ±5, then by default a and b are the median and \(\tilde{s}\), where \(\tilde{s}\) is a robust estimate of the population standard deviation:

\[\tilde{s} = \frac{\mathit{MAD}}{0.6745}\]

where MAD is the Mean Absolute Deviation

\[\mathit{MAD} = \mathit{median} \left\{\left|x_j - \mathit{median}\{x\}\right|\right\}\]

Again, the values of a and b can be overridden using the optional argument supplyCenterSpread.

Method ±1: Bounded Scaling and Unscaling¶

If method=1, then the optional argument scaleLimits is required and a scaling operation is conducted using the scale limits for x using the following calculation:

\[\mathrm{z}[i] = r(\mathrm{x}[i] - \mathrm{realMin}) + \mathrm{targetMin}\]

where

\[r = \frac{\mathrm{targetMax} - \mathrm{targetMin}}{\mathrm{realMax} - \mathrm{realMin}}\]

If method=-1, then optional argument scaleLimits is required and an unscaling operation is conducted by inverting the following calculation:

\[\mathrm{x}[i] = \frac{\mathrm{z}[i] - \mathrm{targetMin}}{r} + \mathrm{realMin}\]

Method +2 or +3: Unbounded z-score Scaling¶

If method=2 or method=3, then a scaling operation is conducted using the scale limits of x using a z-score calculation:

\[\mathtt{z}[i] = \frac{\mathtt{x}[i] - \mathtt{center}}{\mathtt{spread}}\]

If either center or spread are missing, (a NaN), then appropriate values are calculated from the non-missing values of x. If method=2, then center is set equal to the arithmetic average \(\bar{x}\), and spread is set equal to the sample standard deviation, \(s\).

If method=3, then center is set equal to the median \(\tilde{m}\), and spread is set equal to the Mean Absolute Difference (MAD).

Method -2 or -3: Unbounded z-score Unscaling¶

If method=-2 or method=-3, then an unscaling operation is conducted using the inverse calculation for the equation shown in the above section, “Method +2 or +3: Unbounded z-score Scaling.”

\[\mathtt{x}[i] = \mathtt{spread} \cdot \mathtt{z}[i] + \mathtt{center}\]

For these values of method, missing values for center and spread are not allowed. If method=-2, then center and spread are assumed to be equal to the arithmetic average and standard deviation, respectively. These values would normally be the same used in scaling the variable with method=+2. If method= -3, then center and spread are assumed to be equal to the median and mean absolute difference, respectively. These values would normally be the same used in scaling the variable with method=+3.

Method +4 or +5: Bounded z-score Scaling¶

This method is essentially the same as the z-score calculation described for method=+2 and method=+3 with additional scaling or unscaling using the scale limits. If method=4, then the optional argument scaleLimits is required and a scaling operation is conducted using the scale limits for x using the widely known z-score calculation:

\[\mathrm{z}[i] = \frac{r \cdot (\mathrm{x}[i] - \mathrm{center})}{\mathrm{spread}} - r \cdot \mathrm{realMin} + \mathrm{targetMin}\]

If either center or spread are missing, (a NaN), then appropriate values are calculated from the non-missing values in x. If center is missing and method=+4, then center is set equal to the arithmetic average \(\bar{x}\), and spread is set equal to the Sample Standard Deviation, \(s\). If center is missing and method=+5, then center is set equal to the median \(\tilde{m}\), and spread is set equal to the MAD.

In bounded scaling, if z[i] exceeds its bounds, it is set to the boundary it exceeded.

Method -4 or -5: Bounded z-score unscaling¶

If method=-4 or method=-5, then the optional argument scaleLimits is required and an unscaling operation is conducted using the inverse calculation for the equation below.

\[\mathrm{x}[i] = \frac{\mathrm{spread} \cdot (\mathrm{z}[i] - \mathrm{targetMin})}{r} + \mathrm{spread}\cdot \mathrm{realMin} + \mathrm{center}\]

For these values of method, missing values for center and spread are not allowed. If method=-4, then center and spread are assumed to be equal to the arithmetic average and standard deviation, respectively. These values would normally be the same used in scaling x with method=+4. If method=-5, then center and spread are assumed to be equal to the median and mean absolute difference, respectively. These values would normally be the same used in scaling the x with method=+5.

Example¶

In this example two data sets are filtered using bounded z-score scaling.

from __future__ import print_function
from numpy import *
from pyimsl.stat.scaleFilter import scaleFilter
from pyimsl.stat.writeMatrix import writeMatrix

x1 = [3.5, 2.4, 4.4, 5.6, 1.1]
x2 = [3.1, 1.5, -1.5, 2.4, 4.2]
centerSpread1 = {}
centerSpread2 = {}

z1 = scaleFilter(x1, 4, scaleLimits={'realMin': -6., 'realMax': 6., 'targetMin': -3., 'targetMax': 3.},
                 returnCenterSpread=centerSpread1)
z2 = scaleFilter(x2, 5, scaleLimits={'realMin': -3., 'realMax': 3., 'targetMin': -3., 'targetMax': 3.},
                 returnCenterSpread=centerSpread2)

writeMatrix("z1", z1, column=True)
print("Center = %10.6f\nSpread = %10.6f" %
      (centerSpread1['center'], centerSpread1['spread']))
writeMatrix("z2", z2, column=True)
print("Center = %10.6f\nSpread = %10.6f" %
      (centerSpread2['center'], centerSpread2['spread']))

# un-scale z1 and z2.
y1 = scaleFilter(z1, -4, scaleLimits={'realMin': -6., 'realMax': 6., 'targetMin': -3., 'targetMax': 3.},
                 supplyCenterSpread=centerSpread1)
y2 = scaleFilter(z2, -5, scaleLimits={'realMin': -3., 'realMax': 3., 'targetMin': -3., 'targetMax': 3.},
                 supplyCenterSpread=centerSpread2)

writeMatrix("y1", y1, column=True)
writeMatrix("y2", y2, column=True)

Output¶

Center =   3.400000
Spread =   1.742125
Center =   2.400000
Spread =   1.334342
 
      z1
     0.0287
    -0.2870
     0.2870
     0.6314
    -0.6601
 
      z2
      0.525
     -0.674
     -2.923
      0.000
      1.349
 
      y1
        3.5
        2.4
        4.4
        5.6
        1.1
 
      y2
        3.1
        1.5
       -1.5
        2.4
        4.2