Locate and optionally replace dependent variable missing values with nearest neighbor estimates.
Synopsis
#include <imsls.h>
int imsls_f_impute_missing (int n_observations, int n_variables, int n_independent, int indind[], float x[], ..., 0)
The type double function is imsls_d_impute_missing.
Required Arguments
int
n_observations (Input)
Number of observations.
int
n_variables (Input)
Number of variables.
int
n_independent (Input)
Number of independent
variables.
int indind[]
(Input)
Array of size n_independent
designating the indices of the columns of x containing the
independent variables.
float x[]
(Input)
Array of size n_observations × n_variables containing
the observations. Missing values of the dependent variables may be imputed
as functions of the independent variables, but if any of the independent
variables have missing values, then imputation will not be performed and a
warning will be issued. If one of the optional arguments, IMSLS_REPLACEMENT_VALUE,
IMSLS_IMPUTE_METHOD,
or IMSLS_PURGE
is supplied, x_imputed
(see optional argument IMSLS_X_IMPUTED)
contains the imputed data on output.
Return Value
The number of missing values (n_miss) in the data array x.
Synopsis with Optional Arguments
#include <imsls.h>
int
imsls_f_impute_missing (int
n_observations,
int n_variables,
int n_independent,
int indind[],
float x[],
IMSLS_MISSING_VALUE, float
mval,
IMSLS_METRIC_DIAG,
float
g[],
IMSLS_REPLACEMENT_VALUE,
float
replacement_value,
or
IMSLS_IMPUTE_METHOD,
int
method,
int
k,
or
IMSLS_PURGE, int
*n_missing_rows,
int
**missing_row_indices,
IMSLS_MISSING_INDEX, int
**indices,
IMSLS_X_IMPUTED,
float
** x_imputed,
IMSLS_X_IMPUTED_USER,
float
x_imputed[],
0)
Optional Arguments
IMSLS_MISSING_VALUE, float mval, (Input)
Scalar value
(other than NaN) representing a missing value. NaN always represents a missing
value, so if mval is not NaN it
will be treated as a second type of missing value.
IMSLS_METRIC_DIAG, float g[]
(Input)
Array of length n_independent defining
a diagonal metric for independent variable space. This scales the independent
variables in the distance calculations used to determine nearest neighbors. The
default measure of distance is Euclidean (g[i] = 1
for all i).
IMSLS_REPLACEMENT_VALUE, float replacement_value
(Input)
Replace missing values in x with replacement_value.
Output data array is returned in x_imputed. Requires
optional argument IMSLS_X_IMPUTED or
IMSLS_X_IMPUTED_USER.
or
IMSLS_IMPUTE_METHOD, int method, int k
(Input)
The method to be used for imputing missing values
using k nearest
neighbors. Replace missing value of dependent variable y at point
x in the space of independent variables with the mode, mean, median,
geometric mean, or linear regression (method) of
y on those k nearest neighbors of
x which have no missing values. To use all of the data and eliminate the
need to compute neighborhoods, set k ≥ n_observations. If
there are no independent variables, set k ≥ n_observations.
Imputed data is returned in x_imputed. Requires
optional argument IMSLS_X_IMPUTED or
IMSLS_X_IMPUTED_USER.
Valid values for method are:
|
method |
Description |
|
IMSLS_MODE_METH |
Mode |
|
IMSLS_MEAN_METH |
Mean |
|
IMSLS_MEDIAN_METH |
Median |
|
IMSLS_GEOMEAN_METH |
Geometric mean |
|
IMSLS_LINEAR_METH |
Linear regression |
or
IMSLS_PURGE, int *n_missing_rows,
int **missing_row_indices
(Output)
All rows with missing values are removed from x and the resulting
data array is returned in x_imputed. n_missing_rows is the
number of rows that were removed. missing_row_indices
are the indices of the rows that were removed. Requires optional argument IMSLS_X_IMPUTED or IMSLS_X_IMPUTED_USER.
IMSLS_MISSING_INDEX, int **indices
(Output)
Address of a pointer to the internally allocated array of
size n_miss,
containing the indices of x where missing values
occur. n_miss is
the function return value. If the data has no missing values, the pointer
is returned as NULL.
IMSLS_X_IMPUTED, float ** x_imputed
(Output)
Array containing imputed data. This argument is required
when IMSLS_REPLACEMENT_VALUE,
IMSLS_IMPUTE_METHOD,
or IMSLS_PURGE
is supplied. For options IMSLS_REPLACEMENT_VALUE
and IMSLS_IMPUTE_METHOD,
x_imputed
contains all data from x with missing values replaced in the dependent variable
columns. For option IMSLS_PURGE, x_imputed is an array
of size n_observations - n_missing_rows × n_variables,
containing the data from x with the rows of
missing data removed.
IMSLS_X_IMPUTED_USER, float x_imputed[]
(Output)
Storage for array x_imputed is provided
by the user. See IMSLS_X_IMPUTED. The
size of this array must be the same as x, n_observations × n_variables. For the
IMSLS_PURGE option, use only the first n_observations - n_missing_rows × n_variables values on output.
Description
Function imsls_f_impute_missing locates missing values, and optionally, replaces them with estimated values. This replacement process, called imputation, applies only to dependent variables. If x denotes an arbitrary point in independent variable space and y denotes a dependent variable with a missing value at x = xi, then y at xi is estimated as y(xi) = f(xi) where f(x) is some function of x in some neighborhood of xi. imsls_f_impute_missing provides five options (see IMSLS_IMPUTE_METHOD) for the form of f(x), and each option allows neighborhood size to be specified in terms of some given number of nearest neighbors. The neighbors exclude observations with missing values and are determined by distance, the norm relative to metric G,
By default, G = I, but the IMSLS_METRIC_DIAG option can be used to specify any other diagonal metric G. A sixth option, IMSLS_REPLACEMENT_VALUE, allows the user to specify one value to be used as a replacement for all missing values.
Instead of being used for imputation, imsls_f_impute_missing can be used to simply remove all observations which contain missing values. This is accomplished with the option IMSLS_PURGE. With this option, all rows with missing values are removed from the input data matrix. Unlike imputation, this option is not limited to dependent variables and can be used to handle missing values in the independent variables.
Usually either imputation or deletion will be performed, but imsls_f_impute_missing can be used for the more basic task of returning the indices of missing values. The indices could then be used to implement other imputation methods.
Following the standard practice, missing data values are always represented by NaN. Option IMSLS_MISSING_VALUE allows the user to also specify a second value to represent missing values.
Examples
Example 1
Count the missing values in a data set, where the only valid missing value is NaN.
#include <imsls.h>
#include <stdio.h>
#define N_OBSERVATIONS 20
#define N_VARIABLES 4
int main()
{
float x[N_OBSERVATIONS][N_VARIABLES];
int count, i, j;
/* create the test data */
for(i=0;i<N_OBSERVATIONS;i++) {
for(j=0;j<N_VARIABLES;j++) {
x[i][j]= (float)((i*N_VARIABLES)+j);
}
}
/* replace some of the data values */
x[3][1] = imsls_f_machine(6); /* NaN */
x[5][2] = imsls_f_machine(6); /* NaN */
x[7][2] = imsls_f_machine(7); /* positive infinity */
x[9][3] = imsls_f_machine(8); /* negative infinity */
/* declare no independent variables */
/* note +/-inf are not considered 'missing' */
count = imsls_f_impute_missing (N_OBSERVATIONS, N_VARIABLES, 0,
NULL, (float*)x,
0);
printf("number of missing values = %d\n", count);
}
Output
number of missing values = 2
Example 2
Set the value 20 to represent a missing value and find the indices in x which contain the missing value.
#include <imsls.h>
#include <stdio.h>
#define N_OBSERVATIONS 20
#define N_VARIABLES 4
int main()
{
float x[N_OBSERVATIONS][N_VARIABLES];
float mval;
int n_independent, count, i, j;
int indind[2];
int *indices;
/* declare 2 independent variables */
n_independent = 2;
indind[0] = 2; /* declare that column 2 is independent */
indind[1] = 3; /* declare that column 3 is independent */
/* missing value is represented by 20 */
/* and will be located at x[5][0] */
mval = 20.0;
/* create the test data */
for(i=0;i<N_OBSERVATIONS;i++) {
for(j=0;j<N_VARIABLES;j++) {
x[i][j] = (float)((i*N_VARIABLES)+j);
}
}
count = imsls_f_impute_missing (N_OBSERVATIONS, N_VARIABLES,
n_independent, indind, (float*)x,
IMSLS_MISSING_VALUE, mval,
IMSLS_MISSING_INDEX, &indices,
0);
printf("number of missing values = %d\n", count);
for (i=0; i<count;i++) {
printf("indices[%d] = %d\n", i, indices[i]);
}
}
Output
number of missing values = 1
indices[0] = 20
Example 3
In this example both NaN and infinity represent missing values in the original data. In the first call to imsls_f_impute_missing, missing values are replaced by negative infinity. In the second call to imsls_f_impute_missing, negative infinity is set to represent missing values and the rows containing the missing values are purged for the final output.
#include <imsls.h>
#include <stdio.h>
#define N_ROWS 6
#define N_COLS 4
int main()
{
float *x_imputed, *x_purged;
float mval, replacement_value;
float data[N_ROWS][N_COLS];
int n_independent, count, npurge, i, j;
int indind[1];
int *bad_obs;
char *fmt="%6.2f";
/* create the test data */
for(i=0;i<N_ROWS;i++) {
for(j=0;j<N_COLS;j++) {
data[i][j] = (float)((i*N_COLS)+j);
}
}
/* insert bad values into data */
data[1][1] = imsls_f_machine(6); /* NaN */
data[2][2] = imsls_f_machine(7); /* positive infinity */
data[3][3] = imsls_f_machine(8); /* negative infinity */
imsls_f_write_matrix ("Original data with missing values",
N_ROWS, N_COLS, (float*)data,
IMSLS_WRITE_FORMAT, fmt,
0);
/* set the missing value to be +inf */
mval = imsls_f_machine(7);
/* replace missing values with neg inf */
replacement_value = imsls_f_machine(8);
/* declare one independent variable */
n_independent = 1;
indind[0] = 0;
/* replace Nan and +inf values with -inf */
count = imsls_f_impute_missing (N_ROWS, N_COLS, n_independent,
indind, (float*)data,
IMSLS_MISSING_VALUE, mval,
IMSLS_REPLACEMENT_VALUE, replacement_value,
IMSLS_X_IMPUTED, &x_imputed,
0);
imsls_f_write_matrix ("Data with values replaced",
N_ROWS, N_COLS, x_imputed,
IMSLS_WRITE_FORMAT, fmt,
0);
/* now purge all rows containing -inf */
mval = imsls_f_machine(8);
count = imsls_f_impute_missing (N_ROWS, N_COLS, n_independent,
indind, x_imputed,
IMSLS_MISSING_VALUE, mval,
IMSLS_PURGE, &npurge, &bad_obs,
IMSLS_X_IMPUTED, &x_purged,
0);
printf("\n number missing = %d, number of rows purged = %d\n",
count, npurge);
printf("\n Purged row numbers:\n");
for(i=0;i<npurge;i++) {
printf(" %d ",bad_obs[i]);
}
printf ("\n");
imsls_f_write_matrix ("New data with bad rows purged",
N_ROWS-npurge, N_COLS, x_purged,
IMSLS_WRITE_FORMAT, fmt,
0);
}
Output
Original data with missing values
1 2 3 4
1 0.00 1.00 2.00 3.00
2 4.00 ...... 6.00 7.00
3 8.00 9.00 ++++++ 11.00
4 12.00 13.00 14.00 ------
5 16.00 17.00 18.00 19.00
6 20.00 21.00 22.00 23.00
Data with values replaced
1 2 3 4
1 0.00 1.00 2.00 3.00
2 4.00 ------ 6.00 7.00
3 8.00 9.00 ------ 11.00
4 12.00 13.00 14.00 ------
5 16.00 17.00 18.00 19.00
6 20.00 21.00 22.00 23.00
number missing = 3, number of rows purged = 3
Purged row numbers:
1 2 3
New data with bad rows purged
1 2 3 4
1 0.00 1.00 2.00 3.00
2 16.00 17.00 18.00 19.00
3 20.00 21.00 22.00 23.00
Example 4
Replace missing values computed using the mean of the 3 nearest neighbors.
#include <imsls.h>
#include <stdio.h>
#define N_ROWS 10
#define N_COLS 4
int main()
{
float *x_imputed;
float data[N_ROWS][N_COLS];
int n_independent, count, i, j;
int indind[2];
char *fmt="%6.2f";
/* create the test data */
for(i=0;i<N_ROWS;i++) {
for(j=0;j<N_COLS;j++) {
data[i][j] = (float)((i*N_COLS)+j);
}
}
data[1][3] = imsls_f_machine(6); /* insert NaN at row 1 col 3 */
data[4][2] = imsls_f_machine(6); /* insert NaN at row 4 col 2 */
imsls_f_write_matrix ("Original data with missing values",
N_ROWS, N_COLS, (float*) data,
IMSLS_WRITE_FORMAT, fmt,
0);
/* declare two independent variables */
n_independent = 2;
indind[0] = 0;
indind[1] = 1;
/* replace missing values using mean method */
count = imsls_f_impute_missing (N_ROWS, N_COLS, n_independent, indind,
(float*)data, IMSLS_IMPUTE_METHOD,
IMSLS_MEAN_METH, 3,
IMSLS_X_IMPUTED, &x_imputed,
0);
imsls_f_write_matrix ("Imputed data (using mean method)",
N_ROWS, N_COLS,
x_imputed, IMSLS_WRITE_FORMAT, fmt,
0);
}
Output
Original data with missing values
1 2 3 4
1 0.00 1.00 2.00 3.00
2 4.00 5.00 6.00 ......
3 8.00 9.00 10.00 11.00
4 12.00 13.00 14.00 15.00
5 16.00 17.00 ...... 19.00
6 20.00 21.00 22.00 23.00
7 24.00 25.00 26.00 27.00
8 28.00 29.00 30.00 31.00
9 32.00 33.00 34.00 35.00
10 36.00 37.00 38.00 39.00
Imputed data (using mean method)
1 2 3 4
1 0.00 1.00 2.00 3.00
2 4.00 5.00 6.00 9.67
3 8.00 9.00 10.00 11.00
4 12.00 13.00 14.00 15.00
5 16.00 17.00 20.67 19.00
6 20.00 21.00 22.00 23.00
7 24.00 25.00 26.00 27.00
8 28.00 29.00 30.00 31.00
9 32.00 33.00 34.00 35.00
10 36.00 37.00 38.00 39.00
Warning Errors
|
IMSLS_NO_GOOD_ROW |
Each row contains missing values. No imputation is performed. |
|
IMSLS_INDEP_HAS_MISSING |
At least one of the independent variables contains a missing value. No imputation is performed.
|
