naive_bayes_classification

CNL Stat : Data Mining : naive_bayes_classification

Synopsis

Required Arguments

Return Value

Synopsis with Optional Arguments

Classifies unknown patterns using a previously trained Naive Bayes classifier. The classifier is contained in an Imsls_f_nb_classifier data structure, which is optional output from imsls_f_naive_bayes_trainer.

Synopsis

#include <imsls.h>

int *imsls_f_naive_bayes_classification (Imsls_f_nb_classifier *nb_classifier, int n_patterns, …, 0)

The type double function is imsls_d_naive_bayes_classification.

Required Arguments

Imsls_f_nb_classifier *nb_classifier (Input)
Pointer to a structure of the type Imsls_f_nb_classifier from imsls_f_naive_bayes_trainer.

int n_patterns (Input)
Number of patterns to classify.

Return Value

Pointer to an array of size n_patterns containing the predicted classification associated with each input pattern.

Synopsis with Optional Arguments

#include <imsls.h>

int *imsls_f_naive_bayes_classification (Imsls_f_nb_classifier nb_classifier, int n_patterns,

IMSLS_NOMINAL, int nominal[],

IMSLS_CONTINUOUS, float continuous[],

IMSLS_PRINT_LEVEL, int print_level,

IMSLS_USER_PDF, float pdf(),

IMSLS_USER_PDF_WITH_PARMS, float pdf(), void *parms,

IMSLS_PREDICTED_CLASS_PROB, float **pred_class_prob,

IMSLS_PREDICTED_CLASS_PROB_USER, float pred_class_prob[],

IMSLS_RETURN_USER, int classification[],

Optional Arguments

IMSLS_NOMINAL, int nominal[] (Input)
nominal is an array of size n_patterns by nb_classifier->n_nominal containing values for the nominal input attributes. The i-th row contains the nominal input attributes for the i-th pattern. The j-th column of this matrix contains the classifications for the j-th nominal attribute. They must be encoded with integers starting from 0 to nb_classifier->n_categories[i]-1. Any value outside this range is treated as a missing value. If nb_classifier->n_nominal=0, this array is ignored.

IMSLS_CONTINUOUS, float continuous[] (Input)
continuous is an array of size n_patterns by nb_classifier->n_continuous containing values for the continuous input attributes. The i-th row contains the input attributes for the i-th training pattern. The j-th column of this matrix contains the values for the j-th continuous attribute. Missing values should be set equal to imsls_f_machine(6)=NaN. Patterns with missing values are still used to train the classifier unless the IMSLS_IGNORE_MISSING_VALUES option is supplied. If nb_classifier->n_continuous=0, this matrix is ignored.

IMSLS_PRINT_LEVEL, int print_level (Input)
Print levels for printing data warnings and final results. print_level should be set to one of the following values:

print_level	Description
IMSLS_NONE	Printing of data warnings and final results is suppressed.
IMSLS_FINAL	Prints final summary of Naive Bayes classifier training.
IMSLS_DATA_WARNINGS	Prints information about missing values and PDF calculations equal to zero.
IMSLS_TRACE_ALL	Prints final summary plus all data warnings associated with missing values and PDF calculations equal to zero.

Default: IMSLS_NONE.

IMSLS_USER_PDF, float pdf(int index[], float x) (Input)
The user-supplied probability density function and parameters used to calculate the conditional probability density for continuous input attributes is required when the classifier was trained with selected_pdf[i]= IMSLS_USER.

When pdf is called, x will equal continuous[i*n_continuous+j], and index will contain the following values for i, j, and k:

Index	Value
index[0]	i = pattern index
index[1]	j = attribute index
index[2]	k = target classification

The pattern index ranges from 0 to n_patterns-1 and identifies the pattern index for x. The attributes index ranges from 0 to n_categories[i]-1, and k=classification[i].

This argument is ignored if n_continuous = 0. By default the Gaussian PDF is used for calculating the conditional probability densities using either the means and variances calculated from the training patterns or those supplied in IMSLS_GAUSSIAN_PDF.

IMSLS_USER_PDF_WITH_PARMS, float pdf(int index[], float x, void *parms), void *parms (Input)
The user-supplied probability density function and parameters used to calculate the conditional probability density for continuous input attributes is required when selected_pdf[i]= IMSLS_USER. pdf also accepts a pointer to parms supplied by the user. The parameters pointed to by parms are passed to pdf each time it is called. For an explanation of the other arguments, see IMSLS_USER_PDF.

IMSLS_PREDICTED_CLASS_PROB, float **pred_class_prob, (Output)
The address of a pointer to an array of size n_patterns by n_classes, where n_classes is the number of target classifications. The values in the i-th row are the predicted classification probabilities associated with the target classes. pred_class_prob[i*n_classes+j] is the estimated probability that the i-th pattern belongs to the j-th target classes.

IMSLS_PREDICTED_CLASS_PROB_USER, float pred_class_prob[] (Output)
Storage for array pred_class_prob is provided by the user. See IMSLS_PREDICTED_CLASS_PROB for a description.

IMSLS_RETURN_USER, int classification[] (Output)
An array of length n_patterns containing the predicted classifications for each pattern described by the input attributes in nominal and continuous.

Description

Function imsls_f_naive_bayes_classification estimates classification probabilities from a previously trained Naive Bayes classifier. Two arrays are used to describe the values of the nominal and continuous attributes used for calculating these probabilities. The predicted classification returned by this function is the class with the largest estimated classification probability. The classification probability estimates for each pattern can be obtained using the optional argument IMSLS_PREDICTED_CLASS_PROB.

Examples

Example 1

Fisher’s (1936) Iris data is often used for benchmarking classification algorithms. It is one of the IMSL data sets and consists of the following continuous input attributes and classification target:

Continuous Attributes: X0(sepal length), X1(sepal width), X2(petal length), and X3(petal width)

Classification (Iris Type): Setosa, Versicolour or Virginica.

This example trains a Naive Bayes classifier using 150 training patterns from Fisher’s data then classifies ten unknown plants using their sepal and petal measurements.

#include <imsls.h>

#include <stdio.h>

int main(){

int i, j;

int n_patterns =150; /* 150 training patterns */

int n_continuous =4; /* four continuous input attributes */

int n_classes =3; /* three classification categories */

int classification[150], *classErrors, *predictedClass;

float *pred_class_prob, continuous[150*4] ;

float *irisData; /* Fishers Iris Data */

char *classLabel[] = {"Setosa ", "Versicolour", "Virginica "};

char dashes[] = {

"--------------------------------------------------------------"};

Imsls_f_nb_classifier *nb_classifier;

/* irisData[]: The raw data matrix. This is a 2-D matrix with 150

/* rows and 5 columns. The last 4 columns are the

/* continuous input attributes and the 1st column is

/* the classification category (1-3). These data

/* contain no categorical input attributes. */

irisData = imsls_f_data_sets(3,0);

/* Data corrections described in the KDD data mining archive */

irisData[5*34+4] = 0.1;

irisData[5*37+2] = 3.1;

irisData[5*37+3] = 1.5;

/* setup the required input arrays from the data matrix */

for(i=0; i<n_patterns; i++){

classification[i] = (int) irisData[i*5]-1;

for(j=1; j<=n_continuous; j++) {

continuous[i*n_continuous+j-1] = irisData[i*5+j];

}

classErrors = imsls_f_naive_bayes_trainer(

n_patterns, n_classes, classification,

IMSLS_CONTINUOUS, n_continuous, continuous,

IMSLS_NB_CLASSIFIER, &nb_classifier, 0);

printf(" Iris Classification Error Rates\n");

printf("%s\n",dashes);

printf(" Setosa Versicolour Virginica | TOTAL\n");

printf(" %d/%d %d/%d %d/%d | %d/%d\n",

classErrors[0], classErrors[1],

classErrors[2], classErrors[3], classErrors[4],

classErrors[5], classErrors[6], classErrors[7]);

printf("%s\n\n", dashes);

/* CALL NAIVE_BAYES_CLASSIFICATION *************************** */

predictedClass = imsls_f_naive_bayes_classification(

nb_classifier, n_patterns,

IMSLS_CONTINUOUS, continuous,

IMSLS_PREDICTED_CLASS_PROB,

&pred_class_prob, 0);

printf(" PROBABILITIES FOR INCORRECT CLASSIFICATIONS\n",dashes);

printf("\nTRAINING PATTERNS| PREDICTED\t|\n");

printf(" X1 X2 X3 X4 | CLASS\t| CLASS\tP(0) P(1) P(2)|\n");

printf("%s|\n", dashes);

for(i=0; i<n_patterns; i++){

if(classification[i] == predictedClass[i]) continue;

printf(" %4.1f%4.1f%4.1f%4.1f| %s\t| %s\t%4.2f %4.2f %4.2f|\n",

continuous[i*n_continuous], continuous[i*n_continuous+1],

continuous[i*n_continuous+2], continuous[i*n_continuous+3],

classLabel[classification[i]], classLabel[predictedClass[i]],

pred_class_prob[i*n_classes], pred_class_prob[i*n_classes+1],

pred_class_prob[i*n_classes+2]);

}

printf("%s|\n", dashes);

imsls_f_nb_classifier_free(nb_classifier);

}

Output

For Fisher’s data, the Naive Bayes classifier incorrectly classified 6 of the 150 training patterns.

Iris Classification Error Rates

------------------------------------------------------------

Setosa Versicolour Virginica | TOTAL

0/50 3/50 3/50 | 6/150

------------------------------------------------------------

PROBABILITIES FOR INCORRECT CLASSIFICATIONS

TRAINING PATTERNS| PREDICTED |

X1 X2 X3 X4 | CLASS | CLASS P(0) P(1) P(2)|

------------------------------------------------------------|

6.9 3.1 4.9 1.5 | Versicolour | Virginica 0.00 0.46 0.54|

5.9 3.2 4.8 1.8 | Versicolour | Virginica 0.00 0.16 0.84|

6.7 3.0 5.0 1.7 | Versicolour | Virginica 0.00 0.08 0.92|

4.9 2.5 4.5 1.7 | Virginica | Versicolour 0.00 0.97 0.03|

6.0 2.2 5.0 1.5 | Virginica | Versicolour 0.00 0.96 0.04|

6.3 2.8 5.1 1.5 | Virginica | Versicolour 0.00 0.71 0.29|

------------------------------------------------------------|

Example 2

This example uses the spam benchmark data available from the Knowledge Discovery Databases archive maintained at the University of California, Irvine: http://archive.ics.uci.edu/ml/datasets/Spambase.

These data contain of 4601 patterns consisting of 57 continuous attributes and one classification. 41% of these patterns are classified as spam and the remaining as non-spam. The first 54 continuous attributes are word or symbol percentages. That is, they are percents scaled from 0 to 100% representing the percentage of words or characters in the email that contain a particular word or character. The last three continuous attributes are word lengths. For a detailed description of these data visit the KDD archive at the above link.

In this example, percentages are transformed using the arcsin/square root transformation

. The last three attributes, word lengths, are transformed using square roots. Transformed percentages and the first word length attribute are modeled using the Gaussian distribution. The last two word lengths are modeled using the log normal distribution.

#include <imsls.h>

#include <stdlib.h>

#include <stdio.h>

static void printErrorRates(int classification_errors[6],

int n, char *label);

int main(){

int i, j, k;

int condPdfTableLength = 0;

int n_patterns;

int n_variables;

int n_sample = 2000;

int n_classes = 2; /* spam or no spam */

int n_continuous = 57;

int *classErrors = NULL;

int *classification = NULL;

int classSample[2000];

int *predictedClass = NULL;

int *rndSampleIndex = NULL;

int classification_errors[6];

float *continuous, *continuousSample;

char* label1 =

" Trainer from Training Dataset of %d Observations \n";

char* label2 =

" Classifier for Entire Dataset of %d Observations \n";

Imsls_f_nb_classifier *nb_classifier=NULL;

float *spamData;

int n_spam = 0;

spamData = imsls_f_data_sets(11, IMSLS_N_OBSERVATIONS, &n_patterns,

IMSLS_N_VARIABLES, &n_variables, 0);

continuous =

(float*)malloc((n_patterns*n_continuous)*sizeof(float));

continuousSample =

(float*)malloc((n_sample*n_continuous)*sizeof(float));

classification = (int*)malloc(n_patterns*sizeof(int));

/* map continuous attributes into transformed representation */

for(i=0; i<n_patterns; i++){

for(j=0; j<n_continuous; j++) {

if (j < 54 ) {

continuous[i*(n_variables-1)+j] = (float)

asin(sqrt( spamData[i*n_variables+j]/100));

} else {

continuous[i*(n_variables-1)+j] =

spamData[i*n_variables+j];

}

classification[i] = (int)spamData[(i*n_variables)+n_variables-1];

if(classification[i] == 1) n_spam++;

}

printf("Number of Patterns = %d Number Classified as Spam = %d \n\n",

n_patterns, n_spam);

/* select random sample for training Naive Bayes Classifier */

imsls_random_seed_set(1234567);

rndSampleIndex=imsls_random_sample_indices(n_sample, n_patterns, 0);

for(k=0; k<n_sample; k++){

i = rndSampleIndex[k]-1;

classSample[k] = classification[i];

for(j=0; j<n_continuous; j++) {

continuousSample[k*n_continuous+j] =

continuous[i*n_continuous+j];

}

/* Train Naive Bayes Classifier */

classErrors = imsls_f_naive_bayes_trainer(n_sample, n_classes,

classSample,

IMSLS_CONTINUOUS, n_continuous, continuousSample,

IMSLS_NB_CLASSIFIER, &nb_classifier, 0);

/* print error rates for training sample */

printErrorRates(classErrors, n_sample, label1);

/* CALL NAIVE_BAYES_CLASSIFICATION TO CLASSIFIY ENTIRE DATASET */

predictedClass = imsls_f_naive_bayes_classification(nb_classifier,

n_patterns,

IMSLS_CONTINUOUS, continuous, 0);

/* calculate classification error rates for entire dataset */

for(i=0; i<6; i++) classification_errors[i] = 0;

for(i=0; i<n_patterns; i++){

switch (classification[i])

{

case 0:

classification_errors[1]++;

if(classification[i] != predictedClass[i])

classification_errors[0]++;

break;

case 1:

classification_errors[3]++;

if(classification[i] != predictedClass[i])

classification_errors[2]++;

break;

}

classification_errors[5] =

classification_errors[1]+classification_errors[3];

classification_errors[4] =

classification_errors[0]+classification_errors[2];

}

/* print error rates for entire dataset */

printErrorRates(classification_errors, n_patterns, label2);

}

static void printErrorRates(int classification_errors[6],

int n, char *label)

{

double p, p1, p0;

p0 = 100.0*classification_errors[0]/classification_errors[1];

p1 = 100.0*classification_errors[2]/classification_errors[3];

p = 100.0*classification_errors[4]/classification_errors[5];

printf(" Classification Error Rates Reported by\n");

printf(label, n);

printf("----------------------------------------------------\n");

printf(" Not Spam Spam | TOTAL\n");

printf(" %d/%d=%4.1f%% %d/%d=%4.1f%% | %d/%d=%4.1f%%\n",

classification_errors[0], classification_errors[1],

p0, classification_errors[2], classification_errors[3],

p1, classification_errors[4], classification_errors[5], p);

printf("----------------------------------------------------\n\n");

return;

}

Output

It is interesting to note that the classification error rates obtained by training a classifier from a random sample is slightly lower than those obtained from training a classifier with all 4601 patterns. When the classifier is trained using all 4601 patterns, the overall classification error rate was 12.9% (see Example 3 for imsls_f_naive_bayes_trainer). It is 12.4% for a random sample of 2000 patterns.

Number of Patterns = 4601 Number Classified as Spam = 1813

Classification Error Rates Reported by

Trainer from Training Dataset of 2000 Observations

----------------------------------------------------

Not Spam Spam | TOTAL

31/1202= 2.6% 218/798=27.3% | 249/2000=12.4%

----------------------------------------------------

Classification Error Rates Reported by

Classifier for Entire Dataset of 4601 Observations

----------------------------------------------------

Not Spam Spam | TOTAL

81/2788= 2.9% 549/1813=30.3% | 630/4601=13.7%

----------------------------------------------------

Fatal Errors

IMSLS_STOP_USER_FCN

Request from user supplied function to stop algorithm. User flag = "#".