Evaluating Model Performance

This section focuses on evaluating the model using the testing subset, which consists of data that was not used during training or validation.

The AUC value in the first table, located in row zero, is 0.94. This is slightly lower than the AUC based on the training subset 0.95, which is expected. Achieving the same performance on test data as on training data is generally challenging in practice.

The second table below displays the entries of the Confusion Matrix, which helps assess the performance of a classification model by comparing predicted values to actual values [1]. Row zero represents 'True Negatives', where the 'ACTUAL_CLASS' was correctly predicted as 'No' (negative) by the 'PREDICTED_CLASS'. There are 2,538 'True Negative' cases.

Row 1 in the output table below refers to 'False Positives'. False Positives refer to cases that were incorrectly predicted as 'Positive' but are actually 'Negative' - there are only 14 such cases. Row 2 refers to 'False Negatives' (212), where cases were incorrectly predicted as 'Negative' but are actually 'Positive'. Finally, row three represents 'True Positives', where cases were correctly predicted as 'Positive', with 103 instances.

123456789
# Test model generalization using the test data subset (not used during training)

scorepredictions, scorestats, scorecm, scoremetrics = hgbc.score(data=df_test , key= 'EMPLOYEE_ID', label='FLIGHT_RISK', 
                                                                 ntiles=20, impute=True,
                                                                 thread_ratio=1.0)
#display(hgbc.runtime)
display(scorestats.sort('CLASS_NAME').collect())
display(scorecm.filter('COUNT != 0').collect())
#display(scoremetrics.collect())