Caret - different results using train (), pred () and resamples ()

Question

I use Caret to analyze different models, and I evaluate the results using:

• print () [print train results ()],
• pred () and
• resample ().

Why are these results different in the following example?

I am interested in sensitivity (true positives). Why is J48_fit evaluated as .71 sensitivity, then .81, then .71 again

The same thing happens when I launch other models - sensitivity varies depending on the assessment.

NB: here I included two models to illustrate the resamples () function, which should take two models as input, but my main question is the differences between the results depending on which method is used.

In other words, what is the difference between the result of the train () command (C5.0_fit / J48_fit), pred () and resamples ()? What happens behind the scenes and what result should I trust?

Example:

library(C50)
data(churn)

Seed <- 10

# Set train options
set.seed(Seed)
Train_options <- trainControl(method = "cv", number = 10,
                              classProbs = TRUE,
                              summaryFunction = twoClassSummary)

# C5.0 model:
set.seed(Seed)
C5.0_fit <- train(churn~., data=churnTrain, method="C5.0", metric="ROC",
                 trControl=Train_options)

# J48 model:
set.seed(Seed)
J48_fit <- train(churn~., data=churnTrain, method="J48", metric="ROC",
                 trControl=Train_options)
# Get results by printing the outcome
print(J48_fit)

#                      ROC Sens Spec
# Best (sensitivity): 0.87 0.71 0.98  

# Get results using predict()
set.seed(Seed)
J48_fit_predict <- predict(J48_fit, churnTrain)
confusionMatrix(J48_fit_predict, churnTrain$churn)
#             Reference
# Prediction  yes   no
#       yes  389    14
#       no    94  2836
# Sens : 0.81          
# Spec : 0.99

# Get results by comparing algorithms with resamples()
set.seed(Seed)
results <- resamples(list(C5.0_fit=C5.0_fit, J48_fit=J48_fit))
summary(results)
# ROC         mean
# C5.0_fit    0.92  
# J48_fit     0.87
# Sens        mean
# C5.0_fit    0.76  
# J48_fit     0.71
# Spec        mean
# C5.0_fit    0.99  
# J48_fit     0.98

By the way, here is a function to get all three results:

Get_results <- function(...){

  Args <- list(...)
  Model_names <- as.list(sapply(substitute({...})[-1], deparse))

  message("Model names:")
  print(Model_names)

  # Function for getting max sensitivity
  Max_sens <- function(df, colname = "results"){
    df <- df[[colname]]
    new_df <- df[which.max(df$Sens), ]
    x <- sapply(new_df, is.numeric)
    new_df[, x] <- round(new_df[, x], 2)
    new_df
  }

  # Find max Sens for each model
  message("Max sensitivity from model printout:")
  Max_sens_out <- lapply(Args, Max_sens)
  names(Max_sens_out) <- Model_names
  print(Max_sens_out)

  # Find predict() result for each model
  message("Results using predict():")
  set.seed(Seed)
  Predict_out <- lapply(Args, function(x) predict(x, churnTrain))
  Predict_results <- lapply(Predict_out, function(x) confusionMatrix(x, churnTrain$churn))
  names(Predict_results) <- Model_names
  print(Predict_results)

  # Find resamples() results for each model

  message("Results using resamples():")
  set.seed(Seed)
  results <- resamples(list(...),modelNames = Model_names)
  # names(results) <- Model_names
  summary(results)

}

# Test
Get_results(C5.0_fit, J48_fit)

Many thanks!

Answer 1

The best sensitivity you printed is the average model performance in each of 10 folds (from your CV). You can see the performance for each crease J48_fit$resample. Then, to confirm, you can take the average of the first column, ROC, s mean(J48_fit$resample[,1]), and you get 0.865799.

predict() , , , - , 10 % .