KNN classification in MATLAB - a matrix of confusion and ROC?

Question

I am trying to classify a dataset containing two classes using different classifiers (LDA, SVM, KNN) and would like to compare their performance. I made ROC curves for the LDA, changing the a priori probability.

But how can I do the same for the KNN classifier?

I went through the documentation and found some features:

• Class = knnclassify(Sample, Training, Group, k)
• mdl = ClassificationKNN.fit(X,Y,'NumNeighbors',i,'leaveout','On')

I can run (a) and get the confusion matrix using cross-validation of the residual option, but it is not possible to change the a priori probability of making an ROC?

I have not tried (b) yet, but this creates a model in which you can modify mdl.Prior. But I have no idea how to get the matrix of confusion.

Is there an option I missed, or someone who can explain how to fully use this function to get ROC?

Answer 1

This is really not so, because the conclusion of the k-nn classifier is not an estimate from which a decision is made by a threshold, but only a decision based on a majority of votes.

My suggestion: to determine an estimate based on the class ratio in the neighborhood, and then the threshold value of this estimate to calculate the ROC. In short, the score expresses how well the algorithm is defined; it ranges from -1 (maximum certainty for class -1) to +1 (maximum certainty for class +1).

Example: for k = 6, the estimate

• 1 if all six neighbors have class +1;
• -1 if all six neighbors have class -1;
• 0 if half of the neighbors belong to class +1, and half of neighboring elements belong to class -1.

Once you have calculated this score for each datapoint, you can submit it to the standard ROC function.