How to calculate roc curves?

Question

I am writing a classifier (Gaussian mixture model) to classify five human actions. For each observation, the classifier calculates the posterior probability to belong to the cluster.

I want to evaluate the performance of my system, parameterized by a threshold, with values ​​from 0 to 100. For each threshold value for each observation, if the probability of belonging to one of the clusters is greater than the threshold, I accept the result, otherwise I discard it.

For each threshold, I calculate the number true-positive, true-negative, false-positive, false-negative.

How I calculated two functions: sensitivity and specificity as

sensitivity = TP/(TP+FN); specificity=TN/(TN+FP); 

In Matlab:

 plot(1-specificity,sensitivity); 

to have a ROC curve. But the result is not what I expect.

This is a graph of the functions of faults, errors, corrections, sensitivity, and specificity that change the threshold of one action.

This is a graph of the ROC curve of one action

This is the basis of the ROC curve for the same action.

I am wrong, but I do not know where. Perhaps I am mistaken in calculating FP, FN, TP, TN, especially when the result of the classifier is insignificant from the threshold, so I have garbage. What do I need to increase when there is garbage?

Answer 1

Background

I answer this because I need to work with content, and such a question is a great excuse. Thank you for the good opportunity.

I use data from the built-in fish data: http://archive.ics.uci.edu/ml/datasets/Iris

I also use code snippets from the Mathworks tutorial on classification, and for plotroc

• http://www.mathworks.com/products/demos/statistics/classdemo.html
• http://www.mathworks.com/help/nnet/ref/plotroc.html?searchHighlight=plotroc

Description of the problem

In the region there is a clearer border for the classification of "setosa", but there is a match for "versicoloir" vs. "virginica". This is a two-dimensional plot, and some other data has been discarded to produce it. In this case, the ambiguity within the boundaries of the classification is useful.

 %load data load fisheriris %show raw data figure(1); clf gscatter(meas(:,1), meas(:,2), species,'rgb','osd'); xlabel('Sepal length'); ylabel('Sepal width'); axis equal axis tight title('Raw Data') 

Analysis

Suppose we want to define the boundaries for a linear classifier that defines "virginica" versus "non-virginica". We could look at “I versus not-I” for other classes, but they would have their own

So, now we make some linear discriminants and build ROC for them:

 %load data load fisheriris load iris_dataset irisInputs=meas(:,1:2)'; irisTargets=irisTargets(3,:); ldaClass1 = classify(meas(:,1:2),meas(:,1:2),irisTargets,'linear')'; ldaClass2 = classify(meas(:,1:2),meas(:,1:2),irisTargets,'diaglinear')'; ldaClass3 = classify(meas(:,1:2),meas(:,1:2),irisTargets,'quadratic')'; ldaClass4 = classify(meas(:,1:2),meas(:,1:2),irisTargets,'diagquadratic')'; ldaClass5 = classify(meas(:,1:2),meas(:,1:2),irisTargets,'mahalanobis')'; myinput=repmat(irisTargets,5,1); myoutput=[ldaClass1;ldaClass2;ldaClass3;ldaClass4;ldaClass5]; whos plotroc(myinput,myoutput) 

The result is shown below, although it required the removal of duplicate copies of the diagonal:

In the code that I add "myinput" and "myoutput", you can mark them as inputs to the "plotroc" function. You should take the results of your classifier as goals and evidence, and you can get similar results. This compares the actual result of your classifier with the ideal output of your target values. This is a contribution to plotroc.

So, this will give you a “built-in” ROC, which is useful for quick work, but does not force you to study each step in detail.

Questions you may ask at this point include:

• which classifier is better? How to determine what is best in this case?
• What is the convex hull of classifiers? Is there any mixture of classifiers that is more informative than any pure method? Is a bag possible?

Answer 2

You are trying to draw accuracy and recall curves depending on the threshold parameter of the classifier. Definition of accuracy and recall:

 Precision = TP/(TP+FP) Recall = TP/(TP+FN) 

You can check the definition of these parameters at: http://en.wikipedia.org/wiki/Precision_and_recall

There are some curves here: http://www.cs.cornell.edu/courses/cs578/2003fa/performance_measures.pdf

Do you share your data set in your training set, cross-validation set, and test set? (if you don’t share the data, it’s normal that your exact recall curve seems weird)

EDITED: I think there are two possible sources for your problem:

• When you train the classifier for 5 classes, you usually have to train 5 distinctive classifiers. One classifier for (class A = class 1, class B = class 2, 3, 4 or 5), then a second class for class (class A = class 2, class B = class 1, 3, 4 or 5), .. and fifth for class A = class 5, class B = class 1, 2, 3 or 4).

As you said, in order to choose the output for your “composite” classifier, you must transfer your new (test) dataset through five classifiers, and you select the one that has the highest probability.

Then you should have 5 threshold values ​​to determine the weight values ​​that my priorities choose for one classifier over others. You should check how the matlab implementation uses thresholds, but their effect is that you are not choosing a class with a higher probability, but a class with a better weighted probability.

2. As you say, you may not be very good at calculating TP, TN, FP, FN. Your test data must have data that belongs to all classes. Then you have testdata (i, :) and classtestdata (i) - this is a function vector and a class of "true truth" datapoint i. When you evaluate the classifier, you get classifierOutput (i) = 1 or 2 or 3 or 4 or 5. Then you have to calculate the “confusion matrix”, which is a way to calculate TP, TN, FP, FN when you have multiple classes ( > 2): http://en.wikipedia.org/wiki/Confusion_matrix http://www.mathworks.com/help/stats/confusionmat.html (note the relationship between TP, TN, FP, FN, which you calculate for a multiclass problem)

I think you can get the TP, TN, FP, FN data of each subclass (remember that you compute 5 separate classifiers, even if you don't understand it) from the confusion matrix. I'm not sure, but you can draw an accuracy response curve for each subclassifier.

Also check out these slides: http://www.slideserve.com/MikeCarlo/multi-class-and-structured-classification

I don’t know what the ROC curve is, I will test it because machine learning is really interesting for me.

Hope this helps,