AUC Calculation Using Vowpal Wabbit

Question

Do I need to calculate AUC in Vowpal Wabbit at all?

One of the reasons I use Vowpal Wabbit is the large data file size. I can calculate the AUC outside the Wowpal Wabbit environment using the Vowpal Wabbit output, but this can be problematic if the data file is large.

Answer 1

Currently, VW cannot report AUC . Even worse, it cannot optimize directly for AUC. Optimization for AUC is not compatible with online learning, but there are some AUC approximations suitable for optimization .

As for your question, you do not need to store the intermediate file with raw predictions on disk. You can directly transfer it to an external evaluation tool ( perf ):

vw -d test.data -t -i model.vw -r /dev/stdout | perf -roc -files gold /dev/stdin 

Edit: John Langford confirmed that AUC can generally be optimized by changing the ratio of false to false negative losses. In VW, this means setting a different weight of importance for positive and negative examples. You need to set the optimal weight with a downtime set (or cross-validation or progressive loss of validation for one-pass training).