How to calculate alpha if error rate is zero (Adaboost)

Question

I'm wondering what the alpha value (the weight of the weak classifier) ​​should be when it has an error rate (perfect classification), because the alpha algorithm (0.5) * Math.log(((1 - errorRate) / errorRate))

Thanks.

Answer 1

Nominally, the alpha for a weak classifier with zero error should be large , because it correctly classifies all training instances. I assume that you use all the training data to evaluate alpha. Perhaps you only evaluate alpha with a training sample for this round of promotion, and also - in this case, your alpha should be slightly smaller based on the sample size, but the same idea.

In theory, this alpha should be almost infinite if your other alpha is abnormal. In practice, the suggestion to check whether your error is zero and give these alpha values ​​a very high value is reasonable, but the error rate of zero or near zero usually indicates that you are processing (or simply too little training data to evaluate reliable alpha).

This is stated in section 4.2 of the Schapire and Singer Confidence Rated Predictions version of the Adaboost . They suggest adding a small epsilon to your numerator and denominator for stability:

 alpha = (0.5) * Math.log(((1 - errorRate + epsilon) / (errorRate + epsilon))) 

In any case, this alpha should not be set to a small value (it should be large). And setting it to 1 makes sense only if all other alpha for all other promotion rounds are normalized, so the sum of all alpha is almost 1, for example.

Answer 2

If you increase, weighing and passing all the training data to weak students, I would say that you found a weak classifier that is actually strong, because it perfectly classified your data.

In this case, this should happen in the first iteration of Adaboost. Add this weak classifier to your strong classifier with an alpha setting of 1 and stop learning.

Now, if this happened when you boost by re-fetching, and your sample is just a subset of your training data, I believe that you should discard this subset and try again with another sample.

I believe that you have achieved such a result because you are playing with a very simple example, or your training data set is very small or not representative. It is also possible that your weak classifier is too weak and too close to random assumptions.

Answer 3

I encountered this problem several times, and usually I do this to check if the error is 0, and if so, set it to 1/10 of the minimum weight. This is a hack, but it usually ends very well.

Answer 4

In fact, it is better if you do not use such a classifier in your Adaboost prediction, because it will not improve it, because it is not a weak classifier and will tend to eat all the weight.