Well, firstly, you need to know what color space you are using. Just a little guide on color spaces in OpenCV for Mat type CV_8UC3 . (Images from Wikipedia)
Hsv

In the HSV color space (hue, saturation, value), H gives the color the dominant color, S the color saturation, V the lightness. In OpenCV, the ranges are different. S, V are in [0.255], and H in [0, 180]. Usually, H is in the range [0.360] (full circle), but in order to fit in a byte (256 different values), its value is halved.
In the HSV space, it is easier to separate one color, since you can simply set the correct range for H and just make sure that S is not too small (it will be almost white) and V is not too small (it will be dark).
So, for example, if you need almost blue colors, you need H to be around the value 120 (for example, in [110, 130]), and S, V is not too small (for example, in [100,255]).
White is not a hue (the rainbow does not have white in it), but is a combination of color.
In HSV you need to take the whole range of H (H at [0, 180]), very small S values (for example, S in [0, 25]) and very high V values (for example V in [230, 255]). This basically corresponds to the top of the central axis of the cone.
So, to track white objects in HSV space, you need to:
lower_white = np.array([0, 0, 230]) upper_white = np.array([180, 25, 255])
Or, since you have defined a sensitivity value, for example:
sensitivity = 15 lower_white = np.array([0, 0, 255-sensitivity]) upper_white = np.array([180, sensitivity, 255])
For other colors:
green = 60; blue = 120; yellow = 30; ... sensitivity = 15
The value of Red H is 0, so you need to take two ranges and "OR" together:
sensitivity = 15 lower_red_0 = np.array([0, 100, 100]) upper_red_0 = np.array([sensitivity, 255, 255]) lower_red_1 = np.array([180 - sensitivity, 100, 100]) upper_red_1 = np.array([180, 255, 255]) mask_0 = cv2.inRange(hsv, lower_red_0 , upper_red_0); mask_1 = cv2.inRange(hsv, lower_red_1 , upper_red_1 ); mask = cv2.bitwise_or(mask1, mask2)
Now you can track any color!