As suggested in other answers, finding a color might be a good idea. You should consider looking for a specific color range. The best way to do this is to convert the image to the HSL or HSV color space.
cv::cvtColor(src, hsv, COLOR_BGR2HSV);
Additional information on Wikipedia .
Then you have three channels: hue (= color), saturation, and lightness (or value).
With cv::inRange(hsv, cv::Scalar(159, 135, 165), cv::Scalar(179, 255, 200), inRange); Now you can create a black and white image that shows which pixels are in the color scheme. Scalars are low and high values โโfor each channel.
In this example, you get pixels with a color of 159 to 179 (hue), a saturation between 135 and 255, and a value between 165 and 200.
Perhaps this can improve your tracking.
Gray
source share