This is very subjective. However, I personally believe that your best chance is to use hue-saturation (HSL, or HSV, for that matter), and try to get the value out of the difference in values. For example, something like:
value = a*abs(H1-H2)+b*abs(S1-S2)+c*abs(L1-L2);
and try, through trial and error, to find the best constants a , b and c that distinguish colors of the same amount as your human judgment. If you know linear regression, you can have several colors of the sample with the differences you specify and get the values a , b and c with linear regression.
A more suitable formula may contain, for example, the difference in power 2 instead of abs and, more importantly, the values for a , b and c as a function of the colors themselves, and not the constants.
source share