In fact, I have a new, better answer to this question related to a large number of undocumented goodies. Here it is for future offspring:
NSWindow has an undocumented method that does exactly what you want:
@interface NSWindow (Private)
- (void )_setPreventsActivation:(bool)preventsActivation;
@end
[myWindow _setPreventsActivation:true];
This stops the window from activating both itself and its application when the user clicks on it.
API, , : Apple - ( OS X, , ), Mac.