I am not familiar with the new CoreAudio APIs, so it can be a lot easier. This is how I solved the same problem last year.
You can determine if the device requires a password with Bonjour (also called multicast DNS or Zeroconf). You should be able to match the source name with CoreAudio information. For AirPlay audio, you need to check _raop._tcp devices. Here's how it works with the terminal:
Open AirPlay apps first (note that the dns-sd command is called mDNS before Mountain Lion):
laurent ~ $ dns-sd -B _raop._tcp
Browsing for _raop._tcp
DATE: --- Tue 15 Jan 2013 ---
17: 37: 31.977 ... STARTING ...
Timestamp A / R Flags if Domain Service Type Instance Name
17: 37: 31.977 Add 2 4 local. _raop._tcp. 406C8F53F1DD@iMac de Laurent
Then record the full record of the AirPlay instance:
laurent ~ $ dns-sd -L " 406C8F53F1DD@iMac de Laurent" _raop._tcp local
Lookup 406C8F53F1DD@iMac de Laurent._raop._tcp.local
DATE: --- Tue 15 Jan 2013 ---
17: 43: 47.097 ... STARTING ...
17: 43: 47.098 406C8F53F1DD@iMac \ 032de \ 032Laurent._raop._tcp.local. can be reached at iMac-de-Laurent.local.βΊ000 (interface 4)
et = 0,1 ek = 1 ss = 16 raAudioFormats = ALAC, L16 tp = UDP pw = false txtvers = 1 ramach = iMac12,1 vn = 3 md = 0,1,2 sv = false sm = false ch = 2 sr = 44100 rast = afs rastx = iafs cn = 0.1
pw=false indicates that there is no password. This is the main idea, now you should do the same with DNSServiceResolve .
Now, the only way to check if a device that I know about is busy is to actually connect to it. Internally, AirPlay uses RTSP, so you can send this request:
RTSP / 1.0 OPTIONS *
# empty line
If the device responds with a status of 453, it means that it is already streaming.
You can check my AirPlay RTSP stack for more information.