After additional research and experimentation, I can say that:
- The Yahoo Boss Search API works best. For a general search on the Internet, they charge $ 0.80 / 1000 calls from the first call. The API returns JSON, including the total number of results. It seems that it has the same coverage as the website, and it is quite simple to use, but they need requests signed with OAuth (no token required, just a signature); There is an example Ruby code that did the trick and can be adapted to use the OAuth gem. Each request is limited to max. 50 and comes with pagination flags to get more results with individual queries (each of which receives an invoice).
- Google Custom Search APIs Once you go through the setup screen (just fill google.com on search sites), set the control panel option to allow “general” web searches and user sites. The API also returns JSON, only the API key in the request is required (without a token or signature) and returns 10 max results, as well as the total number of results. Cost $ 5 for 5000 API calls, after 100 free calls. Perhaps most disappointingly, the user search API uses a different database and does not return the same results on the Internet; many requests that receive hits on the Internet look empty in the API.
- The Bing API is not really worth it. I think Yahoo uses Bing on the backend, but Yahoo's interface is easier to use and more advanced, and documents are better. Bing API documents are downloadable Word documents (!!!), although their search offer is now part of Azure Cloud. Site navigation is the most obscure.
Update: Greg Lindahl in Blackco responded to personally use and invited us to a meeting. They were very helpful in sharing some of their data, and also gave us an API key. This is another good option, especially for specialty datasets, as is the need in our case.