Headless Java HTTP client for scanning?

Question

I was looking for a crawl tool written in Java to detect an invalid URL on our sites.

The difficulty is that most of the URL is executed using javaScript, CSS3 and Ajax. So just getting the contents of the website URL will not.

An ideal would be a headless tool that could execute javaScript, CSS style and AJAX calls and spit out the different URL that it accessed.

I understand that this is a high order, but perhaps it exists somewhere?

Answer 1

I suggest using http://htmlunit.sourceforge.net/ , which is done for these things.

Answer 2

http://hc.apache.org/httpcomponents-client-ga/index.html