400 Http Errors Using Jsoup in a Multithreaded Program

Question

I created a program that parses html pages. I am using the jsoup connect function inside a class called in ThreadPool. The problem is that I connect to the same website and has a thread pool size of 5+, I get IO exceptions - 400 errors.

How can I not do this?

Answer 1

If you get an HTTP 400 response, check the contents of the response to the error message. 400 means a bad request: you did not provide all the necessary information or did not provide incorrect information. Some people also use it as a kind of trick when a customer does something they don’t like. If you make many different requests, examine the ones that called 400 to see that something is wrong with them. If they all look right, or if you send the same requests repeatedly, then perhaps the site you click on has some kind of speed limit that prevents you from making too many simultaneous requests or too many requests in a certain timeframe. If it is something like this, I would expect a response to what is happening in the response.