The easiest way to properly load html from a webpage to a string in Java

Question

What the title says.

Help with thanks!

Answer 1

An extremely common mistake is the inability to correctly convert the HTTP response from bytes to characters. To do this, you need to know the character encoding of the response. I hope this is indicated as a parameter in the "Content-Type" parameter. But put it in the body, since the "http-equiv" attribute in the meta tag is also an option.

So, it is surprisingly hard to load the page correctly in String , and even third-party libraries like HttpClient do not offer a general solution.

Here is a simple implementation that will handle the most common case:

 URL url = new URL("http://stackoverflow.com/questions/1381617"); URLConnection con = url.openConnection(); Pattern p = Pattern.compile("text/html;\\s+charset=([^\\s]+)\\s*"); Matcher m = p.matcher(con.getContentType()); /* If Content-Type doesn't match this pre-conception, choose default and * hope for the best. */ String charset = m.matches() ? m.group(1) : "ISO-8859-1"; Reader r = new InputStreamReader(con.getInputStream(), charset); StringBuilder buf = new StringBuilder(); while (true) { int ch = r.read(); if (ch < 0) break; buf.append((char) ch); } String str = buf.toString(); 

Answer 2

You can simplify it a bit with org.apache.commons.io.IOUtils :

 URL url = new URL("http://stackoverflow.com/questions/1381617"); URLConnection con = url.openConnection(); Pattern p = Pattern.compile("text/html;\\s+charset=([^\\s]+)\\s*"); Matcher m = p.matcher(con.getContentType()); /* If Content-Type doesn't match this pre-conception, choose default and * hope for the best. */ String charset = m.matches() ? m.group(1) : "ISO-8859-1"; String str = IOUtils.toString(con.getInputStream(), charset); 

Answer 3

I use this:

  BufferedReader bufferedReader = new BufferedReader( new InputStreamReader( new URL(urlToSeach) .openConnection() .getInputStream() )); StringBuilder sb = new StringBuilder(); String line = null; while( ( line = bufferedReader.readLine() ) != null ) { sb.append( line ) ; sb.append( "\n"); } .... in finally.... buffer.close(); 

He works most of the time.