Java Web Scraper

Question

I am not able to find any good Java based web API. The site I need to clean also does not provide an API; I want to pageID over all web pages using some pageID and extract HTML headers / other materials in their DOM trees.

Are there other ways besides cleaning the web pages?

Answer 1

jsoup

Retrieving the title is not difficult, and you have many options, look here for stack overflow "Java HTML parsers." One of them is Jsoup .

You can navigate the page using the DOM if you know the structure of the page, see http://jsoup.org/cookbook/extracting-data/dom-navigation

This is a good library, and I have used it in my latest projects.

Answer 2

It is best to use the Selenium Web Driver as it

• Provides visual feedback to the encoder (see your curettage in action, see where it stops).
• Accurate and consistent as it directly controls the browser you are using.

• Slow. Doesn’t hit web pages like HtmlUnit, but sometimes you don’t want to click too fast.

  Htmlunit is fast but terrible when handling Javascript and AJAX.

Answer 3

HTMLUnit can be used for web cleaning, it supports page calling, filling out and submitting forms. I used this in my project. This is a good java library for web scraping. read here more

Answer 4

mechanize for Java would be a good fit for this, and as Wadjy Essam said, it uses JSoup for HMLT. mechanize is a step-by-step HTTP / HTML client that supports navigation, form submission, and page cleanup.

http://gistlabs.com/software/mechanize-for-java/ (and here GitHub https://github.com/GistLabs/mechanize )

Answer 5

There is also Jaunt Java Web Scraping and JSON Querying - http://jaunt-api.com

Answer 6

Take a look at an HTML parser such as TagSoup, HTMLCleaner, or NekoHTML.

Answer 7

You can try the ui4j or cdp4j library for web scraping. ui4j requires Java 8 and uses the JavaFx WebKit browser, and cdp4j requires Chrome.

Answer 8

You can take a peek at jwht-scrapper !

This is a complete recycling infrastructure that has all the features a developer can expect from a web scraper:

• Proxy support
• Warning sign Support for captchas detection and more
• Complex link of the following functions
• Multithreading
• Various delayed delays if necessary
• Rotating user agent
• Request automatic retry and HTTP redirect support
• HTTP headers, cookies, and additional support
• GET and POST support
• Annotation Configuration
• Detailed sorting metrics
• Scraper client asynchronous processing
• jwht-htmltopojo full-featured framework for displaying HTML on POJO
• Processing custom input formats and building in JSON -> POJO mapping
• Full Exception Handling
• Detailed logging with log4j
• POJO injection
• Custom handlers
• Easy to use and well-documented API

It works with the (jwht-htmltopojo) [ https://github.com/whimtrip/jwht-htmltopojo ) lib, which uses the Jsoup mentioned by several other people here.

Together they will help you create awesome scrapers that directly display HTML for POJO and bypass any classic scraping problems in just a few minutes!

Hope this helps some people!

Disclaimer, I am the one who developed it, feel free to tell me your comments!

Answer 9

Using a web scraper, you can extract useful content from a web page and convert to any format, if applicable.

 WebScrap ws= new WebScrap(); //set your extracted website url ws.setUrl("http://dasnicdev.imtqy.com/webscrap4j/"); //start scrap session ws.startWebScrap(); 

Now your web recycling session begins and is ready to crash or retrieve data in java using the webscrap4j library .

For the title:

 System.out.println("-------------------Title-----------------------------"); System.out.println(ws.getSingleHTMLTagData("title")); 

For Tagline:

 System.out.println("-------------------Tagline-----------------------------"); System.out.println(ws.getSingleHTMLScriptData("<h2 id='project_tagline'>", "</h2>")); 

For all anchor tags:

 System.out.println("-------------------All anchor tag-----------------------------"); al=ws.getImageTagData("a", "href"); for(String adata: al) { System.out.println(adata); } 

For image data:

 System.out.println("-------------------Image data-----------------------------"); System.out.println(ws.getImageTagData("img", "src")); System.out.println(ws.getImageTagData("img", "alt")); 

For Ul-Li data:

 System.out.println("-------------------Ul-Li Data-----------------------------"); al=ws.getSingleHTMLScriptData("<ul>", "</ul>","<li>","</li>"); for(String str:al) { System.out.println(str); } 

For complete source code check out this tutorial .

Answer 10

If you want to automate the cleaning of large pages or data, you can try Gotz ETL .

It is fully model driven like a true ETL tool. The data structure, task workflow, and pages to be cleaned are determined by the set of XML definition files, and encoding is not required. A query can be written either using selectors with JSoup, or XPath using HtmlUnit.

Answer 11

There are many open source Java and python-based scanners available that you can customize to suit your requirements, some of which are described below.

 Apache nutch
 Stormcrawler
 Jsoup
 Jaunt

in your case, since you need a single price on the product page, you can create your own using JSoup, the framework available in Java, or the Beautiful Soup module in Python.

if the scale doesn't matter, and you just want to scan multiple pages daily, I recommend creating your own scanner. otherwise you can use Nutch or StormCrawler

Also for an individual order, please do not use multiple selectors for different web pages, in fact, just find a common tag, CSS or template that will give you a price.