What is a simple javascripts web page cleanup tool

Question

I would like to make a web page cleaning application that can log in to the site (I was able to do this using twill (python)), and also to execute JavaScript that launches access to other pages.

I would prefer to use something in python, but I am ready to try something new. I installed mechanize, watir, Hojocki etc., but not sure if this really helps.

Answer 1

I believe there are several modules (e.g. Ghost ), but I used Selenium / WebDriver for such things. This is supposedly a test environment, but it provides you with many methods that allow you to interact with the page in the same way as if you loaded it as a regular user. You also have the advantage of running it so that the browser really opens, and you can watch the code run (simplifies debugging) or in the headless mode where the code is just running (there are other answers to / SO sites with much better explanations, than I can give :)).

Having said that, Ghost looks great, so try both of them, and I hope you get what you need!

Also see the Javascript engine (and HTML rendering) without a graphical interface for automation? for a similar question, which may contain some additional answers.

Answer 2

I would recommend PhantomJS .

This is a complete Webkit browser, but headless and scripted.

Ideal for this kind of thing.

Answer 3

I would recommend Octoparse, a free web scraper for Windows. It is not programmable, but it is very easy to use. But there is no Mac version. So ... JavaScript can be handled by Octoparse btw.