CasperJS can't run twitter endless scroll

Question

I am trying to get some information from twitter using CasperJS . And I'm stuck with an endless scroll. The thing is, even using jquery to scroll down a page seems to work. Neither scrolling nor triggering the exact event on the window (smth, like uiNearTheBottom) seems to help. The interesting thing is that all these attempts work when injecting JS code through the js console in FF and Chrome. Here is a sample code:

 casper.thenEvaluate(function(){ $(window).trigger('uiNearTheBottom'); }); 

or

 casper.thenEvaluate(function(){ document.body.scrollTop = document.body.scrollHeight; }); 

Answer 1

If casper.scrollToBottom () fails for you or casper.scroll_to_bottom (), then the following will be shown to you:

this.page.scrollPosition = {top: this.page.scrollPosition ["top"] + document.body.scrollHeight, left: 0};

Working example:

 casper.start(url, function () { this.wait(10000, function () { this.page.scrollPosition = { top: this.page.scrollPosition["top"] + document.body.scrollHeight, left: 0 }; if (this.visible("div.load-more")) { this.echo("I am here"); } })}); 

It uses the PhantomJS hidden scrollbar found here

Answer 2

CasperJs is based on PhantomJS, and as per the discussion below, there is no window object for a browser without a browser.

You can check out the discussion here.

Answer 3

On Twitter you can use:

 casper.scrollToBottom(); casper.wait(1000, function () { casper.capture("loadedContent.png"); }); 

But if you enable jQuery ..., the above code will not work!

 var casper = require('casper').create({ clientScripts: [ 'jquery-1.11.0.min.js' ] }); 

The paste script blocks Twitter from endlessly scrolling from loading content. At BoingBoing.net, CasperJS scrollToBottom () works with jQuery without blocking. It really depends on the site.

However, you can add jQuery after loading the content.

 casper.scrollToBottom(); casper.wait(1000, function () { casper.capture("loadedContent.png"); // Inject client-side jQuery library casper.options.clientScripts.push("jquery.js"); // And use like so... var height = casper.evaluate(function () { return $(document).height(); }); }); 

Answer 4

I took this from a previous answer

 var iterations = 5; //amount of pages to go through var timeToWait = 2000; //time to wait in milliseconds var last; var list = []; for (i = 0; i <= iterations; i++) { list.push(i); } //evaluate this in the browser context and pass the timer back to casperjs casper.thenEvaluate(function(iters, waitTime) { window.x = 0; var intervalID = setInterval(function() { console.log("Using setInternal " + window.x); window.scrollTo(0, document.body.scrollHeight); if (++window.x === iters) { window.clearInterval(intervalID); } }, waitTime); }, iterations, timeToWait); casper.each(list, function(self, i) { self.wait(timeToWait, function() { last = i; this.echo('Using this.wait ' + i); }); }); casper.waitFor(function() { return (last === list[list.length - 1] && iterations === this.getGlobal('x')); }, function() { this.echo('All done.') }); 

Essentially, I enter the context of the page, scroll to the bottom and then wait 2 seconds to load the content. Obviously, I would like to use repeated applications of casper.scrollToBottom() or something more complicated, but the loading time did not allow me to do this.