How often to run cron to show the current chronicle?

Question

Webapps that depend on Twitter’s public timeline, how often do they collect data? There must be hundreds of thousands of messages every minute, right? How do they manage to collect all the tweets without missing a single one?

Answer 1

Some services (Friendfeed is a good example) access the Twitter Streaming API , also known as "firehose." This requires approval and written agreement.

Answer 2

Publishing is not a great place to store data. Twitter now uses streaming APIs to output tweets like crazy. The closest comparison to publictimeline is the spritzer method, but this includes only a small sample. If you need to collect all (or more) tweets than the spritzer method, you will need to sign a written agreement in order to access other streaming API channels (HTTP push), such as firehose feed, which returns all public tweets.

Answer 3

The twitter API is limited in speed, as has been said. The public timeline (twitter.com/public_timeline) is not limited in speed in the same sense, but it is updated only every 5 seconds, so most tweets never appear there.

I think three or four companies that have access to firehose are called Twitter full feed. FriendFeed is one of them. The other is Gnip. Gnip resells feed to other companies. This is probably the only possible way to get full twitter.

Answer 4

Go here:

http://twitter.com/help/request_whitelisting

and get your account in the white list (allows 20,000 per hour) if 100 requests per hour are not enough.

@ceejayoz its not 100 GET requests 100 requests in total, excluding several requests such as verify_credentials and rate_limit_status.