High Performance Impression Tracking

Question

In principle, one part of some indicators that I would like to track is the number of impressions that some objects receive on our marketing platform. If you imagine that we are showing a lot of objects, we would like to track every time the object is serviced.

Each object is returned to the client through one gateway / interface. Therefore, if you imagine that a query comes to a page with some search criteria, then the search query is proxied to our Solr index.

Then we get 10 results.

Each of these 10 results should be considered as an impression.

I'm struggling to find an incredibly fast and accurate implementation.

Any suggestions on how you can do this? You can use any number of technologies. We currently use Gearman, PHP, Ruby, Solr, Redis, Mysql, APC, and Memcache.

Ultimately, all impressions should end up being stored in mysql, which I could do every hour. But I'm not sure how to quickly store impressions in memory without affecting the load time of the actual search request.

Ideas (I just added options 4 and 5)

• After the results are returned to the client, the client will then request a base64 encoded encoding on our platform, which contains the identifier of all the objects that they served. This object is then transferred to the gearbox, which then stores the counter for retry. Once per hour, redis is painted over and the counter is incremented for each object in mysql.

• After the results have been returned from Solr, roll over and save directly to Redis. (Did not compare it for speed). Repeat flushing in mysql every hour.

• As soon as the elements are returned from Solr, send all the IDs in one job to the mechanism, which then sends Redis ..

• a new idea. Since there will be around 20 for most of the returned items, I can set the X-Application-Objects header with the base64 header of the returned identifier. These identifiers (in the header) can then be deleted by nginx, and using the custom LUA module nginx, I could write the ID directly to Redis from nginx. However, this may be redundant. The advantage of this is that I can tell nginx to immediately return the response object while it writes redis.

• new idea Use fastcgi_finish_request() to clear the request to nginx, but then paste the results into Redis.

• Any other suggestions?

Change the answer to the question:

The reliability of this data is not significant. This is the best guess so far. I would not want the swing to say that 30% made an impression. But I would allow a tolerance of 10% - / +.

Answer 1

I see two of your best options:

• Using the increment command, I iterate over incremental counts as I delete. Use Id as a key and increase it in Redis. Redis can easily handle hundreds of thousands of increments per second, so it should be fast enough to do without a noticeable impact on the client. You can even pipeline each request if the PHP language binding supports it. I think this is right.

• Use redis as a regular cache. In this option, you simply use the Redis list and execute the rpush of the line containing the identifiers separated, for example. comma. You can use the hour of the day as a key. Then you can have a separate process to pull it out, capturing the previous hour and massaging it, but you want it in MySQL. I would put the expiration date of the keys, which you can clear them after a while, or simply delete the keys using the post-processing process.

You can also use read-slave to export to MySQL if you have very high redis traffic or just want to unload it and get a backup copy as a bonus. If you do this, you can install the master redis instance so that it does not merge with the disk, increasing write performance.

For some additional options related to the use of more advanced functions redis for this kind of tracking, see. The this? Answer . You can also avoid MySQL and pull data from redis, simplifying the overall system.

Answer 2

I would do something like # 2 and pass the data in the fastest queue to update the Redis counters. I am not so familiar with Jirman, but I bet I can. If your Redis client supports asynchronous writes, I would use it or queue it in a separate thread. You do not want to slow down your response while waiting for the counters to update.