Using Redis SCAN in NODE

Question

I have Redis with a large number of keys in some format, and I want to get the keys that match some pattern and perform some operations on them. I do not use the KEYS method, since it is not recommended in production. Using SCAN. I am wondering what is the best way to write code. I should do something like a while loop, but using promises, my current solution looks like this (the code is a bit simplified):

'use strict' const Promise = require('bluebird'); const config = require('./config'); const client = require('./clinet'); let iterator = 0; Promise.coroutine(function* () { do { iterator = yield clinet.scanAsync(iterator, 'myQuery', 'COUNT', config.scanChunkSize) .then(data => { let nextIterator = data[0]; let values = data[1]; //do some magic with values return nextIterator; }) } while (iterator !== '0'); })(); 

Is there a better way to do this that I am missing?

Answer 1

You can use recursion to continue a scan call to completion.

 function scanAsync(cursor, pattern, returnSet){ return redisClient.scanAsync(cursor, "MATCH", pattern, "COUNT", "100").then( function (reply) { cursor = reply[0]; var keys = reply[1]; keys.forEach(function(key,i){ returnSet.add(key); }); if( cursor === '0' ){ return Array.from(returnSet); }else{ return scanAsync(cursor, pattern, returnSet) } }); } 

Go to Set () to make sure the keys are not duplicated.

 myResults = new Set(); scanAsync('0', "NOC-*[^listen]*", myResults).map( function( myResults ){ console.log( myResults); } ); 

Answer 2

You can try this snippet for scan (1000) keys to iterate and delete.

 var cursor = '0'; function scan(pattern,callback){ redisClient.scan(cursor, 'MATCH',pattern,'COUNT', '1000', function(err, reply){ if(err){ throw err; } cursor = reply[0]; if(cursor === '0'){ return callback(); }else{ var keys = reply[1]; keys.forEach(function(key,i){ redisClient.del(key, function(deleteErr, deleteSuccess){ console.log(key); }); }); return scan(pattern,callback); } }); } scan(strkey,function(){ console.log('Scan Complete'); }); 

Answer 3

Go through this, it can help.

https://github.com/fritzy/node-redisscan

don't use the library as such, go through the code available at https://github.com/fritzy/node-redisscan/blob/master/index.js

Answer 4

I think that node bindings for Redis place too much responsibility on the caller here. Therefore, I created my own library for scanning using generators in the node:

 const redis = require('redis') const client = redis.createClient(…) const generators = require('redis-async-gen') const { keysMatching } = generators.using(client) … for await (const key of keysMatching('test*')) { console.info(key) } 

This is the last thing, obviously, that you should take care of. Instead of carefully controlling the iterator, all you have to do is use it for understanding.

I wrote more about it here .