I need to insert ~ 80,000 rows in redis at a time and studied using redis pipelining for this. However, when testing with only 1000 rows inserted, it takes 46 seconds with the conveyor versus 6 seconds without conveyor processing.
In the code below, I have a list of zipcodes grouped by zipcode, which I am trying to insert into redis. They are inserted as a RedisZipCode, which contains the zipcode as id and the list of zipcodes that were collected during grouping.
public class ZipCode { public string city { get; set; } public string state { get; set; } public string zip_code { get; set; } } public class RedisZipCode { public string id { get; set; } public List<ZipCode> zipcodes{ get; set; } }
No pipelining
using (var zipCodeClient = redisClient.GetTypedClient<RedisZipCode>()) { foreach (var item in zipcodes.GroupBy(z => z.zip_code)) { zipCodeClient.Store(new RedisZipCode(item.ToList())); } }
with conveyor
using (var zipCodeClient = redisClient.GetTypedClient<RedisZipCode>()) using (var pipeline = zipCodeClient.CreatePipeline()) { foreach (var item in zipcodes.GroupBy(z => z.zip_code)) { pipeline.QueueCommand(c => c.Store(new RedisZipCode(item.ToList()))); } pipeline.Flush(); }
source share