ServiceStack Redis buffer insert using pipelining

I need to insert ~ 80,000 rows in redis at a time and studied using redis pipelining for this. However, when testing with only 1000 rows inserted, it takes 46 seconds with the conveyor versus 6 seconds without conveyor processing.

In the code below, I have a list of zipcodes grouped by zipcode, which I am trying to insert into redis. They are inserted as a RedisZipCode, which contains the zipcode as id and the list of zipcodes that were collected during grouping.

public class ZipCode { public string city { get; set; } public string state { get; set; } public string zip_code { get; set; } } public class RedisZipCode { public string id { get; set; } public List<ZipCode> zipcodes{ get; set; } } 

No pipelining

 using (var zipCodeClient = redisClient.GetTypedClient<RedisZipCode>()) { foreach (var item in zipcodes.GroupBy(z => z.zip_code)) { zipCodeClient.Store(new RedisZipCode(item.ToList())); } } 

with conveyor

 using (var zipCodeClient = redisClient.GetTypedClient<RedisZipCode>()) using (var pipeline = zipCodeClient.CreatePipeline()) { foreach (var item in zipcodes.GroupBy(z => z.zip_code)) { pipeline.QueueCommand(c => c.Store(new RedisZipCode(item.ToList()))); } pipeline.Flush(); } 
+6
source share

All Articles