FIFO / QUEUE with buffer and thread safe

Question

I get a ton of statistics that I need to insert in db. I would like to implement some kind of Queue or FIFO class that stores all the data and when it reaches a certain count (buffer), it will send this data to SQL through a bulk insert. This should be thread safe.

I know how to do volume insert. Any tips on how to make the queue / list?

thanks

Answer 1

The .net base class library has a ConcurrentQueue(Of T) . Just import System.Collections.Concurrent .

Edit: if you must use a queue, you can create a wrapper class / module that fires an event when the counter (buffer) reaches a certain amount.

Answer 2

If you do not need strict FIFO, I think you should use BlockingCollection .

It is thread safe and the implementation will look something like this:

 var collection = new BlockingCollection<Data>(); var sqlinserter = Task.Factory.StartNew(UpdateSql()); while (true) { Data statistics = FetchStatistics(); if (statistics == null) break; collection.Add(statistics); } collection.CompleteAdding(); sqlinserter.Wait(); 

Edit Saw that you want to insert a certain number of elements in each batch

 void UpdateSql() { var batch = new List<Data>(); foreach (var item in collection.GetConsumingEnumerable()) { batch.Add(item); if (batch.Count > SomeBatchSize) { InsertIntoSql(batch); batch.Clear(); } } if (batch.Count > 0) InsertIntoSql(batch); // insert remaining items } 

Answer 3

This is a safe way to handle this. First of all, you want to avoid any situation where you can get stuck inside a synchronization session.

 Public Class TSQueue(Of T) Private q As New Queue(Of T) Public Property threshold As Integer = 100 Public Event ThresholdHit As EventHandler(Of EventArgs) Public Sub EnqueueSafe(value As T) Dim notify As Boolean = False SyncLock q q.Enqueue(value) If q.Count >= threshold Then notify = True End If End SyncLock If notify Then RaiseEvent ThresholdHit(Me, EventArgs.Empty) End If End Sub Public Function DequeueSafe() As T SyncLock q Return q.Dequeue() End SyncLock End Function Public Function DequeueAllSafe() As T() Dim out() As T SyncLock q out = q.ToArray() q.Clear() End SyncLock Return out End Function Public Function CountSafe() As Integer SyncLock q Return q.Count End SyncLock End Function End Class