Geek Answers Handbook

Large Writer / Reader Engine - C ++ High Frequency Data

I need one writing and multiple reading mechanism (up to 5), which the author clicks on data of almost 1 MB each and 15 packets per second continuously, which will be written in C ++. What I'm trying to do is a single stream that continues to write data, while 5 readers will do some search operations according to the timestamp of the data at the same time. I have to store each data packet for 60 minutes, and then they can be removed from the container.

Since data can grow as much as 15 MB * 60 s * 60 min = 54000 MB / h I need almost 50 GB of storage space and make the operations fast enough for both the writer and readers. The fact is that we cannot store data of this size in cache or RAM, so it must be on a hard drive, such as an SSD (the hard drive will be too slow for this kind of operation)

Until now, what I was thinking about, I need to make a circular buffer (since I can calculate the maximum size) directly embedded in the SSD, and I could not find a suitable example so far, and I do not know if this is possible or not, or implement some kind of mapping mechanism, which will have one circular array in RAM, which simply saves time stamps of the data and the physical memory address to search for data available on the hard disk. At least search operations will be faster, I think.

Since any kind of lock, mutex or semaphore slows down operations (especially critical writing, we cannot lose data due to any read operation) I do not want to use them. I know that there are some common locks, but I think they have some disadvantages. Is there any way / idea to implement such a system without blocking, without waiting and thread safety? Any data structure (container), template, code / project example or other suggestions will be highly appreciated, thanks ...

EDIT: Is there any other idea, not more RAM?

+4
source share
4 answers

( ).

. , , , .
, - ( ), , , pthread_mutex ( , -). .

, , , . , .
, , , (, , ) . , . , , = .

, 15MiB/s , 5-6 , SSD 10-20 . , . , .

54.1GB 1 , (, 64- , , , ), . , 2.
, , 3 , . , .

, , - , , , ( , ). , , , ( , 2-3 1MiB , !).

, , 5 , . , , 5 - - , .

, ( , ) , "", , " ".
, "" , (-/ ). . ( ), , .

, , , , , . , , .

, , (, "emtpy" ) . , .

54.1Gib > 54Gib, 1MiB , . , , , (2 3 ), . ( 1), " ". , , , , ( ).
, , ( 1 1/2 ). , , . , , - , 100% , .
, , "". , , , .

- , 1 , , "" , "" max_size . . , , , , , . , , , , , , "" . , " " , . . , .

1 , , "" 1MiB.

2 , madvise ( DONT_NEED, Linux), , . , , , , , --, .

3 , , .

+3

54 / = 15 /. SSD 300+ /. 1 , , 10x 15 / ( , ).

, RCU - Read-Copy Update. Linux .

0

? 54 ( 416 , ). , SSD, , , , .

, , (, CPU, - 0). , (, , ), .

0

More articles:


All Articles