Save and access real-time data with .net

Question

Does anyone have the experience of receiving and updating a large amount of data, storing, sorting and visualizing it very quickly?

Preferably, I am looking for a .NET solution, but this may not be practical.

Now for the details ...

I get about 1000 updates per second, some updates, some new rows of data records. But it can also be very impulsive, sometimes with 5000 updates and new lines.

By the end of the day, I could have 4 to 5 million rows of data.

I need to both store them and show user updates in the user interface. The user interface allows the user to apply several filters to the data to simply show what they want. I need to update all records and show users these updates.

I have a refresh rate of 1 fps.

Does anyone have any guidance or guidance on this issue? I can not imagine that I am the first to deal with something like this ...

At first, at least some kind of memory database that I think of, but will it be fast enough for update requests at the end of the day, as soon as I get a large enough data set? Or does it all depend on smart indexing and queries?

Thanks in advance.

Answer 1

This is a very interesting and difficult problem.

I would approach the design of a pipeline with processors that implement sorting, filtering, aggregation, etc. The pipeline needs an async (threadafe) input buffer, which is processed in a timely manner (according to your 1fps req. Per second). If you cannot do this, you need to queue the data somewhere, on disk or in memory, depending on the nature of your problem.

Therefore, the user interface should be implemented in the style of pull, not push, you want to update it every second.

For a data warehouse, you have several options. Using a database is a good idea, because in any case you need data (and, I think, also requested). If you use ORM, you can find NHibernate in combination with an excellent second-level cache worthy of choice.

Many of the considerations may also be similar to those of Ayende when developing NHProf, a real-time profiler for NHibernate. He wrote a series of posts about them on his blog .

Answer 2

Maybe Oracle is a better RDBMS solution for you. The problem with your question is that there are too many variables and conditions that you need to deal with at these "critical" levels. Not only the software, but also the equipment that you have (it costs :)), the connection speed, the expected general configuration of the user system and more and more ... Good luck.