Geek Answers Handbook

The best storage solution for small math data, but fast and with aggregate functions

I am looking for a data storage solution for a project with these requirements:

  • The application creates dynamically contained in the repository.
  • For a short period of time (two weeks, for example) that the table / container receives a huge number of inserts in parallel. Last reading should be immediately available.
  • The data entered is very small, 4 to 6 numeric columns.
  • Small query / filtering support is required, but no joins or cross queries.
  • It is required to perform some aggregate functions such as "Count", "Sum", "Max", "Min" and "Average".

Basically, I need something like Windows Azure Table Storage, but with aggregate functions.

What would you recommend?

+7
source share
6 answers

RavenDB supports everything you mentioned, and more. It scales very well and supports aggregate functions using Map / Reduce queries and Linq queries. It can also work in memory.

+4
source

MongoDB is also a good choice. It supports group (aggregated) requests when installing single-node. If you need to scale, you can create a fragmentation cluster and use the map / reduction for aggregation needs, but the Mongo map / performance reduction is not the top level when using other solutions.

If you need more performance in map / reduce, you can take a look at Hadoop

+3
source

Why don't you use SQL Azure ? It supports Aggregate function and meets all your needs, take a look at this table.

+1
source

If it should not be free, kdb + would be a good choice. This is a system used primarily to finance time series analysis. It supports everything you ask for, and incredibly fast. There is a free trial, but for its use in production it is ... expensive. It is fully requested with sql-type syntax and is fully programmed using its own q language, which is in the apl family.

http://www.kx.com/

0
source

For the requirements listed in the questioning authority, any competent SQL server on the market will handle this, although you will need to configure some indexes to make things go fast.

But, as if to say that if you want your car to move fast, you have to put it in D and press the gas pedal for this to happen.

0
source

Actually you need more information about requirements and budget ... but since people are just throwing some answers, I will throw this into the mix:

There are very cool and very expensive hardware solutions designed to run massively parallel SQL queries such as Netezza Skimmer or TwinFin

http://www.netezza.com/data-warehouse-appliance-products/skimmer.aspx

It almost makes me wish our data warehouse was too slow - just so that I could get one of these puppies.

0
source

More articles:


All Articles