Geek Answers Handbook

NoSQL's Best Approach for Processing Over 100 Million Records

I am working on a project because we are batch loading and storing a huge amount of data in an Oracle database, which is constantly queried using Hibernate for this table with over 100 million records (reading is much more common than records). To speed things up, we use Lucene for some queries (especially geo-referenced queries) and a second level Hibernate cache, but this is still not enough. We still have a bottleneck in Hibernate queries against Oracle (we do not cache more than 100 million table elements in the Hibernate second-level cache due to the lack of such memory).

What additional NoSQL solutions (besides Lucene) can I use in this situation?

Some options that I think of are as follows:

  • Use distributed ehcache (Terracotta) for the second level of Hibernate to use more memory on computers and reduce duplicate caches (right now each virtual machine has its own cache).

  • To fully utilize an in-memory SQL database such as H2, but unfortunately, these solutions require loading more than 100 million tables into a single virtual machine.

  • Use Lucene for queries and BigTable (or a distributed hash file) to search for entities by identifier. Which BigTable implementation would be suitable for this? I was considering HBase.

  • Use MongoDB to store data and queries and search by identifier.

+5
6

Cassandra ElasticSearch (100 ). cassandra ES . . Cass MQ, rabbitmq. ES.

+6

. NoSQL , . , , NoSQL. . , node, , node ( ).

CAP, NoSQL (CP AP), CA. , , , .

, , HBase, . , . Mongo , Solr Mongo. Mongo, HBase, (, Google BigTable), , , -.

, , , , , , , , , , . Lily . , , , , , , , , .. , . , , , , , ( - ). Hadoop, HBase, Hive, MongoDB, Solr, MySQL . , , , , Oracle , , . , .

, , - NewSQL - RDBMS... , MySQL- ( ) VoltDB, .

, , NoSQL Non-Rel, , . , SQL-, , ( ), Oracle , .

, . :

MongoDB - - CP

CouchDB - - AP

Redis - ( ) - CP

Cassandra - - (AP)

HBase - - (CP)

Hadoop/

VoltDB - , , ( , ). , , , ( - ).

2c. - , .

+2

Lily (lilyproject.org). HBase Solr. , Solr HBase. solr ( ), .

+1

( ), , , .

.

, 10 , .

( -).

, .

0

100M , , Hibernate, Oracle. Oracle, .

?

0

, MongoDB ( NoSQL) . , , MongoDB, . , MongoDB, / , . usecase , 20 .

0

More articles:


All Articles