What aspect of relational databases makes scaling difficult for services like Google App Engine?

Question

Apparently, the reason for the BigTable architecture is due to the complexity of scaling relational databases when you are dealing with the huge number of servers Google has to deal with.

But technically speaking, what exactly makes scaling relational databases difficult?

In the corporate data centers of large corporations, they seem to be able to do this successfully, so I wonder why it's impossible to just do it at a higher level so that it can scale on Google servers.

Answer 1

In addition to Mitch's answer, there is another aspect: Webapps are generally not well suited for relational databases. Relational databases focus on normalization - basically, they simplify writing, but they are more difficult to read (from the point of view of the work done, and not for you). This works very well for OLAP applications, and not for web applications, which are usually massively weighted in favor of reading over the record.

The strategy adopted by non-relational databases such as Bigtable is the opposite: denormalize to make reading a lot easier by making records more expensive.

Answer 2

When you execute a query that includes relationships that are physically distributed, you must pull this data for each relationship into a central location. This clearly will not scale well for large amounts of data.

A well-tuned RDBMS server will execute most of the hot page requests in RAM with a small physical disk or network I / O.

If you are limited by network I / O, then the benefits of relational data are diminished.

Answer 3

The main reason mentioned is the physical location and network IO. In addition, even large corporations deal with the share of data that search engines work with.

Think of an index in a standard database, perhaps a few feilds ... search engines need a quick text search in large text fields.