How to make a Java application scalable and fault tolerant?

Question

In a simplified form, my Java application can be described as follows:

This is a web application running on a Tomcat SOAP server. The application uses JPA / Hibernate to store data in a MySQL database. The stored data consists of a list of users, a list of hosts and a list of URIs pointing to huge files (10 GB) in the file system. The whole system consists of a central server, on which my application runs, and a group of working hosts. The user can connect to the SOAP interface and ask the system to copy the files belonging to him to a specific work node, where he can somehow analyze the data (we cannot use NFS, we need to copy the data to the local storage of the working host). Then the database is stored for each user on which his file is stored.

Currently, the system works with one central server with the Tomcat application and the MySQL database, as well as with 10 working hosts and about 30 users who have 100 files (10 GB on average) that are stored on the working nodes.

But in the future I have to scale the system 100-1000 times. Therefore, I may have to deal with 10,000 users, 100,000 files, and 10,000 hosts. And the system should also become fault tolerant, so I do not have a single central server (which is now the only point of failure in the system), but there may be several. In addition, if one of the work nodes fails, the system must be notified, so it does not try to copy files on this server.

Now my question is: what Java technologies could I use to make the application scalable and fault tolerant? What architecture would you recommend? Should I have a huge database that stores all the information about all files, hosts and users in the system in one place, or is it better for me to distribute my database on several hosts and somehow synchronize them?

Answer 1

The necessary technology is called Architecture.

No matter what technology you use, you need a well-designed system for scalability and redundancy. Make a diagram of the entire system architecture, how it currently works. Mark each component with its restrictions for users, tasks, bandwidth, hard disk space, memory, or any parts that limit your application. This will give you a basic design.

Now draw the same diagram as for the requirements for scalability and redundancy. You may need to break up the pieces to make them work, or develop completely new items. This chart allows you to clearly understand what you need.

One specific thing I want to refer to is the database. If you can divide the database along logistic lines so as not to join any queries from one to another, then you should have separate databases. In addition, the best configuration for the database is the presence of each database on one fast machine with lots of memory and very fast access times. If you do, the only thing that will slow down your database is bad queries or poorly indexed tables. In my experience, database synchronization should be avoided unless you have one primary database with write access and it is replicated to other read-only databases. Despite this, this may be the last step after you have profiled all your requests, and you literally need additional equipment.