TL; DR - Estimating the expected cluster power is partly art, partly science. Most likely, you can get a good lower bound that you can push higher, but for the most part, deploying, running, and collecting data under your workload is the best way to answer this question.
1) As a rule, collections on this computer are limited by the amount of available memory or the amount of available disk space on the node, depending on which is smaller. Today we store all the data in collections in memory and save it to disk. Thus, the maximum amount that your collections can store in the cluster is usually (the amount of available memory in the cluster) / (the size of the target replica set).
Please note that “Available memory” is what remains of other code that runs on machines, including the OS. In the above example, although you are not working with all the nodes, you can only get 3 of them. Thus, (unrealistic) assuming that due to these other factors, an additional load of 0 is occurring, you can expect that you can place about 3.5 GB of data in this stateful replica before you run out of memory on the nodes on which he worked. There would still be 2 nodes in the cluster.
Let's take another example. Suppose it is about the same as in the above example, except that in this case you are setting up a stateful service for partitioning. Suppose you select the number of partitions 5. So, now on each node you have a primary replica and 2 secondary replicas from other partitions. In this case, each partition can contain a maximum of about 1.16 GB of state, but now in general you can pack 5.83 GB of state in a cluster (since all nodes can now be fully used). By the way, just to prove the mathematical work, (3.5 GB of memory per node * 5 nodes in a cluster) [17.5] / (target replica set size 3) = 5.83.
In all of these examples, we also assumed that the memory consumption for all partitions and all replicas is the same. It often turns out that this is not so (at least temporarily) - some sections may require more or less work, which leads to uneven consumption of resources. We also suggested that the secondary were always the same as the primary. In the case of the number of states, it is probably fair to assume that they will be tracked fairly evenly, although this may not happen for other resources (just something to keep in mind). In the event of uneven consumption, this will really help the rest of the Service Fabric cluster resource management, as we can learn about the consumption of various replicas and pack them in a cluster efficiently to use the available space. Automatic reporting on the consumption of resources related to the status in the collections is on our radar, and we want to do something, so in the future it will be automatic, but today you will have to report this consumption yourself.
2) By default, we will balance the services in accordance with the default metrics (more about the metrics here ). Thus, by default, different replicas of these two different services can appear on the computer, but in your example you will get 4 nodes with 1 replica from the service on it, and then 1 node with two replicas of two different services. This means that each service (each with 1 partition according to your example) will be able to consume only 1.75 GB of memory in each service, for a total of 3.5 GB in a cluster. This is again less than the total available cluster memory, as there are some parts of the nodes that you are not using.
Please note that this is the maximum possible consumption, and it is assumed that consumption is not included in the service itself. Taking this as a maximum is not recommended. You will want to reduce it for several reasons, but the most practical reason is that if there are updates and crashes, there is sufficient available capacity in the cluster. As an example, suppose you have 5 update domains and 5 failure domains. Now suppose that the nodes of the failed nodes fail when the update domain is updated. This means that (a little less) 40% of the capacity of your cluster can be lost at any time, and you probably want enough space left on the remaining nodes to continue working. This means that if your cluster could previously contain 5.83 GB of state (according to our previous calculations), in reality you probably won’t want to add more than 3.5 GB of state to it, since with a larger number this service will not be able to return to 100% performance (also note that we do not create replacement replicas immediately, so the nodes must not be accessible to your ReplicaRestartWaitDuration before you encounter this case). This article discusses a lot of additional information about metrics, capacity, buffered capacity (which can be used to provide free space on nodes for failure cases), and failure and update domains.
There are some other things that will practically limit the amount of state that you can store. You want to do a few things:
- Estimate the size of your data. You can estimate in advance how large your data is by calculating the size of each field that your objects contain. Be sure to take into account 64-bit links. This will give you the lower bound of the starting point.
- Overhead storage. Each object that you store in the collection will have some overhead for storing this object. In reliable collections, depending on the collection and operations currently being performed (copying, enumerating, updating, etc.), these costs can vary from 100 to 700 bytes per item (string) stored in the collections. Know also that we are always looking for ways to reduce the amount of overhead that we introduce.
We also strongly recommend that you start your service over a period of time and measure the actual consumption of resources using performance counters. Modeling some kind of real workload, and then measuring the actual use of the metrics that interest you, will be very useful to you. The reason why we recommend this, in particular, is because you can see consumption from things like, for example, a bunch of CLR objects into which your objects are placed, how often the GC works, leaks or other similar things that affect the amount. memory you can really use.
I know this was a long answer, but I hope you find it useful and complete.