Thanks to everyone for the answer. I have the opportunity to speak with a Microsoft Azure representative and document the conversation here for the community.
The answer to the initial question
The Service Fabric cluster must support a minimum number of Primary node types so that system services support quorum and ensure cluster health. You can learn more about the reliability level and number of instances at https://azure.microsoft.com/en-gb/documentation/articles/service-fabric-cluster-capacity/ . Thus, stopping all virtual machines will cause the Service Fabric cluster to move to a loss of quorum. Often, nodes can be returned and Service Fabric will automatically recover from a loss of quorum, but this is not guaranteed and the cluster can never recover.
However, if you do not need to maintain state in your cluster, it may be easier to simply delete and recreate the entire cluster (the entire Azure resource group) every day. Creating a new cluster from scratch by deploying a new resource group usually takes less than an hour and a half, and this can be automated using Powershell to deploy the ARM template. https://azure.microsoft.com/en-us/documentation/articles/service-fabric-cluster-creation-via-arm/ shows how to set up an ARM template and deploy using Powershell. You can optionally use a fixed domain name or a static IP address so that clients do not reconfigure to connect to the cluster. If you need to support other resources, such as a storage account, you can also configure the ARM template only to remove the VM scale set and SF Cluster resource while saving the network, load balancing, storage accounts, etc.
Q) Is there a better way to stop / start virtual machines, and not directly from a set of scales?
If you want to stop virtual machines to save costs, then starting / stopping virtual machines directly from a set of scales is the only option.
Q) Can we make a core set with the cheapest virtual machines that we can find and add an extra set with powerful virtual machines that we can turn on and off?
Yes, it’s possible to create two types of node - the primary one, which is small / cheap, and the “Worker with a larger size” and set deployment restrictions on your application only for deployment for these larger virtual machines, However, if your Service Fabric service stores state then you will still encounter a similar problem, which, as soon as you lose the quorum (below 3 replicas / nodes) of your working VM, then there is no guarantee that your SF service will return with all the states itself. In this case, your cluster by itself will be fine, since the primary nodes are running, but the state of your services may be in an unknown replication state.
I think you have several options:
- Instead of storing state in trusted Service Fabrics collections, instead, you save your state externally into something like Azure Storage or SQL Azure. You can additionally use something like a Redis cache or robust Service Fabrics collections to maintain a faster read cache, just make sure all records are stored in external storage. Thus, you can freely delete and recreate your cluster at any time.
- Use Fabric Service backup / restore to maintain state and delete the entire resource group or cluster in one night, and then recreate it and restore state in the morning. The duration of the backup / restore will depend entirely on how much data you store and where you export the backup.
- Use something like Azure Batch. Service Fabric is not really designed for a temporary high-performance platform that you can start and stop regularly, so if this is your goal, you might want to take a look at an HPC platform such as Azure Batch, which offers its own capabilities for quickly multiplying computing capacity.