[in advance, apologize to the group for using part of the answer space for meta-like questions]
From OP, Lars D:
I do not consider this answer to be the answer to the question, because it does not bring me closer to a solution. I know what cloud computing is, and I know that the algorithm can be ideally divided into more than 300,000 servers, if necessary, although the additional costs will not give additional additional performance due to network latency.
Lars
I sincerely apologize for reading and answering your question on a naive and general level. I hope you will see how the lack of specificity of the question itself, especially in its original form, as well as the somewhat unusual nature of the problem (1) would prompt me to answer the question in this way. This and the fact that such questions about SO, as a rule, come from hypotheses by people who think little and research this process, are my excuses for believing that I, who are indifferent (to massively distributed systems), could help your quest. Many of these answers (some of which were useful for the additional information that you provided), as well as numerous comments and additional questions addressed to you, indicate that I was not alone with this thinking.
(1) Non-obvious problem: [apparently] mainly a computational process (without mention of distributed / replicated storage structures), very highly parallelizable (1,500 servers), into fifty millisecond-sized tasks that together provide a sub-second answer (? For consumption man?). And yet a process that will only be needed a few times [daily ..?].
Just look back!
In practical terms, you can consider some of the following to help improve this SO question (or move it to other / alternative questions) and, therefore, facilitate assistance from domain experts .
- re-publication as a separate (more specific) issue. In fact, perhaps a few questions: for example. on the [probable] weak latency and / or overhead of redistribution processes, at current prices (for specific TOS data and volume), on awareness of distributed processes from different suppliers, etc.
- Change title
- Add information about the process that you have (see the many questions in the notes for both the question and the many answers).
- in some issues, add tags specific to the supplier or the delivery method (EC2, Azure ...), as this may lead to the possibility that it is not completely unbuyist, but it’s useful, nevertheless, a comment from agents in these companies
- Show that you understand that your quest has a slightly higher order.
- I explicitly declare that you want feedback from effective performers of basic technologies (perhaps also include people who also “wash their feet” with these technologies, because, with the exception of people with physics / high energy, etc. that BTW has traditionally worked with clusters, not with clouds, many of the technologies and practices are relatively new)
In addition, I will be happy to give you a hint (with an implicit non-veto from other people on this page) to remove my answer if you find that this will help improve the answers.
- original answer -
Warning: Not all processes or mathematical calculations can be easily divided into separate parts, which can then be performed in parallel ...
Perhaps you can check out the Wikipedia entry for Cloud Computing , realizing that cloud computing, however, is not the only architecture that allows parallel computing.
If your process / calculation can be effectively divided into parallelizable parts, perhaps you can look at Hadoop , or other MapReduce implementations, for a general understanding of these parallel processes. In addition, (and, I believe, using the same or similar algorithms), there are also commercially available structures such as EC2 from amazon .
Beware, however, that the above systems are not particularly well suited for very fast response times. They work better with clocks (and then with some) hashing data / numbers and similar tasks, rather than with small calculations like the one you want to parallelize, so it gives results in 1/10 of a second.
The above frameworks are general, in a sense, that they can trigger processes of almost any nature (again, those that can be at least partially separated), but there are also various suggestions for specific applications, such as search or DNA matching, etc. .d. Search applications, in particular, can have a very short response time (for example, Google) and BTW, this is partly due to the fact that such tasks can be easily and quickly transferred for parallel processing.