Architecture. How to effectively scan a network using 10,000 machines?

Question

Let's pretend that I have a network of 10,000 cars. I want to use all these machines to scan on the Internet as quickly as possible. All pages should be loaded only once. In addition, there should not be a single point of failure, and we must minimize the number of messages required between machines. How would you do that?

Is there anything more efficient than using sequential hashing to distribute the load on all machines and minimize communication between them?

Answer 1

• Map Reduction, Hadoop .
• , .
• , , , . URL-. Chord, - .

Answer 2

( ), , , .

\

\ \1

\ \...

\ \server10000

\

\

1.) , , , .

2.) , , , ,

3.) (1-N)

4.) , , , "" .

5.) , , , , .

6.) \Complete

7.) in-process server

8.) 1000 10 in-process , . , , .

Getrive, in-process complete 1 , hash\abc\def\123\ abcdef123FFFFFF..., .

mongo , , ..