Geek Answers Handbook

Should I worry about bit raids on Amazon S3?

I have some data that I want to save on Amazon S3 . Some of this data is encrypted, and some are compressed. Should I worry about one-time coups? I know an MD5 hash file that can be added. This (in my experience) will prevent the upheavals in the most unreliable part of the transaction (network communication), however I am still wondering if I need to protect against turning over on the disk?

+5
source share
4 answers

I am pretty sure that the answer is no, but if you want to be super paranoid, you can pre-calculate the MD5 hash before downloading, compare this to the MD5 hash that you get after loading, and then, when loading, calculate the MD5 hash of the downloaded data and compare them with a saved hash.

I'm not sure what risk you are worried about. At some point, you need to put off the risk to someone else. Is “corrupted data” covered by Amazon's SLA? Presumably, they know that there should be a file hash, and if the hash of the data that they give you does not match, then this is clearly their problem.

I believe there are other approaches:

  • Save your data with FEC so you can detect and correct N bit errors before you select N.
  • Amazon S3, , ( , ), RAID-, , - .

, , , .

+12

, .

, , - , . , 11.5 , Amazon

, , , . (..omiss..) , , , , , , .

, , . , ( - ), .

, , . , . ? : , Amazon, , ( , , -, )

+3

, : Amazon -, , , . , , .

S3 MD5 / . S3, MD5 . , MD5 . MD5 , , GET. .

, MD5, , . , , . , , , .

S3 S3, MD5 ( , ) MD5, GET .

: https://forums.aws.amazon.com/thread.jspa?threadID=38587

+1

:

  • " Amazon S3 ?"
  • " , Amazon S3 ?"

(1) "". , , - .

(2). , , , , , , . , CRC , , , , , . (, ZFS, , CRC raid controller. ZFS , , .)

Generally, you should check that your system is working as you expect. Using a hash function is a good approach. Which approach you take when you find a failure depends on your requirements. Storing multiple copies is probably the best approach (and, of course, the easiest), because you can get protection from site crashes, connection failures, and even provider failures (by choosing a second provider), and not just redundancy in the data itself using FEC .

-1
source

More articles:


All Articles