Windows Azure Cloud Storage - the impact of a huge number of files in the root

Question

Sorry if I am mistaken in any terminology, but I hope you get what I mean.

I use Windows Azure Cloud Storage to store a huge number of small files (images of 20 KB each).

These files are currently stored in the root directory. I understand that this is not a regular file system, so maybe root is not the right term.

I tried to find information on the long-term consequences of this plan, but no luck, so if anyone could give me some information, I would be grateful.

Basically, I ran into problems if the number of files stored in this root turned out to be in the hundreds of thousands / millions?

Thanks,

Stephen

Answer 1

I was in a similar situation when we stored ~ 10 M small files in one blob container. Code-specific access to individual files was good and there were no performance issues.

In cases where we had problems, it was managing a large number of files outside the code. If you use the storage explorer (the one that comes with VS2010 or any other), the ones I came across do not support return files using the API prefix, you can only list the first 5K, then the next 5K and so on. You can see how this can be a problem if you want to see the 125,000th file in the container.

Another problem is that there is no easy way to find out how many files are in your container (which may be important in order to know exactly how much of this blob repository is worth to you) without writing something that just iterates over all the drops and counts them.

This was a simple problem for us, because our blobs had sequential numerical names, so we just divided them into folders of 1,000 items each. Depending on the number of items you have, you can group 1K of these folders into subfolders.

Answer 2

http://social.msdn.microsoft.com/Forums/en-US/windowsazure/thread/d569a5bb-c4d4-4495-9e77-00bd100beaef

Short answer: No

Middle answer: Kindof?

Long answer: No, but if you request a list of files, it will return 5000. You will need to require every 5k to get a complete list according to this MSDN page.

Edit: The root is great for describing it. 99.99% of people will remember what you are trying to say.