Improving the speed of azure light storage requests

Question

We currently have a blob repository with thousands of files in one Azure container. The file naming convention looks something like this:

StorageName \ Command \ SubTeam \ FileName

I am writing a tool that displays files for each specific subelement. The code gets a list of blocks for the container, and then for each of them it tries to match the correct \ Subteam command (see the code example below).

This works, but very slowly (because I need to go through all the files to see if they correspond to a particular subtopic). Is there a way to improve query speed? I might think of optimizations such as "Find the first file that matches the command you are looking for, and then track when you find another command to exit it earlier," but this assumes that the BlobList will be sorted and will not be fixed worst case scenario.

Unfortunately, splitting files in different containers is currently not an option.

Here is a sample code:

IEnumerable<IListBlobItem> blobs = blobContainer.ListBlobs( new BlobRequestOptions() { UseFlatBlobListing = true, BlobListingDetails = BlobListingDetails.Metadata }).OfType<CloudBlob>(); foreach (var blob in blobs) { var cloudy = blob as CloudBlob; string blobTeamId = cloudy.Uri.Segments[2].Trim('/'); if (blobTeamId != teamId) continue; //Do something interesting with the file 

Answer 1

1st solution With the REST interface you can go through

 http://somwhere.com/mycontainername/?restype=container&comp=list&delimiter=/&prefix=\Team\SubTeam 

and this will return the xml-document only with the files in the "Folder" subcommand (I know that this is not a folder, but it looks like one of the tools)

You may need to create a shared signature in order to access it, you should mark it at the end of the URL.

read here

Where it shows that you can filter the blobname prefix.

2nd solution This is probably closer to what you want. If you can use the new repository client that was updated in azure sdk 1.3, now you can use

IEnumerable blobList = client.ListBlobsWithPrefix ("Team / SubTeam"); A.

Where Client is an instance of CloudBlobClient.

EDIT - November 18, 2013 it seems that resttype is no longer supported as a parameter, and it should be repeated. It seems to have happened calmly on the weekend. I changed the url example above.

Answer 2

Just an update ...

You can use the blob list with GetDirectoryRefence and then the blobs list ...

var subDirectory = blobContainer.GetDirectoryReference (String.Format ("{0} /", folder)); return subDirectory.ListBlobs (false, BlobListingDetails.Metadata);

Answer 3

Do you really need BlobListDetails.Metadata? which leads to the loading of a large amount of additional information. I think all you need is a name