Geek Answers Handbook

RESTful API for SAN / NAS

I need to create a RESTful API for my files on cluster file system volumes. I have 20 servers that use the same file systems. All I need is RESTful API services that will allow me stat (), read (), write (), listFolder (), delete (), setacl (), etc. Everything else is handled by the cluster file system, so I just need to have the above functions. I need something very mature, so it supports access control lists, has a high-performance API (like java tags), a library or project is supported, and it runs Linux, and blocking support will be very useful. I would like to add additional functions like getDuration (), so if it opens the source code it will be useful. If you know about code that can help me build something like this, I would really appreciate it.

The goal is to let the BPM system check if the files are in order on different Stornext volumes. Since these systems are behind different firewalls, and NFS or SMB are not very good due to their high availability, the best option seems to be the RESTful API as a single source for all file operations between zones of the firewall in some convenient way by HTTP request (S) run NFS or SSH.

+7
source share
3 answers

If you want a very general web interface for managing files

Look at the design of the WebDAV api OK, if you do not want to use it AS IS, you just consider it as an inspiration from the API. See how stat() , listFolders() and setacl() can be just one command. If you look at something time-tested - this is one thing. This API was designed to access the web file, people put wrappers around it so that it can be mounted just like any other file system - see davfs2 , for me this is proof of a complete and complete API.

Assuming you don't need a complete DAV - but something simpler, I would take a look at some libraries that can help me create a similar API. Look at them: Jackrabbit WebDAV library , milton.io . There is also, of course, a jigsaw project for steel code. Use them to expose your APO APO or select StorNext API calls via http.

If you need a less generic block management API

Drop the Amazon S3 API as inspiration and code like littles3 as an example implementation. There are many such projects, check this search

Note that what you want is between what is already available:

  • webDAV (full stack, from APIs to server implementations) that hides and abstracts the main file system. Very high level, so you cannot use StorNext functions.
  • StorNext API, which is very low level, so there is no suitable web level.

If you need an API specifically designed for your domain

As a rule, when you encounter a similar problem, like yours, people use their knowledge and use in their domains. If you need this API to store and retrieve images, forget about common file actions and model your API around a collection of images. You know there is a lot of information ahead that makes the design of the API much simpler, for example:

  • min / max / average file size
  • usage patterns, read / write i / o
  • no streaming needed
  • immutability of the file content (without additional changes)
  • etc.
+4
source

I would recommend looking at the WebDAV implementation - they are usually integrated into a web server (for example, Apache) and support most of the standard file system operations that you need.

If you really want to create it yourself, you can also launch an object storage platform, such as OpenStack "Swift", with support for your SAN or NAS device via NFS / iSCSI.

EDIT . You want to save a large number of photos. There are various NoSQL databases that also solve this problem. However, you can also solve the problem using your own network file protocol, such as NFS.

NFS will work well (v4.1 + anyway) in most of your typical file system read and write operations. However, you'll also need a way to index and retrieve photo metadata and provide access control mechanisms, and that is where performance can get complicated.

When the file is uploaded to your HTTP API, you must calculate its MD5 hash content by storing the original file name, owner ID, and other metadata in the relational database. Then write a photo of your NFS mount in a specific “bucket”.

For example, suppose you have a photo with MD5 content: e240a38624f4a370bd2ec65cf771134b . Assuming your NFS mount is in /srv/content , you should write the photo to the path /srv/content/e240/a38624f4/a370bd2ec65cf771134b.jpg - split the MD5 hash to create prefix folders.

When your user later wants to get the image, he can request it through the data stored in the relational database, your API can find the MD5 hash file and then find it in the file system using a similar operation.

Remember that using MD5 can lead to conflicts if you have so many different files, so you can use a different hash scheme or a combination of two or more to prevent this from happening.

0
source

Have you looked at rails api? I'm not sure if it supports all the features you need, but it is also open source.

https://github.com/rails-api/rails-api

You can also include a ruby ​​stone to process access control lists.

https://www.ruby-toolbox.com/projects/acl9

0
source

More articles:


All Articles