Geek Answers Handbook

Big data - what is the best way to send it?

we have this scenario:

A server containing the necessary data and the client component that this data wants.

There are 2 types of data stored on the server: - some information - just a few lines mostly - binary data

We have a problem with getting binary data. Both sides are written in Java 5, so we have several ways ...

Web service is not the best solution because of speed, memory, etc.

So what would you prefer?

I would like to skip the connection to the low level connector if possible ...

early

Vitek

+4
source share
8 answers

I think the only way to do BIG data is to have access to a raw socket.

You remove memory problems in large files by most other methods.

Socket processing is really quite straightforward in Java, and it will allow you to transfer data without loading the entire file into memory (which happens behind the scenes without your own buffering).

Using this strategy, I was able to create a system that allowed transferring arbitrarily large files (I used a 7 + GB DVD image to test the system) without running into memory problems.

+5
source

Take a look at the standard WOMC MTOM for binary data transfer as part of the SOAP service. It is efficient in that it sends in binary code and can also send in buffered fragments. He will also interact with other clients or providers:

How to perform MTOM Interop

Server Side - Submitting Applications Using SOAP

+5
source

Perhaps you should take a look at protobuf , which is the library that Google uses to exchange data. Its very efficient and extensible. On the sidebar, never underestimate the bandwidth of a station wagon filled with 1 TB hard drives!

+2
source

I tried converting the binary data to Base64 and then send it via SOAP calls, and this worked for me. I do not know if this is considered a web service, but if so, then you are almost stuck in sockets.

+1
source

Some options:

  • You can use RMI, which will hide socket-level stuff and possibly gzip data for you ... but if the connection fails, it will not resume for you. There will probably also be memory issues.

  • just HTTP data with the binary mime type (again, perhaps setting up gzip on a web server). similar problem in resume.

  • type something like wget (I think it might resume)

  • If the client already has data (previous version), rsync will only copy the changes

+1
source

How about old, affordable, and reliable FTP? For example, you can easily embed an FTP server in your server components, and then encode the FTP client. FTP was born for just that (File Transfer Protocol, right?), While SOAP with attachments was not designed with this material in mind and can work very poorly. For example, you can see:

http://mina.apache.org/ftpserver/

But there are other implementations, Apache Mina is just the first I can remember.

Good luck and wishes

0
source

Is sneakernet an option ?: P

RMI is well known for its ease of use and memory leaks. Keep in mind. Depending on how much data we are talking about, sneakernet and sockets are both good options.

0
source

Consider GridFTP as your transport layer. See also this question .

0
source

More articles:


All Articles