Can anyone provide an example of searching, reading and writing a file> 4 GB using boost iostreams

Question

I read that boosting iostreams supposedly supports 64-bit access to large files in a semi-portable way. Their FAQ mentions 64-bit offset functions , but there are no examples of how to use them. Has anyone used this library to process large files? A simple example of opening two files, finding their middle and copying one to the other would be very useful.

Thanks.

Answer 1

Short answer

Just turn it on

#include <boost/iostreams/seek.hpp> 

and use seek , as in

 boost::iostreams::seek(device, offset, whence); 

Where

• device is a file, stream, streambuf, or any object convertible to seekable ;
• offset is a 64-bit offset of type stream_offset ;
• whence - BOOST_IOS::beg , BOOST_IOS::cur or BOOST_IOS::end .

The return value of seek is of type std::streampos , and can be converted to stream_offset using position_to_offset .

Example

Here is a long, tedious and repetitive example that shows how to open two files, search for attachments> 4 GB and copy data between them.

A WARNING. This code will create very large files (several GB). Try this example on an OS / file system that supports sparse files. Linux is fine; I have not tested it on other systems such as Windows.

 /* * WARNING: This creates very large files (several GB) * unless your OS/file system supports sparse files. */ #include <boost/iostreams/device/file.hpp> #include <boost/iostreams/positioning.hpp> #include <cstring> #include <iostream> using boost::iostreams::file_sink; using boost::iostreams::file_source; using boost::iostreams::position_to_offset; using boost::iostreams::seek; using boost::iostreams::stream_offset; static const stream_offset GB = 1000*1000*1000; void setup() { file_sink out("file1", BOOST_IOS::binary); const char *greetings[] = {"Hello", "Boost", "World"}; for (int i = 0; i < 3; i++) { out.write(greetings[i], 5); seek(out, 7*GB, BOOST_IOS::cur); } } void copy_file1_to_file2() { file_source in("file1", BOOST_IOS::binary); file_sink out("file2", BOOST_IOS::binary); stream_offset off; off = position_to_offset(seek(in, -5, BOOST_IOS::end)); std::cout << "in: seek " << off << std::endl; for (int i = 0; i < 3; i++) { char buf[6]; std::memset(buf, '\0', sizeof buf); std::streamsize nr = in.read(buf, 5); std::streamsize nw = out.write(buf, 5); std::cout << "read: \"" << buf << "\"(" << nr << "), " << "written: (" << nw << ")" << std::endl; off = position_to_offset(seek(in, -(7*GB + 10), BOOST_IOS::cur)); std::cout << "in: seek " << off << std::endl; off = position_to_offset(seek(out, 7*GB, BOOST_IOS::cur)); std::cout << "out: seek " << off << std::endl; } } int main() { setup(); copy_file1_to_file2(); }