Looking for a way to make short read on Linux

Question

I am looking for a way to create short reads in linux, so I can unit test process the code around them.

I have several methods that, at lower levels, call pread / pread64 to read from a file on the file system. They are designed to handle situations when a short read occurs (the number of bytes read is less than the requested number).

I have seen situations where short reads occur (via network file systems).

Ideally, I could create a file that will read N bytes, and then a short read of M bytes will be read, and then a normal read, as expected. This would allow unit tests to point to a file / file system.

Thanks!

Answer 1

If you know that the library call that you want to intercept, you can insert into the call with a shared object loaded through LD_PRELOAD .

shortread.c:

 #include <sys/types.h> #include <dlfcn.h> #define MAX_FDS 1024 static int short_read_array[ MAX_FDS ]; // #define these to match your system values // (need to be really careful with header files since // getting open() declared would make things very // difficult - just try this with open( const char *, int, ...); // declared to see what I mean...) #define O_RDONLY 0 #define O_WRONLY 1 #define O_RDWR 2 // note that the mode bits for read/write are // not a bitwise-or - they are distinct values #define MODE_BITS 3 // it much easier to *NOT* even deal with the // fact that open() is a varargs function // but that means probably having to do some // typedef and #defines to get this to compile // typedef some function points to make things easier typedef int ( *open_ptr_t )( const char *name, int flags, mode_t mode ); typedef ssize_t ( *read_ptr_t )( int fd, void *buf, size_t bytes ); typedef int ( *close_ptr_t )( int fd ); // function points to the real IO library calls static open_ptr_t real_open = NULL; static read_ptr_t real_read = NULL; static close_ptr_t real_close = NULL; // this will return non-zero if 'filename' is a file // to cause short reads on static int shortReadsOnFd( const char *filename ) { // add logic here based on the file name to // return non-zero if you want to do // short reads on this file // // return( 1 ); return( 0 ); } // interpose on open() int open( const char *filename, int flags, mode_t mode ) { static pthread_mutex_t open_mutex = PTHREAD_MUTEX_INITIALIZER; int fd; pthread_mutex_lock( &open_mutex ); if ( NULL == real_open ) { real_open = dlsym( RTLD_NEXT, "open" ); } pthread_mutex_unlock( &open_mutex ); fd = real_open( filename, flags, mode ); if ( ( -1 == fd ) || ( fd >= MAX_FDS ) ) { return( fd ); } int mode_bits = flags & MODE_BITS; // if the file can be read from, check if this is a file // to do short reads on if ( ( O_RDONLY == mode_bits ) || ( O_RDWR == mode_bits ) ) { short_read_array[ fd ] = shortReadsOnFd( filename ); } return( fd ); } ssize_t read( int fd, void *buffer, size_t bytes ) { static pthread_mutex_t read_mutex = PTHREAD_MUTEX_INITIALIZER; if ( ( fd < MAX_FDS ) && ( short_read_array[ fd ] ) ) { // read less bytes than the caller asked for bytes /= 2; if ( 0 == bytes ) { bytes = 1; } } pthread_mutex_lock( &read_mutex ); if ( NULL == real_read ) { real_read = dlsym( RTLD_NEXT, "read" ); } pthread_mutex_unlock( &read_mutex ); return( real_read( fd, buffer, bytes ) ); } int close( int fd ) { static pthread_mutex_t close_mutex = PTHREAD_MUTEX_INITIALIZER; pthread_mutex_lock( &close_mutex ); if ( NULL == real_close ) { real_close = dlsym( RTLD_NEXT, "close" ); } pthread_mutex_unlock( &close_lock ); if ( fd < MAX_FDS ) { short_read_array[ fd ] = 0; } return( real_close( fd ) ); } 

Compile something like:

 gcc -shared [-m32|-m64] shortread.c -o libshortread.so 

Then:

 export LD_PRELOAD=/path/to/libshortread.so 

Be extremely careful with such LD_PRELOAD - all processes in the process tree will be forced to load the library. A 32-bit process will not work if it needs to load a 64-bit library, as well as a 64-bit process that will be forced to load a 32-bit library. You can add the init function to the source above, which removes the LD_PRELOAD environment variable (or sets it to something harmless) to control it a bit.

You also probably need to be careful if any application uses the O_DIRECT flag for open() . Changing the number of bytes read can violate direct I / O for some Linux file systems and / or implementations, since only page-size I / O can be supported.

And this code only processes read() . You may also have to deal with creat() . Also pread() , readat() , aio_read() and lio_listio() (and maybe even a few others that I can't remember at the moment), although this is admittedly not very likely. And be careful with 32-bit processes that process large files. It has been some time since I talked to them, but it can get ugly, as I recall.

Another caveat is that calls like fopen() and fread() cannot call open() and read() library calls and can issue the corresponding system call directly. In this case, you cannot easily change the behavior of these calls. Interacting with the entire STDIO-based call family that can read data like fgets() can be very difficult without breaking anything.

And if you know that your applications are single-threaded, you can opt out of mutexes.

Answer 2

I ended up going with a solution using mkfifo() .

I create a named pipe, then attach a writer to it (and end it in the JNI library that will be used with Java). Then, the asynchronous recorder may be asked to write data at the correct time, after which the connected reader will receive only the available / written bytes, and not the total number requested.