I am doing very fast file access in ruby (2.0.0 p39474) and keep getting Too many open files exception
Having looked at this thread , here are various other sources, I know the OS limitations well (set to 1024 on my system).
The part of my code that performs this file access is a mutex and takes the form:
File.open( filename, 'w'){|f| Marshal.dump(value, f) }
where filename undergoes a quick change, depending on the thread invoking the section. I understand that this form discards its file descriptor after the block.
I can check the number of open File objects using ObjectSpace.each_object(File) . This reports that up to 100 are resident in memory, but only one of them is always open, as expected.
In addition, the exception itself is thrown at a time when there are only 10-40 File objects reported by ObjectSpace . In addition, manual garbage collection does not improve any of these indicators, and also slows down my script by inserting sleep calls.
My question is therefore:
I fundamentally misunderstand the nature of OS limitations - does this cover the whole process of the process?If so, how can web servers avoid crashes after accessing ulimit -n files?Is ruby storing its file descriptors outside its object system, or is the kernel just really slow in counting "simultaneous" access?
Edit 20130417: strace indicates that Ruby does not write all of its data to a file, returning and freeing the mutex before doing this. Thus, the file processes the stack to the limit of the OS.
In an attempt to fix this, I used syswrite / sysread , synchronous mode and called flush to close . None of these methods worked.
My question is thus revised to: Why does ruby fail to close its file descriptors and how can I get it to do this?