How many concurrent readers can pthread_rwlock have?

Question

I have a multi-threaded application that creates 48 threads that all need to access a common attribute (stl :: map). The card will be recorded only when the streams begin, and the rest of the time the card will be read. This seems like the perfect use case for pthread_rw_lock, and everything seems to work well.

I ran into a completely unrelated seg error and started analyzing the kernel. Using gdb, I ran the info threads command and was very surprised by the results. I noticed that multiple threads are actually reading from the map, as expected, but the weird part is that multiple threads were blocked in pthread_rwlock_rdlock (), waiting on rw_lock.

Here is the stack trace for a thread waiting to block:

 #0 0xffffe430 in __kernel_vsyscall () #1 0xf76fe159 in __lll_lock_wait () from /lib/libpthread.so.0 #2 0xf76fab5d in pthread_rwlock_rdlock () from /lib/libpthread.so.0 #3 0x0804a81a in DiameterServiceSingleton::getDiameterService(void*) () 

With so many threads, it’s hard to say how many were read and how many were blocked, but I don’t understand why any threads will be blocked, waiting for reading, given that other threads are already reading.

So, here is my question: why are some threads blocked to wait for rw_lock to read when other threads are already reading from it? It seems that there is a limit to the number of threads that can be read at one time.

Ive looked at the pthread_rwlock_attr_t functions and did not see anything related.

OS is Linux, SUSE 11.

Here is the related code:

 { pthread_rwlock_init(&serviceMapRwLock_, NULL); } // This method is called for each request processed by the threads Service *ServiceSingleton::getService(void *serviceId) { pthread_rwlock_rdlock(&serviceMapRwLock_); ServiceMapType::const_iterator iter = serviceMap_.find(serviceId); bool notFound(iter == serviceMap_.end()); pthread_rwlock_unlock(&serviceMapRwLock_); if(notFound) { return NULL; } return iter->second; } // This method is only called when the app is starting void ServiceSingleton::addService(void *serviceId, Service *service) { pthread_rwlock_wrlock(&serviceMapRwLock_); serviceMap_[serviceId] = service; pthread_rwlock_unlock(&serviceMapRwLock_); } 

Update:

As mentioned in MarkB's comments, if I set pthread_rwlockattr_getkind_np () to give priority to writers, and there is a wait lock script, then the observed behavior would make sense. But, I use the default value, which, I believe, gives priority to readers. I just checked that there are no threads blocked for writing. I also update the code suggested by @Shahbaz in the comments and get the same results.

Answer 1

You just noticed the inherent performance issues associated with acquiring locks. It takes some time, and you just managed to catch these threads in the middle of it. This is especially true if the operation protected by the lock has a very short duration.

Edit: Reading the source, glibc uses lll_lock to protect critical sections inside its own pthread library data structures. pthread_rwlock_rdlock checks for several flags and increments counters, so it does these things by holding the lock. Once this is done, the lock will be released using lll_unlock .

To demonstrate, I performed a short routine that sleeps after acquiring rwlock . The main thread is waiting for them to complete. But before you wait, it prints the concurrency reached by the threads.

 enum { CONC = 50 }; pthread_rwlock_t rwlock; pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER; pthread_cond_t cond = PTHREAD_COND_INITIALIZER; unsigned count; void *routine(void *arg) { int *fds = static_cast<int *>(arg); pthread_rwlock_rdlock(&rwlock); pthread_mutex_lock(&mutex); ++count; if (count == CONC) pthread_cond_signal(&cond); pthread_mutex_unlock(&mutex); sleep(5); pthread_rwlock_unlock(&rwlock); pthread_t self = pthread_self(); write(fds[1], &self, sizeof(self)); return 0; } 

And the main thread waits until the counter reaches 50:

 int main() { int fds[2]; pipe(fds); pthread_rwlock_init(&rwlock, 0); pthread_mutex_lock(&mutex); for (int i = 0; i < CONC; i++) { pthread_t tid; pthread_create(&tid, NULL, routine, fds); } while (count < CONC) pthread_cond_wait(&cond, &mutex); pthread_mutex_unlock(&mutex); std::cout << "count: " << count << std::endl; for (int i = 0; i < CONC; i++) { pthread_t tid; read(fds[0], &tid, sizeof(tid)); pthread_join(tid, 0); } pthread_rwlock_destroy(&rwlock); pthread_exit(0); } 

Edit: Simplified example of using C ++ 11 thread support:

 enum { CONC = 1000 }; std::vector<std::thread> threads; pthread_rwlock_t rwlock; std::mutex mutex; std::condition_variable cond; unsigned count; void *routine(int self) { pthread_rwlock_rdlock(&rwlock); { std::unique_lock<std::mutex> lk(mutex); if (++count == CONC) cond.notify_one(); } sleep(5); pthread_rwlock_unlock(&rwlock); return 0; } int main() { pthread_rwlock_init(&rwlock, 0); { std::unique_lock<std::mutex> lk(mutex); for (int i = 0; i < CONC; i++) { threads.push_back(std::thread(routine, i)); } cond.wait(lk, [](){return count == CONC;}); } std::cout << "count: " << count << std::endl; for (int i = 0; i < CONC; i++) { threads[i].join(); } pthread_rwlock_destroy(&rwlock); pthread_exit(0); } 

Answer 2

As a note, the code above is broken. You cannot access iter-> second from the rw_lock'd section, because as soon as you unlock rw_lock, the writer can delete any element on the map, thereby canceling any iterator on it.

I know that you do not do this in your case, since you do not write anything before starting the program, but it’s worth mentioning anyway.

Also, as a side note, since the behavior of your description is similar to serialization (authors write on the map at the beginning, readers now read “read-only”), you should probably write this like this:

 int writerDoneWithMap = 0; // pthread_cond & mutex init here // The writer write to the map here // Then they signal the reader that they are done with it while (!__sync_bool_compare_and_swap(&writerDoneWithMap, 1, writerDoneWithMap)); pthread_cond_broadcast here // The readers will then simply do this: while (!writerDoneWithMap) { // pthread_cond_wait here } // Read the data without locks. 

The above code avoids blocking on readers if the writer has finished filling out the card, otherwise you will resort to the typical pthread_cond / mutex technique. The above code is correct if and only if you have THEN readers (but that is what you said), otherwise it will fail.