How can I limit the amount of data locked in memory when using mmap?

Question

In my application, I mmap a very large file and access locations randomly. However, after a certain number of accesses, the memory is fully used, and the program starts trashing.

I tried madvise(fd, range, MADV_RANDOM)and madvise(fd, range, MADV_DONT_NEED)for each access, but it does not work. I also tried to setrlimit.RLIMIT_MEMLOCKlimit the amount of memory locked in RAM.