If you only allocate inside the heap (you know, if you allocate outside the heap), use (Gc.quick_stat()).Gc.heap_words to get the heap size, a good approximation for all the memory used by OCaml. As the name implies, this field is expressed in either 32- or 64-bit words.
You must compare this with some limit that OCaml cannot guess for you, this is the maximum memory that you want to use in your program. The value depends on how much memory you want to leave for other processes, regardless of whether you consider it acceptable for sharing, and so on. This second value depends on your system, not OCaml. Even if you want to calculate this value as a fixed fraction of the total amount of RAM on your computer, I don’t know any portable way to do this with OCaml: you have to tell us more about your system.
Do not try to use free_words and similar fields in GC stats . They will tell you how the space is divided inside the heap . "Free" words still take their place, and the role of the garbage collector is to preserve this value at a reasonable fraction of everything. He resizes the heap to do this.