M. does nothing here.
Your code builds the chain, one link at a time:
-:`(>:@(3&*))@.(2&|)^:(>&1)^:a:"0 M. 5 5 5 16 8 4 2 1 5 16 8 4 2 1
Here he remembers that 5 leads to 16, 16 leads to 8, 8 leads to 4, etc. But what does it do for you? It replaces some simple arithmetic with a memory search, but arithmetic is so trivial that it is more likely to be faster than a search. (I am surprised that your example takes 10 minutes, but this is not relevant.)
For the memories to make sense, you need to replace the more expensive calculations.
For this specific task, you may need a function that takes an integer and returns 1 if and when the sequence arrives at 1. For example:
-:`(>:@(3&*))@.(2&|)^:(>&1)^:_"0 M. 5 5 1 1
All I did was replace ^:a: with ^:_ to undo intermediate results. Even then, it doesn't really matter, but you can use timespacex to see the effect:
timespacex '-:`(>:@(3&*))@.(2&|)^:(>&1)^:_"0 i.100000' 17.9748 1.78225e7 timespacex '-:`(>:@(3&*))@.(2&|)^:(>&1)^:_"0 M. i.100000' 17.9625 1.78263e7
Addendum: Placing M. relative to "0 seems to make a huge difference. I thought I could make a mistake, but a quick test showed that replacing them caused a huge performance loss both in time and in space:
timespacex '-:`(>:@(3&*))@.(2&|)^:(>&1)^:_ M. "0 i.100000' 27.3633 2.41176e7
M. retains the rank of the basic verb, so the two are semantically equivalent, but I suspect that with "0 outside, like this, M. does not know that he deals with scalars. Therefore, I think the lesson here is to make sure M. knows what he is dealing with. :)
By the way, if the Collatz hypothesis turned out to be false, and you submitted this code to a counterexample, it would go into an infinite loop, and would not give an answer.
To actually detect a counterexample, you want to track intermediate results until you find a loop, and then return the lowest number in the loop. To do this, you probably want to implement a custom dialect to replace ^:n .