Why is the switching function called twice when changing the ref link in Clojure?

Question

I think I understand the main difference between the ideas of switching and changes in a Clojure transaction.

Essentially “blocks” the identifier from the beginning to the end of the transaction, so that multiple transactions must be performed sequentially.

Commute only applies the lock to the actual change of values ​​for the identifier, so that other actions in the transaction can be performed at different times and with different representations of the world.

But I'm embarrassed by something. Let me define a function with side effects and a reflector:

(defn fn-with-side-effects [state] (println "Hello!") (inc state)) (def test-counter (ref 0)) 

Now, if we use alter, we see the expected behavior:

 user=> (dosync (alter test-counter fn-with-side-effects)) Hello! 1 

But if we use dial-up:

 user=> (dosync (ref-set test-counter 0)) 0 user=> (dosync (commute test-counter fn-with-side-effects)) Hello! Hello! ; hello is printed twice! 1 

Thus, in the switched version, the function explicitly changes only once, since the final value is 1. But the side effects of the modifier function are executed twice. Why is this happening?

Answer 1

I get it.

This is because switching functions are always performed twice .

Commute allows for more potential concurrency than alter because it does not block credentials throughout the transaction.

Instead, it reads the identifier value once at the start of the transaction, and when the switching operation is called, it returns the switching function applied to THIS VALUE.

It is possible that this value is deprecated, because some other thread could change it after some time between the start of the transaction and the execution of the commute function.

However, integrity is maintained because the commute function is executed AGAIN at the time of commit, when it actually changes the link.

This site has a very clear explanation of the difference: http://squirrel.pl/blog/2010/07/13/clojure-alter-vs-commute/

In fact, when a dial-up is called, it instantly returns the result of a function by reference. At the very end of the transaction, it performs the calculation again, this time synchronously (like alter) updating the link. That's why in the end the counter value is 51, although the last thread 45 is printed.

So be careful if your switching function has side effects, because they will be executed twice !!

Answer 2

I did some experiments to understand how commute works. I would like to break my explanation into 3 parts:

• Compare and set semantics
• alter
• commute

Compare and set semantics

I think Clojure for the Brave and True explained this pretty well:

swap! implements "compare-and-set" semantics, meaning it does the following internally:

1. He reads the current state of the atom
2. He then applies the update function to this state.
3. Then it checks to see if the value read in step 1 matches the current atom value.
4. If so, then swap! updates the atom, referring to the result of step 2
5. If it is not, then swap! retries, repeating the process again from step 1.

swap! designed for atom , but knowing this will help us understand alter and commute , because they use a similar method to update ref .

Unlike atom , the modification of ref (via alter , commute , ref-set ) must be enclosed in a transaction. When a transaction is started (or repeated), it takes a snapshot of everything containing ref (because alter needs it). ref will be changed only after the transaction.

alter

In a transaction, all ref that will be modified using alter form a group. If any of the ref in the group cannot change it, the transaction will be repeated. Basically alter does the following:

1. Compare its change to ref with the snapshot taken by the transaction. If they look different, repeat the transaction; yet
2. Create a new state from the snapshot using the provided function.
3. Compare the ref to the snapshot again. If they look different, repeat the transaction; yet
4. Try to lock the ref entry, do not let anyone modify it until the end of this transaction. If a failure occurs ( ref already locked), wait a while (for example, 100 ms), then repeat the transaction.
5. Tell the transaction to update this ref in a new state when it is executing a commission.

Let me show you a smooth change. First we are going to create thread t1 for alter 3 counters c1 , c2 and c3 with slow-inc .

 (ns testing.core) (def start (atom 0)) ; Record start time. (def c1 (ref 0)) ; Counter 1 (def c2 (ref 0)) ; Counter 2 (def c3 (ref 0)) ; Counter 3 (defn milliTime "Get current time in millisecond." [] (int (/ (System/nanoTime) 1000000))) (defn lap "Get elapse time since 'start' in millisecond." [] (- (milliTime) @start)) (defn slow-inc "Slow increment, takes 1 second." [x x-name] (println "slow-inc beg" x-name ":" x "|" (lap) "ms") (Thread/sleep 1000) (println "slow-inc end" x-name ":" (inc x) "|" (lap) "ms") (inc x)) (defn fast-inc "Fast increment. The value it prints is incremented." [x x-name] (println "fast-inc " x-name ":" (inc x) "|" (lap) "ms") (inc x)) (defn -main [] ;; Initialize c1, c2, c3 and start. (dosync (ref-set c1 0) (ref-set c2 0) (ref-set c3 0)) (reset! start (milliTime)) ;; Start two new threads simultaneously. (let [t1 (future (dosync (println "transaction start |" (lap) "ms") (alter c1 slow-inc "c1") (alter c2 slow-inc "c2") (alter c3 slow-inc "c3") (println "transaction end |" (lap) "ms"))) t2 (future)] ;; Dereference all of them (wait until all 2 threads finish). @t1 @t2 ;; Print final counters' values. (println "c1 :" @c1) (println "c2 :" @c2) (println "c3 :" @c3))) 

And we got this:

 transaction start | 3 ms ; 1st try slow-inc beg c1 : 0 | 8 ms slow-inc end c1 : 1 | 1008 ms slow-inc beg c2 : 0 | 1009 ms slow-inc end c2 : 1 | 2010 ms slow-inc beg c3 : 0 | 2010 ms slow-inc end c3 : 1 | 3011 ms transaction end | 3012 ms c1 : 1 c2 : 1 c3 : 1 

The smooth procedure. Not surprising.

Let's see what happens if ref (say c3 ) is changed before it changes ( (alter c3 ...) ). We will change it during c1 change. Change the let binding for t2 to:

 t2 (future (Thread/sleep 900) ; Increment at 900 ms (dosync (alter c3 fast-inc "c3"))) 

Result:

 transaction start | 2 ms ; 1st try slow-inc beg c1 : 0 | 7 ms fast-inc c3 : 1 | 904 ms ; c3 being modified in thread t2 slow-inc end c1 : 1 | 1008 ms slow-inc beg c2 : 0 | 1009 ms slow-inc end c2 : 1 | 2010 ms transaction start | 2011 ms ; 2nd try slow-inc beg c1 : 0 | 2011 ms slow-inc end c1 : 1 | 3012 ms slow-inc beg c2 : 0 | 3013 ms slow-inc end c2 : 1 | 4014 ms slow-inc beg c3 : 1 | 4015 ms slow-inc end c3 : 2 | 5016 ms transaction end | 5016 ms c1 : 1 c2 : 1 c3 : 2 

As you can see, in step 1 in the 1st attempt (alter c3 ...) he realizes that c3 (val = 1) looks different than the snapshot (val = 0) captured by the transaction, so it repeats the transaction.

Now, what if ref (say c1 ) was changed during its change ( (alter c1 ...) )? We will change c1 in thread t2 . Change the let binding for t2 to:

 t2 (future (Thread/sleep 900) ; Increment at 900 ms (dosync (alter c1 fast-inc "c1"))) 

Result:

 transaction start | 3 ms ; 1st try slow-inc beg c1 : 0 | 8 ms fast-inc c1 : 1 | 904 ms ; c1 being modified in thread t2 slow-inc end c1 : 1 | 1008 ms transaction start | 1009 ms ; 2nd try slow-inc beg c1 : 1 | 1009 ms slow-inc end c1 : 2 | 2010 ms slow-inc beg c2 : 0 | 2011 ms slow-inc end c2 : 1 | 3011 ms slow-inc beg c3 : 0 | 3012 ms slow-inc end c3 : 1 | 4013 ms transaction end | 4014 ms c1 : 2 c2 : 1 c3 : 1 

This time, at step 3 of the 1st attempt - (alter c1 ...) he found that the ref was changed, so he causes a retry of the transaction.

Now let's try changing ref (say c1 ) after changing it ( (alter c1 ...) ). We will change it during c2 change.

 t2 (future (Thread/sleep 1600) ; Increment at 1600 ms (dosync (alter c1 fast-inc "c1"))) 

Result:

 transaction start | 3 ms ; 1st try slow-inc beg c1 : 0 | 8 ms slow-inc end c1 : 1 | 1009 ms slow-inc beg c2 : 0 | 1010 ms fast-inc c1 : 1 | 1604 ms ; try to modify c1 in thread t2, but failed fast-inc c1 : 1 | 1705 ms ; keep trying... fast-inc c1 : 1 | 1806 ms fast-inc c1 : 1 | 1908 ms fast-inc c1 : 1 | 2009 ms slow-inc end c2 : 1 | 2011 ms slow-inc beg c3 : 0 | 2012 ms fast-inc c1 : 1 | 2110 ms ; still trying... fast-inc c1 : 1 | 2211 ms fast-inc c1 : 1 | 2312 ms fast-inc c1 : 1 | 2413 ms fast-inc c1 : 1 | 2514 ms fast-inc c1 : 1 | 2615 ms fast-inc c1 : 1 | 2716 ms fast-inc c1 : 1 | 2817 ms fast-inc c1 : 1 | 2918 ms ; and trying.... slow-inc end c3 : 1 | 3012 ms transaction end | 3013 ms ; 1st try ended, transaction committed. fast-inc c1 : 2 | 3014 ms ; finally c1 modified successfully c1 : 2 c2 : 1 c3 : 1 

Since the 1st attempt (alter c1 ...) blocked c1 (step 4), no one can change c1 until this round of trial transactions ends.

What is this alter .

So, what if we do not want c1 , c2 , c3 all groups together? Suppose I want to repeat a transaction only when c1 or c3 could not change them (mutable by another thread during the transaction). I don't care about state c2 . There is no need to repeat the transaction if c2 was changed during the transaction, so that I can save time. How do we achieve this? Yes, through commute .

commute

Essentially, commute does the following:

1. Run the provided function using ref directly (not from the snapshot), but do nothing with the result.
2. Ask the transaction to call real-commute with the same arguments before the transaction is committed. ( real-commute is just a name coined by me.)

Actually, I don’t know why commute should perform step 1. It seems to me that one step 2 is enough. real-commute does the following:

1. Read-and-write-lock ref until the end of this test transaction, if it has not been locked, otherwise retry the transaction.
2. Create a new state from ref using this function.
3. Tell the transaction to update this ref in a new state when performing a commission.

Let's look at it. Change the let binding to:

 t1 (future (dosync (println "transaction start |" (lap) "ms") (alter c1 slow-inc "c1") (commute c2 slow-inc "c2") ; changed to commute (alter c3 slow-inc "c3") (println "transaction end |" (lap) "ms"))) t2 (future) 

Result:

 transaction start | 3 ms slow-inc beg c1 : 0 | 7 ms ; called by alter slow-inc end c1 : 1 | 1008 ms slow-inc beg c2 : 0 | 1009 ms ; called by commute slow-inc end c2 : 1 | 2009 ms slow-inc beg c3 : 0 | 2010 ms ; called by alter slow-inc end c3 : 1 | 3011 ms transaction end | 3012 ms slow-inc beg c2 : 0 | 3012 ms ; called by real-commute slow-inc end c2 : 1 | 4012 ms c1 : 1 c2 : 1 c3 : 1 

Thus, slow-inc is called twice if you use commute , once commute and once real-commute before committing the transaction. The first commute did nothing with the result of slow-inc .

slow-inc can be called more than two times. For example, let's try changing c3 in thread t2 :

 t2 (future (Thread/sleep 500) ; modify c3 at 500 ms (dosync (alter c3 fast-inc "c3"))) 

Result:

 transaction start | 2 ms slow-inc beg c1 : 0 | 8 ms fast-inc c3 : 1 | 504 ms ; c3 modified at thread t2 slow-inc end c1 : 1 | 1008 ms slow-inc beg c2 : 0 | 1009 ms ; 1st time slow-inc end c2 : 1 | 2010 ms transaction start | 2012 ms slow-inc beg c1 : 0 | 2012 ms slow-inc end c1 : 1 | 3013 ms slow-inc beg c2 : 0 | 3014 ms ; 2nd time slow-inc end c2 : 1 | 4015 ms slow-inc beg c3 : 1 | 4016 ms slow-inc end c3 : 2 | 5016 ms transaction end | 5017 ms slow-inc beg c2 : 0 | 5017 ms ; 3rd time slow-inc end c2 : 1 | 6018 ms c1 : 1 c2 : 1 c3 : 2 

In the first test of the transaction, after evaluating (commute c2 ...) , (alter c3 ...) found that c3 is different from the snapshot, thus causing the transaction to retry. If (alter c3 ...) preceded by (commute c2 ...) , then a re-check will be run before the evaluation or (commute c2 ..) . Thus, placing all commute after all alter can save you time.

Let's see what happens if we change c2 in thread t2 during transaction evaluation in t1 .

 t2 (future (Thread/sleep 500) ; before evaluation of (commute c2 ...) (dosync (alter c2 fast-inc "c2")) (Thread/sleep 1000) ; during evaluation of (commute c2 ...) (dosync (alter c2 fast-inc "c2")) (Thread/sleep 1000) ; after evaluation of (commute c2 ...) (dosync (alter c2 fast-inc "c2"))) 

Result:

 transaction start | 3 ms slow-inc beg c1 : 0 | 9 ms fast-inc c2 : 1 | 504 ms ; before slow-inc end c1 : 1 | 1009 ms slow-inc beg c2 : 1 | 1010 ms fast-inc c2 : 2 | 1506 ms ; during slow-inc end c2 : 2 | 2011 ms slow-inc beg c3 : 0 | 2012 ms fast-inc c2 : 3 | 2508 ms ; after slow-inc end c3 : 1 | 3013 ms transaction end | 3013 ms slow-inc beg c2 : 3 | 3014 ms slow-inc end c2 : 4 | 4014 ms c1 : 1 c2 : 4 c3 : 1 

As you can see, the transaction was not retried, and c2 is still updated to the expected value (4), thanks to real-commute .

Now I want to demonstrate the effect of step 1 in real-commute : its ref locked for reading and writing. First confirm that it is locked for reading:

 t2 (future (Thread/sleep 3500) ; during real-commute (println "try to read c2:" @c2 " |" (lap) "ms")) 

Result:

 transaction start | 3 ms slow-inc beg c1 : 0 | 9 ms slow-inc end c1 : 1 | 1010 ms slow-inc beg c2 : 0 | 1010 ms slow-inc end c2 : 1 | 2011 ms slow-inc beg c3 : 0 | 2012 ms slow-inc end c3 : 1 | 3012 ms transaction end | 3013 ms slow-inc beg c2 : 0 | 3013 ms slow-inc end c2 : 1 | 4014 ms try to read c2: 1 | 4015 ms ; got printed after transaction trial ended c1 : 1 c2 : 1 c3 : 1 

@c2 locks until c2 is unlocked. That's why println is evaluated after 4000 ms, even if our order is inactive for 3500 ms.

Since commute and alter must read their ref order to perform this function, they will be blocked until their ref is unlocked. You can try replacing (println ...) with (alter c2 fast-inc "c2") . The effect should be the same as in this example.

So, to confirm that it is write-locked, we can use ref-set :

 t2 (future (Thread/sleep 3500) ; during real-commute (dosync (ref-set c2 (fast-inc 9 " 8")))) 

Result:

 transaction start | 3 ms slow-inc beg c1 : 0 | 8 ms slow-inc end c1 : 1 | 1008 ms slow-inc beg c2 : 0 | 1010 ms slow-inc end c2 : 1 | 2011 ms slow-inc beg c3 : 0 | 2012 ms slow-inc end c3 : 1 | 3013 ms transaction end | 3014 ms slow-inc beg c2 : 0 | 3014 ms fast-inc 8 : 9 | 3504 ms ; try to ref-set but failed fast-inc 8 : 9 | 3605 ms ; try again... fast-inc 8 : 9 | 3706 ms fast-inc 8 : 9 | 3807 ms fast-inc 8 : 9 | 3908 ms fast-inc 8 : 9 | 4009 ms slow-inc end c2 : 1 | 4015 ms fast-inc 8 : 9 | 4016 ms ; finally success, c2 ref-set to 9 c1 : 1 c2 : 9 c3 : 1 

From here you can also guess what the ref-set does:

• If its ref locked from writing, repeat the transaction after a while (for example, 100 ms); also order transactions to update this ref to the given value when executing the commission.

real-commute can also fail when its ref blocked in step 1. Unlike alter or ref-set , it does not wait for a while before retrying the transaction. This can cause problems if ref locked for too long. For example, we will try to change c1 after changing it using commute :

 t2 (future (Thread/sleep 2500) ; during alteration of c3 (dosync (commute c1 fast-inc "c1"))) 

Result:

 transaction start | 3 ms slow-inc beg c1 : 0 | 8 ms slow-inc end c1 : 1 | 1008 ms slow-inc beg c2 : 0 | 1010 ms slow-inc end c2 : 1 | 2011 ms slow-inc beg c3 : 0 | 2012 ms fast-inc c1 : 1 | 2506 ms fast-inc c1 : 1 | 2506 ms fast-inc c1 : 1 | 2506 ms ... Exception in thread "main" java.util.concurrent.ExecutionException: java.lang.RuntimeException: Transaction failed after reaching retry limit, compiling: ... 

Recall that c1 locked from the alter record after changing it, so real-commute continues to fail and real-commute transaction. Without buffer time, he reached the limit of transaction retries and thundered.

Note

commute helps improve concurrency by allowing the user to reduce ref , which will cause a transaction to retry, while the cost of calling this function is at least doubled to update its ref . There are several scenarios that commute may be slower than alter . For example, when the only thing to do in a transaction is to update ref , commute will cost more than alter :

 (def c (ref 0)) ; counter (defn slow-inc [x] (Thread/sleep 1000) (inc x)) (defn add-2 "Create two threads to slow-inc c simultaneously with func. func can be alter or commute." [func] (let [t1 (future (dosync (func c slow-inc))) t2 (future (dosync (func c slow-inc)))] @t1 @t2)) (defn -main [& args] (dosync (ref-set c 0)) (time (add-2 alter)) (dosync (ref-set c 0)) (time (add-2 commute))) 

Result:

 "Elapsed time: 2003.239891 msecs" ; alter "Elapsed time: 4001.073448 msecs" ; commute 

Here are the alter procedures:

• 0 ms: t1 alter running.
• 1 ms: t2 alter running.
• 1000 ms: t1 alter successfully, t1 fixed, c became 1.
• 1001 ms: t2 alter detected that c is different from its snapshot (step 2), repeat the transaction.
• 2001 ms: t2 alter successful, t2 fixed, c became 2.

And the commute :

• 0 ms: t1 commute running.
• 1 ms: t2 commute running.
• 1000 ms: t1 real-commute running. c blocked.
• 1001 ms: t2 real-commute running. He found that c locked, so he retried the transaction (step 1).
• 1002 ms: t2 commute running, but c locked, so it is locked.
• 2000 ms: t1 real-commute completed, transaction completed. c became 1. t2 unlocked.
• 3002 ms: t2 real-commute running.
• 4002 ms: t2 real-commute completed, transaction completed. c became 2.

This is why commute slower than alter in this example.

This may seem contrary to the example of commuting from clojuredocs.org. The main difference is that in his example, a delay (100 ms) occurred in the body of the transaction, but the delay occurred in slow-inc in my example. This difference leads to the fact that its real-commute is very fast, which reduces the lock time and lock time. Less lock time means less chance of re-review. That's why in his example, commute faster than alter . Change it inc to slow-inc and you will get the same observation as mine.

It's all.