Best clock or number generator function for concurrency / scalability on Erlang OTP 18?

Question

Currently in the process of converting an erlang application from version 17 to 18. Scalability and performance are the main design guidelines. A program needs a way to differentiate and sort new input data, either with a lot of unique monotonically increasing numbers (a continuous stream of them), or some other mechanism. The current version (17) was not used now () for this, because it is a scalability bottleneck (global lock), therefore it happened due to reading the clock and performing other actions to create tags for incoming data. trying to find the best way to do this at the age of 18 and get some interesting test results that I performed.

I was expecting erlang: unique_integer ([monotonic]) to have bad results, because I was expecting it to have a global lock, like now (). I expected that one of the functions of the watch would have better results, suggesting that the watch could be read in parallel. Instead, erlang: unique_integer ([monotonic]) gets the best results from all the functions that I tested, and the watch functions are getting worse.

Can someone explain the results, tell me which erlang functions SHOULD give the best results, and which things (clocks, number generators, etc.) are globally locked at 18? In addition, if you see any problems with my testing methodology, be sure to list them.

TEST PLATFORM / METHODOLOGY

windows 7 64 bit erlang otp 18 (x64) 2 intel cores (celeron 1.8GHz) 2 erlang processes spawned to run each test function concurrently 500000 times for a total of 1000000 times, timed with timer:tc each test run 10 times in succession and all results recorded 

BASIC TEST SEQUENTIAL

 erlang:unique_integer([monotonic]) 47000-94000 

PARALLEL TIMES

 erlang:unique_integer([monotonic]) ~94000 ets:update_counter 450000-480000 erlang:monotonic_time 202000-218000 erlang:system_time 218000-234000 os:system_time 124000-141000 calendar:universal_time 453000-530000 

Answer 1

If you ask about the testing methodology, I would expect that you also include your code, because there may be a slight error in the reference code, which can ruin the result. So I write one and made a Gist so that we can compare the result using the same code. YMMV, especially because I use Linux and timers, is very OS dependent. There are my results:

 $ uname -a Linux hynek-notebook 4.1.0-1-amd64 #1 SMP Debian 4.1.3-1 (2015-08-03) x86_64 GNU/Linux $ grep 'model name' /proc/cpuinfo model name : Intel(R) Core(TM) i5 CPU M 520 @ 2.40GHz model name : Intel(R) Core(TM) i5 CPU M 520 @ 2.40GHz model name : Intel(R) Core(TM) i5 CPU M 520 @ 2.40GHz model name : Intel(R) Core(TM) i5 CPU M 520 @ 2.40GHz $ erl Erlang/OTP 18 [erts-7.0] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] Eshell V7.0 (abort with ^G) 1> c(test). {ok,test} 2> test:bench_all(1). [{unique_monotonic_integer,{38341,39804}}, {update_counter,{158248,159319}}, {monotonic_time,{217531,218272}}, {system_time,{224630,226960}}, {os_system_time,{53489,53691}}, {universal_time,{114125,116324}}] 3> test:bench_all(2). [{unique_monotonic_integer,{40109,40238}}, {update_counter,{307393,338993}}, {monotonic_time,{120024,121612}}, {system_time,{123634,124928}}, {os_system_time,{29606,29992}}, {universal_time,{177544,178820}}] 4> test:bench_all(20). [{unique_monotonic_integer,{23796,26364}}, {update_counter,{514835,527087}}, {monotonic_time,{91916,93662}}, {system_time,{94615,96249}}, {os_system_time,{27194,27598}}, {universal_time,{317353,340187}}] 5> 

The first thing I should note is that only erlang:unique_integer/0,1 and ets:update_counter/3,4,5 generates a unique value. Even erlang:monotonic_time/0 can generate two identical timestamps! Therefore, if you need a unique number, you have no choice but to use erlang:unique_integer/0,1 . If you want a unique monotone timestamp, you can use {erlang:monotonic_time(), erlang:unique_integer()} , or if you do not need a time part, you can use erlang:unique_integer([monotonic]) . If you do not need monotonous and unique, you can use other options. Therefore, if you need a unique monotonous number, there is only one good option, and it is erlang:unique_integer([monotonic]) .

The second time, I should note that spawning of two processes is not enough to test scalability. As you can see, when I use os:timestamp/0 with 20 processes, they start to catch up with erlang:unique_integer/0,1 . And there is one more problem. We both use HW with only two processors. Too scalable to check. Imagine how the result will look on an HW with 64 or more cores.

Edit : Using {write_concurrency, true} will improve ets:update_counter , but still far beyond erlang:unique_integer/0,1 .

 2> test:bench(test:update_counter(),1). {203830,213657} 3> test:bench(test:update_counter(),2). {129148,140627} 4> test:bench(test:update_counter(),20). {471858,501198} 

Answer 2

According to the erlang code base , erlang:unique_integer([monotonic]) is just an increase in the atomic integer. It works fast. Although this still creates a memory barrier, atomic work is still cheap compared to conventional global blocking.