Erlang/OTP 24 Highlights

May 12, 2021 · by Lukas Larsson

Finally Erlang/OTP 24 is here! A release that for me has been about 10 years in the making. As is tradition by now, this blog post will go through the additions to Erlang/OTP that I am most excited about!

Erlang/OTP 24 includes contributions from 60+ external contributors totalling 1400+ commits, 300+ PRs and changing 0.5 million(!) lines of code. Though I’m not sure the line number should count as we vendored all of AsmJit and re-generated the wxWidgets support. If we ignore AsmJit and wx, there are still 260k lines of code added and 320k lines removed, which is about 100k more than what our releases normally contain.

You can download the readme describing the changes here: Erlang/OTP 24 Readme. Or, as always, look at the release notes of the application you are interested in. For instance here: Erlang/OTP 24 - Erts Release Notes - Version 12.0.

This years highlights are:

BeamAsm - the JIT compiler for Erlang #

The most anticipated feature of Erlang/OTP 24 has to be the JIT compiler. A lot has already been said about it:

and even before released the WhatsApp team has shown what it is capable of.

However, besides the performance gains that the JIT brings, what I am the most excited about is the benefits that come with running native code instead of interpreting. What I’m talking about is the native code tooling that now becomes available to all Erlang programmers, such as integration with perf.

As an example, when building a dialyzer plt of a small core of Erlang, the previous way to profile would be via something like eprof.

> eprof:profile(fun() ->
    dialyzer:run([{analysis_type,'plt_build'},{apps,[erts]}])
  end).

This increases the time to build the PLT from about 1.2 seconds to 15 seconds on my system. In the end, you get something like the below that will guide you to what you need to optimize. Maybe take a look at erl_types:t_has_var*/1 and check if you really need to call it 13-15 million times!

> eprof:analyze(total).
FUNCTION                      CALLS        %     TIME [uS / CALLS]
--------                      -----  -------     ---- [----------]
erl_types:t_sup1/2          2744805     1.68   752795 [      0.27]
erl_types:t_subst/2         2803211     1.92   858180 [      0.31]
erl_types:t_limit_k/2       3783173     2.04   913217 [      0.24]
maps:find/2                 4798032     2.14   957223 [      0.20]
erl_types:t_has_var/1      15943238     5.89  2634428 [      0.17]
erl_types:t_has_var_list/1 13736485     7.51  3360309 [      0.24]
------------------------  ---------  ------- -------- [----------]
Total:                    174708211  100.00% 44719837 [      0.26]

In Erlang/OTP 24 we can get the same result without having to pay the pretty steep cost of profiling with eprof. When running the same analysis as above using perf it takes roughly 1.3 seconds to run.

$ ERL_FLAGS="+JPperf true" perf record dialyzer --build_plt \
    --apps erts

Then we can use tools such as perf report, hotspot or speedscope to analyze the results.

$ hotspot perf.data

alt text

In the above, we can see that we get roughly the same result as when using eprof, though interestingly not exactly the same. I’ll leave the whys of this up to the reader to find out :)

With this little overhead when profiling, we can run scenarios that previously would take too long to run when profiling. For those brave enough it might even be possible to run always-on profiling in production!

The journey with what can be done with perf has only started. In PR-4676 we will be adding frame pointer support which will give a much more accurate call frames when profiling and, in the end, the goal is to have mappings to Erlang source code lines instead of only functions when using perf report and hotspot to analyze a perf recording.

Improved error messages #

Erlang’s error messages tend to get a lot of (valid) criticism for being hard to understand. Two great new features have been added to help the user understand why something has failed.

Column number in warnings and errors #

Thanks to the work of Richard Carlsson and Hans Bolinder, when you compile Erlang code you now get the line and column of errors and warnings printed in the shell together with a ^-sign showing exactly where the error actually was. For example, if you compile the below:

foo(A, B) ->
  #{ a => A, b := B }.

you would in Erlang/OTP 23 and earlier get:

$ erlc t.erl
t.erl:6: only association operators '=>' are allowed in map construction

but in Erlang/OTP 24 you now also get the following printout:

$ erlc test.erl
t.erl:6:16: only association operators '=>' are allowed in map construction
%    6|   #{ a => A, b := B }.
%     |                ^

This behavior also extends into most of the Erlang code editors so that when you use VSCode or Emacs through Erlang LS or flycheck you also get a narrower warning/error indicator, for example in Emacs using Erlang LS.

alt text

EEP-54: Improved BIF error information #

One of the other big changes when it comes to error information is the introduction of EEP-54. In the past many of the BIFs (built-in functions) would give very cryptic error messages:

1> element({a,b,c}, 1).
** exception error: bad argument
     in function  element/2
        called as element({a,b,c},1)

In the example above, the only thing we know is that one or more of the arguments are invalid, but without checking the documentation there is no way of knowing which one and why. This is especially a problem for BIFs where the arguments may fail for different reasons depending on factors not visible in the arguments. For example in the ets:update_counter call below:

> ets:update_counter(table, k, 1).
** exception error: bad argument
     in function  ets:update_counter/3
        called as ets:update_counter(table,k,1)

We don’t know if the call failed because the table did not exist at all or if the key k that we wanted to update did not exist in the table.

In Erlang/OTP 24 both of the examples above will have a much clearer error messages.

1> element({a,b,c}, 1).
** exception error: bad argument
     in function  element/2
        called as element({a,b,c},1)
        *** argument 1: not an integer
        *** argument 2: not a tuple
2> ets:new(table,[named_table]).
table
3> ets:update_counter(table, k, 1).
** exception error: bad argument
     in function  ets:update_counter/3
        called as ets:update_counter(table,k,1)
        *** argument 2: not a key that exists in the table

That looks much better and now we can see what the problem was! The standard logging formatters also include the additional information so that if this type of error happens in a production environment you will get the extra error information:

1> proc_lib:spawn(fun() -> ets:update_counter(table, k, 1) end).
<0.94.0>
=CRASH REPORT==== 10-May-2021::11:20:35.367023 ===
  crasher:
    initial call: erl_eval:'-expr/5-fun-3-'/0
    pid: <0.94.0>
    registered_name: []
    exception error: bad argument
      in function  ets:update_counter/3
         called as ets:update_counter(table,k,1)
         *** argument 1: the table identifier does
                         not refer to an existing ETS table
    ancestors: [<0.92.0>]

EEP-54 is not only useful for error messages coming from BIFs but can be used by any application that wants to provide extra information about their exceptions. For example, we have been working on providing better error information around io:format in PR-4757.

Improved receive optimizations #

Since Erlang/OTP R14 (released in 2010), the Erlang compiler and run-time system have co-operated to optimize for the pattern of code used by gen_server:call like functionality to avoid scanning a potentially huge mailbox. The basic pattern looks like this:

call(To, Msg) ->
  Ref = make_ref(),
  To ! {call, Ref, self(), Msg},
  receive
    {reply, Ref, Reply} -> Reply
  end.

The compiler can from this figure out that when Ref is created, there can be no messages in the mailbox of the process that contains Ref and therefore it can skip all of those when receiving the Reply.

This has always worked great in simple scenarios like this, but as soon as you had to make the scenarios a little more complex it tended to break the compiler’s analysis and you would end up scanning the entire mailbox. For example, in the code below Erlang/OTP 23 will not optimize the receive.

call(To, Msg, Async) ->
  Ref = make_ref(),
  To ! {call, Ref, self(), Msg},
  if
    Async ->
      {ok, Ref};
    not Async ->
      receive
        {reply, Ref, Reply} -> Reply
      end
  end.

That all changes with Erlang/OTP 24! Many more complex scenarios are now covered by the optimization and a new compiler flag has been added to tell the user if an optimization is done.

$ erlc +recv_opt_info test.erl
test.erl:6: Warning: OPTIMIZED: reference used to mark
                                a message queue position
%    6|   Ref = make_ref(),
test.erl:12: Warning: OPTIMIZED: all clauses match reference
                                 created by make_ref/0
                                 at test.erl:6
%   12|       receive

Even patterns such as multi_call are now optimized to not scan the mailbox of the process.

multi_call(ToList, Msg) ->
  %% OPTIMIZED: reference used to mark a message queue position
  Ref = make_ref(),
  %% INFO: passing reference created by make_ref/0 at test.erl:18
  [To ! {call, Ref, self(), Msg} || To <- ToList],
  %% INFO: passing reference created by make_ref/0 at test.erl:18
  %% OPTIMIZED: all clauses match reference
  %%            in function parameter 2
  [receive {reply, Ref, Reply} -> Reply end || _ <- ToList].

There are still a lot of places where this optimization does not trigger. For instance as soon as any of the make_ref/send/receive are in different modules it will not work. However, the new improvements in Erlang/OTP 24 make the number of scenarios a lot fewer and now we also have the tools to check and see if the optimization is triggered!

You can read more about this optimization and others in the Efficiency Guide.

EEP-53: Process aliases #

When doing a call to another Erlang process, the pattern used by gen_server:call, gen_statem:call and others normally looks something like this:

call(To, Msg, Tmo) ->
  MonRef = erlang:monitor(process, To),
  To ! {call, MonRef, self(), Msg},
  receive
    {'DOWN',MonRef,_,_,Reason} ->
      {error, Reason};
    {reply, MonRef, Reply}
      erlang:demonitor(MonRef,[flush]),
      {ok, Reply}
    after Tmo ->
      erlang:demonitor(MonRef,[flush]),
      {error, timeout}
  end.

This normally works well except for when a timeout happens. When a timeout happens the process on the other end has no way to know that the reply is no longer needed and so will send it anyway when it is done with it. This causes all kinds of problems as the user of a third-party library would never know what messages to expect to be present in the mailbox.

There have been numerous attempts to solve this problem using the primitives that Erlang gives you, but in the end, most ended up just adding a handle_info in their gen_servers that ignored any unknown messages.

In Erlang/OTP 24, EEP-53 has introduced the alias functionality to solve this problem. An alias is a temporary reference to a process that can be used to send messages to. In most respects, it works just as a PID except that the lifetime of an alias is not tied with the lifetime of the process it represents. So when you try to send a late reply to an alias that has been deactivated the message will just be dropped.

The code changes needed to make this happen are very small and are already used behind the scenes in all the standard behaviors of Erlang/OTP. The only thing needed to be changed in the example code above is that a new option must be given to erlang:monitor and the reply reference should now be the alias instead of the calling PID. That is, like this:

call(To, Msg, Tmo) ->
  MonAlias = erlang:monitor(process, To, [{alias, demonitor}]),
  To ! {call, MonAlias, MonAlias, Msg},
  receive
    {'DOWN', MonAlias, _ , _, Reason} ->
      {error, Reason};
    {reply, MonAlias, Reply}
      erlang:demonitor(MonAlias,[flush]),
      {ok, Reply}
    after Tmo ->
      erlang:demonitor(MonAlias,[flush]),
      {error, timeout}
  end.

You can read more about this functionality in the alias documentation.

EEP-48: Documentation chunks for edoc #

In Erlang/OTP 23 erl_docgen was extended to be able to emit EEP-48 style documentation. This allowed the documentation to be used by h(lists) in the Erlang shell and external tools such as Erlang LS. However, there are very few applications outside Erlang/OTP that use erl_docgen to create documentation, so EEP-48 style documentation was unavailable to those applications. Until now!

Radek Szymczyszyn has added support for EEP-48 into edoc which means that from Erlang/OTP 24 you can view both the documentation of lists:foldl/3 and recon:info/1.

$ rebar3 as docs shell
Erlang/OTP 24 [erts-12.0] [source] [jit]

Eshell V11.2.1  (abort with ^G)
1> h(recon,info,1).
 -spec info(PidTerm) ->
   [{info_type(), [{info_key(), Value}]}, ...]
     when PidTerm :: pid_term().

  Allows to be similar to erlang:process_info/1, but excludes
  fields such as the mailbox, which tend to grow
  and be unsafe when called in production systems. Also includes
  a few more fields than what is usually given (monitors,
  monitored_by, etc.), and separates the fields in a more
  readable format based on the type of information contained.

For more information about how to enable this in your project see the Doc chunks section in the Edoc User’s Guide.

socket support in gen_tcp #

The gen_tcp module has gotten support for optionally using the new socket nif API instead of the previous inet driver. The new interface can be configured to be used either on a system level through setting the application configuration parameter like this: -kernel inet_backend socket, or on a per connection bases like this: gen_tcp:connect(localhost,8080,[{inet_backend,socket}]).

If you do this you will notice that the Socket returned by gen_tcp no longer is a port but instead of a tuple containing (among other things) a PID and a reference.

1> gen_tcp:connect(localhost,8080,[{inet_backend,socket}]).
{ok,{'$inet',gen_tcp_socket,
             {<0.88.0>,{'$socket',#Ref<0.2959644163.2576220161.68602>}}}}

This data structure is and always has been opaque, and therefore should not be inspected directly but instead only used as an argument to other gen_tcp and inet functions.

You can then use inet:i/0 to get a listing of all open sockets in the system:

2> inet:i().
Port      Module         Recv Sent Owner    Local Address   Foreign Address    State Type   
esock[19] gen_tcp_socket 0    0    <0.98.0> localhost:44082 localhost:http-alt CD:SD STREAM

The gen_tcp API should be completely backward compatible with the old implementation, so if you can, please test it and report any bugs that you find back to us.

Why should you want to test this? Because in some of our benchmarks, we get up to 4 times the throughput vs the old implementation. In others, there is no difference or even a loss of throughput. So, as always, you need to measure and check for yourself!

EEP-56: Supervisor automatic shutdown #

When creating supervisor hierarchies for applications that manage connections such as ssl or ssh, there are times when there is a need for terminating that supervisor hierarchy from within. Some event happens on the socket that should trigger a graceful shutdown of the processes associated with the connection.

Normally this would be done by using supervisor:terminate_child/2. However, this has two problems.

  1. It requires the child to know the ID of the child that needs to be terminated and the PID of the supervisor to talk to. This is simple when there is just one process in the supervisor, but when there are supervisors under supervisors, this becomes harder and harder to figure out.
  2. Calling supervisor:terminate_child/2 is a synchronous operation. This means that if you do the call in the child, you may end up in a deadlock as the top supervisor wants to terminate the child while the child is blocking in the call to terminate itself.

To solve this problem EEP-56 has added a mechanism in which a child can be marked as significant and if such a child terminates, it can trigger an automatic shutdown of the supervisor that it is part of.

This way a child process can trigger the shutdown of a supervisor hierarchy from within, without the child having to know anything about the supervisor hierarchy nor risking dead-locking itself during termination.

You can read more about automatic shutdown in the supervisor documentation.

Edwards-curve Digital Signature Algorithm #

With Erlang/OTP 24 comes support for Edwards-curve Digital Signature Algorithm (EdDSA). EdDSA can be used when connecting to or acting as a TLS 1.3 client/server.

EdDSA is a type of elliptic curve signature algorithm (ECDSA) that can be used for secure communication. The security of ECDSA relies on a strong cryptographically secure random number which can cause issues when the random number is by mistake not secure enough, as has been the case in several uses of ECDSA (none of them in Erlang as far as we know :).

EdDSA does not rely on a strong random number to be secure. This means that when you are using EdDSA, the communication is secure even if your random number generator is not.

Despite the added security, EdDSA is claimed to be faster than other elliptic curve signature algorithms. If you have OpenSSL 1.1.1 or later, then as of Erlang/OTP 24 you will have access to this algorithm!

> crypto:supports(curves).
[...
  c2tnb359v1, c2tnb431r1, ed25519, ed448, ipsec3, ipsec4
 ...]                     ^        ^