My last post on this topic caught some attention, so I’m going to continue exploring some of the systemd features that may be useful to people writing network services. Here are some more things about socket activation I didn’t cover in the previous post.
Firstly, it’s not something that needs a binding to a specific library to work (although systemd does make one available for this task). It uses conventions that already exist in Unix, and have done for a long time. The network sockets are presented as file descriptors, and some information about them as environment variables.
As I mentioned in the previous post, socket activation is pretty similar to how
inetd is a “super-server”, which first appeared in 4.3BSD in 1986. There are other implementations of the same idea–
xinetd is a common one using a different configuration format.
The idea is that it listens on a set of ports normally configured in
/etc/inetd.conf, and then any time a new connection arrives on that port, it will spawn a new instance of the configured process and send the incoming data to its standard input, and the process’s output back to the client. An example configuration for the Quote of the Day service in the original
inetd may look like:
qotd stream tcp nowait root /usr/sbin/tcpd /usr/games/fortune
The fields are:
- The service name. This references a service list in
/etc/services. In this case,
qotdmaps to port 17.
- It’s a
streamservice operating over TCP, as opposed to a
dgramservice operating on UDP
nowaitmeans to spawn a new instance of the process for every connection, instead of waiting for a single process to deal with each connection
- This service is configured to run as root, which seems like an awful idea but may possibly be required for
- The final fields are the command to run on each incoming connection.
tcpdis a program that does some basic filtering of incoming traffic. I’ll skip over this for the moment.
Using systemd like inetd
We can use systemd for the same purpose as
inetd, by configuring it to create processes on demand and connecting the network socket to standard input and output.
Let’s try it out, and use some of the features I talked about before to protect the service, as the
fortune command isn’t really designed to be connected to a network so we don’t want it to get exploited. I’m on Ubuntu 21.04 here, but this should work on most distributions.
Create a file called
/etc/systemd/system/fortune@.service, and put the following in it:
[Unit] QoTD [Service] -/usr/games/fortune socket yes true true [Install] multi-user.target
Note the comments–some of the features I talked about before are enabled automatically when choosing
DynamicUser=true. Also note the
@ in the filename - this is significant as it indicates the service is a template, and that a new instance of the service will be run on every connection.
Then put this in
[Socket] 17 yes [Install] sockets.target
After this, you’ll need to do
systemctl daemon-reload followed by
systemctl start fortune.socket (and
systemctl enable fortune.socket if you want it to start on boot).
Now, we’ve built a QOTD service–port 17 is the IANA registered port number for this service. To try it out, you can use
nc or perhaps
mgdm@io:~$ nc localhost 17 Never trust an operating system.
It’s good advice!
Socket presentation in systemd
Socket activation using systemd’s own convention is similar to
inetd, but has more room for expansion. The sockets themselves are presented to the process as file descriptors, similar to the inetd approach. The main difference that systemd does not use the standard streams, to allow it to supply more than one socket. It uses file descriptors numbered 3 and above. You can configure more than just the one I demonstrated above. If you specify more than one, they’ll be presented in the same order as they appear in the
In order for the process to be able to understand what to do with the sockets, a couple of environment variables are used. Firstly,
LISTEN_FDS gives the number of sockets configured. For example, if this is set to
2, the process will be able to work out that it has been given listening sockets on FDs 3 and 4.
It’s handy that you can configure several listening ports, but you also need to be able to work out which is which. Some software may want to listen on more than one port, and it may or may not be obvious which protocol should be configured on each–consider a hypothetical web server that may want to present plaintext HTTP on port 80 and HTTPS on port 443. In this case it’s probably quite easy to work out which service needs to go where from the port numbers, but regardless, it is possible to supply a name for each socket to make this slightly more foolproof. A variable called
LISTEN_FDNAMES can be set with a colon-separated list of the name to be given to each socket.
lunchd from my previous post, and in the process finish the exercise I left at the end, by making it listen on both HTTP and HTTPS. On the HTTP port, we’ll just redirect every request we get to HTTPS.
lunchd.socket looked like this:
[Socket] 443 both [Install] sockets.target
We need to modify it slightly and then rename it. Add the extra
FileDescriptorName lines so it looks like below, and then rename it to
[Socket] 443 both https lunchd.service [Install] sockets.target
Then, copy it to
lunchd-http.service and change the port number there to 80 and the
[Socket] 80 both http lunchd.service [Install] sockets.target
We now have two
.socket files which is required if we want to present more than one different class of listening socket. By default, the names handed to the process for each socket type will be the same as the
.socket files, but in this case we’re using the
FileDescriptorName option to rename them.
Next up, we need to make a little change to the actual
lunchd code. The CoreOS library I’ve been using since the last post knows to look for the
LISTEN_FDS variable to find out how many file descriptors have been presented, and also has the ability to look at
LISTEN_FDNAMES to group them by name. To make use of this latter feature we’re going to switch to using
activation.ListenersWithNames() instead of
activation.Listeners(). This returns a
map[string]net.Listener. The keys in the map are the names configured in each
FileDescriptorName, and the values are a slice of
net.Listeners (as you can configure multiple
ListenStream options in each file to listen on several interfaces or ports).
This code is now reaching the point where I should be refactoring it a little into more than one file, but I’ll keep it as one here just so everything’s on the same page. The key changes are that instead of assuming it’s going to listen on one port, it now looks into the map to work out what to listen on, and spins up a new
http.ServeMux for each. Any sockets presented in the key
http will listen on plain HTTP, and those in
https will run TLS. Any other keys in the map are ignored. In the case where the map is empty (indicating the process is not being run using systemd, or not with socket activation), it’ll default to an unprivileged port (8443). In this way the code still runs happily on other Unix systems such as macOS.
I’ve added a little more logging to this code, so you can see what the environment variables look like when systemd starts the process. In this case, I had port 80 configured for HTTP and port 443 configured for HTTPS, so
LISTEN_FDNAMES looked like the below. The first line indicates there are 2 sockets, and the second denotes the protocols for each.
Jul 18 11:30:23 io lunchd: 2021/07/19 11:30:23 LISTEN_FDS is 2 Jul 18 11:30:23 io lunchd: 2021/07/19 11:30:23 LISTEN_FDNAMES is http:https
- This post is quite heavily inspired by systemd for Developers I by Lennart Poettering
- Thanks to Kevin McDermott and James O’Gorman for some advice on the Go code on both this and my last post
1 package main 2 3 import ( 4 "crypto/tls" 5 "errors" 6 "flag" 7 "fmt" 8 "log" 9 "math/rand" 10 "net" 11 "net/http" 12 "sync" 13 "time" 14 15 "github.com/coreos/go-systemd/activation" 16 ) 17 18 var lunchOptions = string