My last post on this topic caught some attention, so I’m going to continue exploring some of the systemd features that may be useful to people writing network services. Here are some more things about socket activation I didn’t cover in the previous post.

Firstly, it’s not something that needs a binding to a specific library to work (although systemd does make one available for this task). It uses conventions that already exist in Unix, and have done for a long time. The network sockets are presented as file descriptors, and some information about them as environment variables.

inetd

As I mentioned in the previous post, socket activation is pretty similar to how inetd operates. inetd is a “super-server”, which first appeared in 4.3BSD in 1986. There are other implementations of the same idea–xinetd is a common one using a different configuration format.

The idea is that it listens on a set of ports normally configured in /etc/inetd.conf, and then any time a new connection arrives on that port, it will spawn a new instance of the configured process and send the incoming data to its standard input, and the process’s output back to the client. An example configuration for the Quote of the Day service in the original inetd may look like:

qotd stream tcp nowait root /usr/sbin/tcpd /usr/games/fortune

The fields are:

1. The service name. This references a service list in /etc/services. In this case, qotd maps to port 17.
2. It’s a stream service operating over TCP, as opposed to a dgram service operating on UDP
3. nowait means to spawn a new instance of the process for every connection, instead of waiting for a single process to deal with each connection
4. This service is configured to run as root, which seems like an awful idea but may possibly be required for tcpd.
5. The final fields are the command to run on each incoming connection. tcpd is a program that does some basic filtering of incoming traffic. I’ll skip over this for the moment.

Using systemd like inetd

We can use systemd for the same purpose as inetd, by configuring it to create processes on demand and connecting the network socket to standard input and output.

Let’s try it out, and use some of the features I talked about before to protect the service, as the fortune command isn’t really designed to be connected to a network so we don’t want it to get exploited. I’m on Ubuntu 21.04 here, but this should work on most distributions.

Create a file called /etc/systemd/system/fortune@.service, and put the following in it:

[Unit]
Description=QoTD

[Service]
# Note the - to make systemd ignore the exit code
ExecStart=-/usr/games/fortune

# This is the part that makes it work like inetd
StandardOutput=socket

# Run as a dynamic user
DynamicUser=yes

# ProtectSystem=strict # Implied by DynamicUser=yes
# PrivateTmp=true # Also implied by DynamicUser=yes

# DynamicUser implies ProtectHome=read-only, but we do not need home directories
ProtectHome=true

# We also do not need to see users
PrivateUsers=true


[Install]
WantedBy=multi-user.target

Note the comments–some of the features I talked about before are enabled automatically when choosing DynamicUser=true. Also note the @ in the filename - this is significant as it indicates the service is a template, and that a new instance of the service will be run on every connection.

Then put this in /etc/systemd/system/fortune.socket:

[Socket]
# Listen on TCP port 17 
# Use ListenDatagram to listen on UDP
ListenStream = 17

# Call accept(3) on the socket before handing it to the process
# This is necessary because fortune knows nothing about the network
Accept=yes

[Install]
WantedBy = sockets.target

After this, you’ll need to do systemctl daemon-reload followed by systemctl start fortune.socket (and systemctl enable fortune.socket if you want it to start on boot).

Now, we’ve built a QOTD service–port 17 is the IANA registered port number for this service. To try it out, you can use nc or perhaps telnet:

mgdm@io:~$ nc localhost 17
Never trust an operating system.

It’s good advice!

Socket presentation in systemd

Socket activation using systemd’s own convention is similar to inetd, but has more room for expansion. The sockets themselves are presented to the process as file descriptors, similar to the inetd approach. The main difference that systemd does not use the standard streams, to allow it to supply more than one socket. It uses file descriptors numbered 3 and above. You can configure more than just the one I demonstrated above. If you specify more than one, they’ll be presented in the same order as they appear in the .socket file.

In order for the process to be able to understand what to do with the sockets, a couple of environment variables are used. Firstly, LISTEN_FDS gives the number of sockets configured. For example, if this is set to 2, the process will be able to work out that it has been given listening sockets on FDs 3 and 4.

It’s handy that you can configure several listening ports, but you also need to be able to work out which is which. Some software may want to listen on more than one port, and it may or may not be obvious which protocol should be configured on each–consider a hypothetical web server that may want to present plaintext HTTP on port 80 and HTTPS on port 443. In this case it’s probably quite easy to work out which service needs to go where from the port numbers, but regardless, it is possible to supply a name for each socket to make this slightly more foolproof. A variable called LISTEN_FDNAMES can be set with a colon-separated list of the name to be given to each socket.

Example

Let’s extend lunchd from my previous post, and in the process finish the exercise I left at the end, by making it listen on both HTTP and HTTPS. On the HTTP port, we’ll just redirect every request we get to HTTPS.

The original lunchd.socket looked like this:

[Socket]
ListenStream = 443
BindIPv6Only = both

[Install]
WantedBy = sockets.target

We need to modify it slightly and then rename it. Add the extra Service and FileDescriptorName lines so it looks like below, and then rename it to lunchd-https.socket.

[Socket]
ListenStream = 443
BindIPv6Only = both
FileDescriptorName = https
Service=lunchd.service

[Install]
WantedBy = sockets.target

Then, copy it to lunchd-http.socket and change the port number there to 80 and the FileDescriptorName to http.

[Socket]
ListenStream = 80
BindIPv6Only = both
FileDescriptorName = http
Service=lunchd.service

[Install]
WantedBy = sockets.target

We now have two .socket files which is required if we want to present more than one different class of listening socket. By default, the names handed to the process for each socket type will be the same as the .socket files, but in this case we’re using the FileDescriptorName option to rename them.

Next up, we need to make a little change to the actual lunchd code. The CoreOS library I’ve been using since the last post knows to look for the LISTEN_FDS variable to find out how many file descriptors have been presented, and also has the ability to look at LISTEN_FDNAMES to group them by name. To make use of this latter feature we’re going to switch to using activation.ListenersWithNames() instead of activation.Listeners(). This returns a map[string][]net.Listener. The keys in the map are the names configured in each FileDescriptorName, and the values are a slice of net.Listeners (as you can configure multiple ListenStream options in each file to listen on several interfaces or ports).

This code is now reaching the point where I should be refactoring it a little into more than one file, but I’ll keep it as one here just so everything’s on the same page. The key changes are that instead of assuming it’s going to listen on one port, it now looks into the map to work out what to listen on, and spins up a new http.ServeMux for each. Any sockets presented in the key http will listen on plain HTTP, and those in https will run TLS. Any other keys in the map are ignored. In the case where the map is empty (indicating the process is not being run using systemd, or not with socket activation), it’ll default to an unprivileged port (8443). In this way the code still runs happily on other Unix systems such as macOS.

I’ve added a little more logging to this code, so you can see what the environment variables look like when systemd starts the process. In this case, I had port 80 configured for HTTP and port 443 configured for HTTPS, so LISTEN_FDS and LISTEN_FDNAMES looked like the below. The first line indicates there are 2 sockets, and the second denotes the protocols for each.

Jul 18 11:30:23 io lunchd[10425]: 2021/07/19 11:30:23 LISTEN_FDS is 2
Jul 18 11:30:23 io lunchd[10425]: 2021/07/19 11:30:23 LISTEN_FDNAMES is http:https

The code is below, but it’s also available in a branch in the GitHub repo from last time.

References

• This post is quite heavily inspired by systemd for Developers I by Lennart Poettering
• Thanks to Kevin McDermott and James O’Gorman for some advice on the Go code on both this and my last post

Final code

1
package main
2

3
import (
4
	"crypto/tls"
5
	"errors"
6
	"flag"
7
	"fmt"
8
	"log"
9
	"math/rand"
10
	"net"
11
	"net/http"
12
	"sync"
13
	"time"
14

15
	"github.com/coreos/go-systemd/activation"
16
)
17

18
var lunchOptions = []string{
19
	"Sandwich",
20
	"Soup",
21
	"Salad",
22
	"Burger",
23
	"Sushi",
24
}
25

26
func getRandomLunch() string {
27
	return lunchOptions[rand.Intn(len(lunchOptions))]
28
}
29

30
func getDefaultListeners() (map[string][]net.Listener, error) {
31

32
	listener, err := net.Listen("tcp", ":8443")
33

34
	if err != nil {
35
		return nil, err
36
	}
37

38
	return map[string][]net.Listener{
39
		"https": {listener},
40
	}, nil
41
}
42

43
func getListeners() (map[string][]net.Listener, error) {
44
	listeners, err := activation.ListenersWithNames()
45

46
	if err != nil || len(listeners) == 0 {
47
		log.Printf("Received no listeners from socket activation, defaulting to HTTPS on port 8443")
48
		listeners, err = getDefaultListeners()
49
	}
50

51
	return listeners, err
52
}
53

54
func getCertificatePaths() (string, string, error) {
55
	keyPath := flag.String("key", "", "The path to the private key")
56
	certPath := flag.String("certificate", "", "The path to the certificate")
57
	flag.Parse()
58

59
	if *keyPath == "" || *certPath == "" {
60
		return "", "", errors.New("Either or both of -key or -certificate not set")
61
	}
62

63
	return *keyPath, *certPath, nil
64
}
65

66
func getTLSConfig(keyPath string, certPath string) (*tls.Config, error) {
67
	config := &tls.Config{
68
		Certificates:             make([]tls.Certificate, 1),
69
		NextProtos:               []string{"h2", "http/1.1"},
70
		PreferServerCipherSuites: true,
71
	}
72

73
	var err error
74

75
	log.Printf("Loading certs from key: %s and cert: %s", keyPath, certPath)
76

77
	config.Certificates[0], err = tls.LoadX509KeyPair(
78
		certPath,
79
		keyPath,
80
	)
81

82
	if err != nil {
83
		log.Printf("Failed to configure TLS: %s", err)
84
		return nil, err
85
	}
86

87
	return config, nil
88
}
89

90
func createWebServers() (*http.ServeMux, *http.ServeMux) {
91
	httpMux := http.NewServeMux()
92
	httpMux.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
93
		log.Printf("HTTP request from %s\n", req.RemoteAddr)
94
		hostname := fmt.Sprintf("https://%s", req.Host)
95
		http.Redirect(w, req, hostname+req.RequestURI, http.StatusMovedPermanently)
96
	})
97

98
	httpsMux := http.NewServeMux()
99
	httpsMux.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
100
		log.Printf("HTTPS request from %s\n", req.RemoteAddr)
101
		fmt.Fprintf(w, "<h1>%s</h1>", getRandomLunch())
102
	})
103

104
	return httpMux, httpsMux
105
}
106

107
func main() {
108
	rand.Seed(time.Now().UnixNano())
109

110
	log.Printf("LISTEN_FDS is %s\n", os.Getenv("LISTEN_FDS"))
111
	log.Printf("LISTEN_FDNAMES is %s\n", os.Getenv("LISTEN_FDNAMES"))
112

113
	listeners, err := getListeners()
114
	httpMux, httpsMux := createWebServers()
115

116
	if err != nil {
117
		log.Fatalf("Could not set up listeners: %s", err)
118
	}
119

120
	// This is used to block the main goroutine and wait for the others 
121
	var wg sync.WaitGroup
122

123
	if tlsListeners, ok := listeners["https"]; ok {
124
		keyPath, certPath, err := getCertificatePaths()
125

126
		if err != nil {
127
			log.Fatalf("Could not load certificates: %s", err)
128
		}
129

130
		tlsConfig, err := getTLSConfig(keyPath, certPath)
131

132
		for i := range tlsListeners {
133
			wg.Add(1)
134

135
			go func(l net.Listener) {
136
				log.Printf("Starting secure web server on port %s\n", l.Addr())
137
				tl := tls.NewListener(l, tlsConfig)
138
				log.Fatal(http.Serve(tl, httpsMux))
139
			}(tlsListeners[i])
140
		}
141
	}
142

143
	if plainListeners, ok := listeners["http"]; ok {
144
		for i := range plainListeners {
145
			wg.Add(1)
146

147
			go func(l net.Listener) {
148
				log.Printf("Starting plaintext web server on port %s\n", l.Addr())
149
				log.Fatal(http.Serve(l, httpMux))
150
			}(plainListeners[i])
151
		}
152
	}
153

154
	// The code never calls `wg.Done()` so this will block forever
155
	// Letting the other goroutines carry on serving
156
	wg.Wait()
157
}