A software developer and technical architect, based in Scotland. Zola 2024-06-06T00:00:00+00:00 https://mgdm.net/atom.xml 2024-06-06T00:00:00+00:00 2024-06-06T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/nixos-with-private-flakes/ <p>I’ve been using Nix and NixOS for a while now, but I’d got into a way of working where it was doing what I needed to do, so I haven’t felt the need to modernise it with the new features that have come along since I started using it. On the other hand, I did want to use some code that I’m not ready to open source yet and I wanted to find a neat way to do that. It’s likely that none of this post will be new or interesting to a lot of people who use Nix and NixOS, but I haven’t seen it written down anywhere in exactly this way.</p> <p>Until now, to make this code work, I had been using a pretty awful and in no way hermetic solution with a local git checkout and some overlays. For some time I had imagined there must be a better way to do this using Flakes. I had searched around a couple of times to figure out how this might work but I never found a way to meet two preferences I had:</p> <ol> <li>The code should be pulled from GitHub from a private repository without having to be proxied somehow</li> <li>I shouldn’t have to have any credentials for GitHub stored on the box in the clear</li> </ol> <p>While I was at <a href="https://emfcamp.org">EMF</a>, I dropped into the <a href="https://nix.camp">nix.camp</a> tent a couple of times, and finally managed to work it out after a couple of pointers from Matt. I haven’t really used Flakes an awful lot so it took a couple of tries.</p> <h2 id="key-changes">Key changes<a class="zola-anchor" style="display: none;" href="#key-changes" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>There were three key points:</p> <h3 id="don-t-use-etc-nixos-any-more">Don’t use /etc/nixos any more<a class="zola-anchor" style="display: none;" href="#don-t-use-etc-nixos-any-more" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <p>The configuration for my machine was still stored in <code>/etc/nixos</code>, which was how I’d installed it in the first place. With Flakes, you can keep the configuration anywhere on the filesystem. With the config in my home directory, the rebuild command becomes:</p> <p><code>sudo nixos-rebuild switch --flake .#</code></p> <h3 id="don-t-use-sudo-with-nixos-rebuild">Don’t use <code>sudo</code> with nixos-rebuild<a class="zola-anchor" style="display: none;" href="#don-t-use-sudo-with-nixos-rebuild" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <p>The next problem was how to use the <code>ssh</code> credentials I have as my normal user, without having to copy them to root. It turns out this is also fairly simple, though the name of the option is a little non-obvious given my use of it – <code>--use-remote-sudo</code>. By using this, you don’t need the whole command to be run with <code>sudo</code>, and instead it will ask for the password after it’s completed the work in the Nix store when it needs to apply the changes to the system. With this, the command is:</p> <p><code>sudo nixos-rebuild switch --flake .# --use-remote-sudo</code></p> <h3 id="use-ssh-agent-forwarding">Use SSH agent forwarding<a class="zola-anchor" style="display: none;" href="#use-ssh-agent-forwarding" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <p>I’m not normally keen on this feature of SSH, and I don’t have it switched on permanently, but in this particular case it works for me. For this to work I need to have <code>ssh-agent</code> running locally, I need to use <code>ssh -A</code> to connect, and then everything should work. If it doesn’t work for some reason I get some errors when it’s retrieving the flakes. This saves me having to keep something like a GitHub access token or deploy key on the machine.</p> <h2 id="the-service-s-flake">The service’s flake<a class="zola-anchor" style="display: none;" href="#the-service-s-flake" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>In my service’s repo, my <code>flake.nix</code> looks like this (all my services are written in Rust):</p> <pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">inputs</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">naersk</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">url</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>github:nix-community/naersk/master<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">nixpkgs</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">url</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>github:NixOS/nixpkgs/nixpkgs-unstable<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">utils</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">url</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>github:numtide/flake-utils<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">outputs</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-entity z-function z-2 z-nix">{</span> <span class="z-variable z-parameter z-function z-1 z-nix">self</span><span class="z-keyword z-operator z-nix">,</span> <span class="z-variable z-parameter z-function z-1 z-nix">nixpkgs</span><span class="z-keyword z-operator z-nix">,</span> <span class="z-variable z-parameter z-function z-1 z-nix">utils</span><span class="z-keyword z-operator z-nix">,</span> <span class="z-variable z-parameter z-function z-1 z-nix">naersk</span> <span class="z-punctuation z-definition z-entity z-function z-nix">}</span><span class="z-punctuation z-definition z-function z-nix">:</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">utils</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">lib</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">eachDefaultSystem</span> <span class="z-punctuation z-definition z-expression z-nix">(</span><span class="z-variable z-parameter z-function z-4 z-nix">system</span><span class="z-punctuation z-definition z-function z-nix">:</span> </span><span class="z-source z-nix"> <span class="z-keyword z-other z-nix">let</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">pkgs</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-support z-function z-nix">import</span> <span class="z-variable z-parameter z-name z-nix">nixpkgs</span> <span class="z-punctuation z-definition z-attrset z-nix">{</span> <span class="z-keyword z-other z-inherit z-nix">inherit</span> <span class="z-entity z-other z-attribute-name z-single z-nix">system</span><span class="z-punctuation z-terminator z-inherit z-nix">;</span> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">naersk-lib</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-variable z-parameter z-name z-nix">pkgs</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">callPackage</span> <span class="z-variable z-parameter z-name z-nix">naersk</span> <span class="z-punctuation z-definition z-attrset z-nix">{</span> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-keyword z-other z-nix">in</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">packages</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">default</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-variable z-parameter z-name z-nix">naersk-lib</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">buildPackage</span> <span class="z-string z-unquoted z-path z-nix">./.</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">devShell</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-keyword z-other z-nix">with</span> <span class="z-variable z-parameter z-name z-nix">pkgs</span>; <span class="z-variable z-parameter z-name z-nix">mkShell</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">buildInputs</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-list z-nix">[</span> <span class="z-variable z-parameter z-name z-nix">cargo</span> <span class="z-variable z-parameter z-name z-nix">cargo-edit</span> <span class="z-variable z-parameter z-name z-nix">rustc</span> <span class="z-variable z-parameter z-name z-nix">rustfmt</span> <span class="z-variable z-parameter z-name z-nix">pre-commit</span> <span class="z-variable z-parameter z-name z-nix">rustPackages</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">clippy</span> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">RUST_SRC_PATH</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-variable z-parameter z-name z-nix">rustPlatform</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">rustLibSrc</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-expression z-nix">)</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset z-nix">}</span> </span></code></pre> <h2 id="system-configuration">System configuration<a class="zola-anchor" style="display: none;" href="#system-configuration" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>First off, I added a very basic <code>flake.nix</code> inspired by the documentation. It includes my own package, and then imports the existing <code>configuration.nix</code>.</p> <pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">description</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>Flake-based NixOS configuration<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">inputs</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">nixpkgs</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">url</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>github:NixOS/nixpkgs/nixos-24.05<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">myPackage</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">url</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>git+ssh://git@github.com/mgdm/myPackage.git<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">outputs</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-entity z-function z-2 z-nix">{</span> <span class="z-variable z-parameter z-function z-1 z-nix">self</span><span class="z-keyword z-operator z-nix">,</span> <span class="z-variable z-parameter z-function z-1 z-nix">nixpkgs</span><span class="z-keyword z-operator z-nix">,</span> <span class="z-keyword z-operator z-nix">... </span><span class="z-punctuation z-definition z-entity z-function z-nix">}</span>@<span class="z-variable z-parameter z-function z-3 z-nix">inputs</span><span class="z-punctuation z-definition z-function z-nix">:</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">nixosConfigurations</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">metis</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-variable z-parameter z-name z-nix">nixpkgs</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">lib</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">nixosSystem</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">system</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>x86_64-linux<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">specialArgs</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-attrset z-nix">{</span> <span class="z-keyword z-other z-inherit z-nix">inherit</span> <span class="z-entity z-other z-attribute-name z-single z-nix">inputs</span><span class="z-punctuation z-terminator z-inherit z-nix">;</span> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">modules</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-list z-nix">[</span> </span><span class="z-source z-nix"> <span class="z-string z-unquoted z-path z-nix">./configuration.nix</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset z-nix">}</span> </span></code></pre> <p>And then in my <code>configuration.nix</code>, I made a few changes.</p> <p>First off, I added inputs at the top:</p> <pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-punctuation z-definition z-entity z-function z-2 z-nix">{</span> <span class="z-variable z-parameter z-function z-1 z-nix">config</span><span class="z-keyword z-operator z-nix">,</span> <span class="z-variable z-parameter z-function z-1 z-nix">pkgs</span><span class="z-keyword z-operator z-nix">,</span> <span class="z-variable z-parameter z-function z-1 z-nix">inputs</span><span class="z-keyword z-operator z-nix">,</span> <span class="z-keyword z-operator z-nix">... </span><span class="z-punctuation z-definition z-entity z-function z-nix">}</span><span class="z-punctuation z-definition z-function z-nix">:</span> </span></code></pre> <p>To make referring to the package involve a little less typing, I gave it a shorter name:</p> <pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-keyword z-other z-nix">let</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">myPackage</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-variable z-parameter z-name z-nix">inputs</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">myPackage</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">packages</span><span class="z-keyword z-operator z-nix">.</span><span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span><span class="z-markup z-italic"><span class="z-punctuation z-section z-embedded z-begin z-nix">${</span><span class="z-variable z-parameter z-name z-nix">pkgs</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">system</span><span class="z-punctuation z-section z-embedded z-end z-nix">}</span></span><span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">default</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"><span class="z-keyword z-other z-nix">in</span> </span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">nix</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">extraOptions</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>experimental-features = nix-command flakes<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-comment z-line z-number-sign z-nix"># ...</span> </span></code></pre> <p>Then later I can reference that in <code>environment.systemPackages</code>:</p> <pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">environment</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">systemPackages</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-keyword z-other z-nix">with</span> <span class="z-variable z-parameter z-name z-nix">pkgs</span>; <span class="z-punctuation z-definition z-list z-nix">[</span> <span class="z-variable z-parameter z-name z-nix">vim</span> <span class="z-variable z-parameter z-name z-nix">ripgrep</span> <span class="z-variable z-parameter z-name z-nix">git</span> <span class="z-variable z-parameter z-name z-nix">tmux</span> <span class="z-variable z-parameter z-name z-nix">myPackage</span> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset z-nix">}</span> </span></code></pre> <p>I have a <code>systemd</code> service configured for this in <code>configuration.nix</code>, but my next move is to investigate how to provide a module in the service’s <code>flake.nix</code> and move that configuration there. Eventually I’ll look into building this remotely so I can keep the configuration on my local computer.</p> <p>If there are better ways to do any of this, I’d be happy to hear about them.</p> 2022-12-22T00:00:00+00:00 2022-12-22T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/adventures-in-digital-radio/ <p>A few years ago I got my foundation amateur radio licence. It was something I’d been considering doing for quite a long time but there were few opportunities to do so where I was at the time. I used it a bit as MM6YHF, mainly on 2m and 70cm analogue FM repeaters. It’s been a while since I did very much radio, as where I stay now is a bit further away from the repeaters so it became harder to do.</p> <p>A few weeks ago I spent a weekend at the <a href="https://57north.org.uk">57 North hackspace</a> and saw <a href="https://foxk.it">Hibby</a> using a DMR radio. What I found interesting about this was that he was using a hotspot running <a href="https://www.pistar.uk">Pi-Star</a> elsewhere in the hackspace, so it was possible to communicate with people without having to be within range of a repeater. This particular one was on the <a href="https://brandmeister.network">Brandmeister network</a>, though other networks are available.</p> <p>Eventually I got a TyT MD-UV390 radio, and borrowed Hibby’s hotspot to try it out for myself.</p> <h2 id="programming-the-radio">Programming the radio<a class="zola-anchor" style="display: none;" href="#programming-the-radio" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>It turns out that DMR radio, having been designed for commercial use, is a bit more complex than analogue FM. Working out how to program it was a bit of an expedition round some blogs, forums, and YouTube channels. As well as having the RF configuration for input and output frequencies, each repeater has two time slots upon which can be carried several talkgroups. The radio additionally has channels, scan groups, and zones. On my particular radio, I have opted to have one zone per digital repeater, including the hotspot, and another zone to contain the analogue FM repeaters. Each talkgroup on each repeater is then programmed as a separate channel. There is a feature for dynamically calling talkgroups but I’ll figure that one out later on.</p> <p>The configuration of the radio is referred to as a “codeplug”. The software used to program the radio, referred to as “CPS” perhaps for “codeplug software” or maybe “customer programming software”, takes a bit of getting used to. There are open source options including <a href="https://dm3mat.darc.de/qdmr/">qdmr</a> which I have tried out briefly. I made a <a href="https://github.com/mgdm/qdmr">Nix flake for qmdr</a>, although I later found that it doesn’t quite cover all of the features of the radio. In the end I found a variety of YouTube videos which described the features, and finally <a href="https://mcbainsite.co.uk/how-to-make-a-dmr-codeplug-tyt/">a very helpful blog post</a> that described exactly how to make a codeplug from scratch. Before programming or using the radio, you need to register for a DMR ID at <a href="https://radioid.net">RadioID</a>.</p> <h2 id="firmware">Firmware<a class="zola-anchor" style="display: none;" href="#firmware" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>The firmware my radio arrived with was version <code>D019.005</code>. The <code>D</code> indicates that it’s configured to reserve space in the internal storage for recording audio. You can replace the firmware with another that has a version number starting with <code>S</code> that allows the storage to instead be used for storing contacts. It should hold around 100,000, which should let the radio identify most people I’m likely to talk to. You can obtain lists of contacts, which are mappings from DMR IDs to callsigns and first names, from RadioID.</p> <p>Finding the firmware was a bit of fun. The <a href="https://www.tyt888.com/?mod=download">manufacturer’s website</a> doesn’t make it simple to identify what version you’re downloading, and I couldn’t find any that was newer than <code>S018</code>. It turns out however that this hardware is also sold as the Retevis RT3. The Retevis site was much easier to navigate than the TyT one, so I obtained version <code>S019.005</code> from there and flashed that. It worked fine, although the graphic displayed when the radio is switched on will be different now.</p> <h2 id="loading-contacts">Loading contacts<a class="zola-anchor" style="display: none;" href="#loading-contacts" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>One issue I’ve had is that while I can compile a contact list in CSV from the database on <a href="https://radioid.net">RadioID</a>, programming that to the radio using the TyT CPS requires a copy of Microsoft Excel. I don’t have that. <code>qdmr</code> has a UI for doing this, but trying it out displays a message about my radio not being supported. I’ve not quite worked around this yet. I’d quite like to avoid having to get a trial version of Office.</p> <h2 id="pi-star">Pi-Star<a class="zola-anchor" style="display: none;" href="#pi-star" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>Pi-Star is a Raspbian-based Linux distribution that runs on Raspberry Pis and similar devices with appropriate radio hardware attached. It can provide repeater functionality for a variety of digital radio protocols including DMR, and will interface with networks like Brandmeister. The configuration is completely handled by a web UI, which I found intuitive once I had an understanding of talkgroups and time slots. It will backhaul the radio traffic over wifi to your chosen network, which feels a little like cheating.</p> <p>As it’s been a little while since I used the radio for much I’ve mainly been listening in a bit to remind myself how it works before getting too involved. Thanks go to Hibby both for the loan of the hotspot and also putting up with many elementary questions during the process of getting set up.</p> 2022-10-25T00:00:00+00:00 2022-10-25T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/using-jq-in-sqlite/ <p>A while ago I was doing some work with data that I was loading into SQLite. I had been using <code>jq</code> to do some exploratory work to get different parts of the data out, but those queries didn’t directly translate to the JSON operators that exist in SQLite. I wondered if it might be possible to add an extension to SQLite to add operators that speak <code>jq</code>’s syntax.</p> <p>It turns out that it’s relatively easy. I opted to use Go to write this addon, as there is both a solid <a href="https://github.com/itchyny/gojq">native Go implementation of <code>jq</code></a> and a <a href="https://github.com/riyaz-ali/sqlite">Go library for writing SQLite addons</a>, and I didn’t fancy writing it in C. The extension is <a href="https://github.com/mgdm/sqlite-jq">available on GitHub</a>.</p> <h2 id="installing">Installing<a class="zola-anchor" style="display: none;" href="#installing" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>The instructions are a little different between macOS and Linux.</p> <p>On macOS, run <code>make</code>, then you can load the resulting extension into <code>sqlite3</code> using <code>.load sqlite_jq.dylib</code> (or perhaps <code>.load sqlite_jq.so</code>, depending where you got your compiler from).</p> <p>On Linux, run <code>make</code> to build, though you will then have to place the extension somewhere on <code>LD_LIBRARY_PATH</code>. Alternatively, for testing, you can set this directly:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-storage z-modifier z-shell">export</span> <span class="z-variable z-other z-readwrite z-assignment z-shell">LD_LIBRARY_PATH</span><span class="z-keyword z-operator z-assignment z-shell">=</span><span class="z-string z-unquoted z-shell"><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">PWD</span></span>:LD_LIBRARY_PATH</span></span> </span></code></pre> <p>Then, you can load the resulting extension with <code>.load sqlite_jq</code>.</p> <h2 id="overview">Overview<a class="zola-anchor" style="display: none;" href="#overview" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>It provides a scalar function, <code>jq()</code>, which lets you extract values from a JSON field using a <code>jq</code> expression:</p> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-keyword z-other z-DML z-sql">select</span> jq(<span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>{&quot;a&quot;: &quot;xyz&quot;}<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>, <span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>.a<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>); </span><span class="z-source z-sql"><span class="z-comment z-line z-double-dash z-sql"><span class="z-punctuation z-definition z-comment z-sql">--</span> returns &quot;xyz&quot; </span></span></code></pre> <p>There is also a <a href="https://www.sqlite.org/vtab.html#tabfunc2">table-valued function</a> called <code>jq_each()</code> that can be used to create a table of values from JSON fields.</p> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-keyword z-other z-DML z-sql">select</span> <span class="z-variable z-language z-star z-sql">*</span> <span class="z-keyword z-other z-DML z-sql">from</span> jq_each(<span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>{&quot;rows&quot;: [&quot;hello&quot;, &quot;world&quot;]}<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>, <span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>.rows[]<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>); </span><span class="z-source z-sql"><span class="z-comment z-line z-double-dash z-sql"><span class="z-punctuation z-definition z-comment z-sql">--</span> returns two rows, &quot;hello&quot; and &quot;world&quot; </span></span></code></pre> <h2 id="worked-example">Worked example<a class="zola-anchor" style="display: none;" href="#worked-example" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>For this example, I’m going to use <a href="https://twitter.com/jonty">Jonty’s</a> <a href="https://github.com/Jonty/airline-route-data">airline routes data</a>. It’s a large JSON file, which looks like this:</p> <pre data-lang="json" class="language-json z-code"><code class="language-json" data-lang="json"><span class="z-source z-json"><span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>AAA<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>city_name<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>Anaa<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>continent<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>OC<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>country<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>French Polynesia<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>country_code<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>PF<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>display_name<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>Anaa, Anaa Airport (AAA), French Polynesia<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>elevation<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-constant z-numeric z-integer z-decimal z-json">23</span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>iata<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>AAA<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>icao<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>NTGA<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>latitude<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>-17.355648<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>longitude<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>-145.50913<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>name<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>Anaa Airport<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>routes<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-punctuation z-section z-sequence z-begin z-json">[</span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span></span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-meta z-mapping z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>carriers<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-punctuation z-section z-sequence z-begin z-json">[</span> </span></span></span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>Air Tahiti<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span> </span></span></span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-punctuation z-section z-sequence z-end z-json">]</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>iata<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>FAC<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>km<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-constant z-numeric z-integer z-decimal z-json">76</span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>min<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-constant z-numeric z-integer z-decimal z-json">25</span> </span></span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-meta z-mapping z-value z-json"> <span class="z-punctuation z-section z-mapping z-end z-json">}</span></span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-punctuation z-section z-sequence z-end z-json">]</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>timezone<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>Pacific/Tahiti<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> <span class="z-punctuation z-section z-mapping z-end z-json">}</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"> <span class="z-comment z-line z-double-slash z-js"><span class="z-punctuation z-definition z-comment z-json">//</span> ... </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-section z-mapping z-end z-json">}</span></span> </span></code></pre> <p>It’s a single large object where the keys are the IATA codes for each airport. To start with, I’m going to create a table with a single column, and import each of these sub-objects into that table, one per row.</p> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-meta z-create z-sql"><span class="z-keyword z-other z-create z-sql">create</span> <span class="z-keyword z-other z-sql">table</span> </span><span class="z-meta z-toc-list z-full-identifier z-sql"><span class="z-entity z-name z-function z-sql">raw_data</span></span> ( </span><span class="z-source z-sql"> raw <span class="z-storage z-type z-sql">text</span> </span><span class="z-source z-sql">); </span></code></pre> <p>Then I’ll insert the rows directly from the file:</p> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-keyword z-other z-DML z-sql">insert into</span> raw_data <span class="z-keyword z-other z-DML z-sql">select</span> <span class="z-variable z-language z-star z-sql">*</span> <span class="z-keyword z-other z-DML z-sql">from</span> jq_each(readfile(<span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>airline_routes.json<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>), <span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>.[]<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>); </span></code></pre> <p>Now, I can use these functions to extract the data into other tables. Let’s create a table for the airports.</p> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-meta z-create z-sql"><span class="z-keyword z-other z-create z-sql">create</span> <span class="z-keyword z-other z-sql">table</span> </span><span class="z-meta z-toc-list z-full-identifier z-sql"><span class="z-entity z-name z-function z-sql">airports</span></span> ( </span><span class="z-source z-sql"> iata <span class="z-storage z-type z-sql">char</span>(<span class="z-constant z-numeric z-sql">3</span>) <span class="z-storage z-modifier z-sql">primary key</span>, </span><span class="z-source z-sql"> icao <span class="z-storage z-type z-sql">char</span>(<span class="z-constant z-numeric z-sql">4</span>), </span><span class="z-source z-sql"> latitude float, </span><span class="z-source z-sql"> longitude float, </span><span class="z-source z-sql"> name <span class="z-storage z-type z-sql">varchar</span>(<span class="z-constant z-numeric z-sql">255</span>), </span><span class="z-source z-sql"> timezone <span class="z-storage z-type z-sql">varchar</span>(<span class="z-constant z-numeric z-sql">255</span>) </span><span class="z-source z-sql">); </span></code></pre> <p>I can then use the <code>jq()</code> function to load data into this table.</p> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-keyword z-other z-DML z-sql">insert into</span> airports <span class="z-keyword z-other z-DML z-sql">select</span> </span><span class="z-source z-sql"> jq(raw, <span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>.iata<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>), </span><span class="z-source z-sql"> jq(raw, <span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>.icao<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>), </span><span class="z-source z-sql"> jq(raw, <span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>.latitude<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>), </span><span class="z-source z-sql"> jq(raw, <span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>.longitude<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>), </span><span class="z-source z-sql"> jq(raw, <span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>.name<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>), </span><span class="z-source z-sql"> jq(raw, <span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>.timezone<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>) </span><span class="z-source z-sql"><span class="z-keyword z-other z-DML z-sql">from</span> raw_data; </span></code></pre> <p>This can, of course, be done just as easily with SQLite’s own <code>json_extract()</code> function, with a slightly different syntax, or in newer releases of SQLite using the <code>-&gt;</code> or <code>-&gt;&gt;</code> operators.</p> <p>Now, we’ll extract the route data, into another table.</p> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-meta z-create z-sql"><span class="z-keyword z-other z-create z-sql">create</span> <span class="z-keyword z-other z-sql">table</span> </span><span class="z-meta z-toc-list z-full-identifier z-sql"><span class="z-entity z-name z-function z-sql">routes</span></span> ( </span><span class="z-source z-sql"> origin_iata <span class="z-storage z-type z-sql">char</span>(<span class="z-constant z-numeric z-sql">3</span>), </span><span class="z-source z-sql"> destination_iata <span class="z-storage z-type z-sql">char</span>(<span class="z-constant z-numeric z-sql">3</span>), </span><span class="z-source z-sql"> distance <span class="z-storage z-type z-sql">int</span>, </span><span class="z-source z-sql"> <span class="z-storage z-type z-sql">time</span> <span class="z-storage z-type z-sql">int</span>, </span><span class="z-source z-sql"> carrier <span class="z-storage z-type z-sql">varchar</span>(<span class="z-constant z-numeric z-sql">255</span>) </span><span class="z-source z-sql">); </span></code></pre> <p>And we can use a much longer <code>jq</code> expression to extract the route and carrier information. I’m using <code>json_extract()</code> here to demonstrate that the two extensions interoperate.</p> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-keyword z-other z-DML z-sql">insert into</span> routes <span class="z-keyword z-other z-DML z-sql">select</span> </span><span class="z-source z-sql"> jq(<span class="z-constant z-other z-database-name z-sql">raw_data</span>.<span class="z-constant z-other z-table-name z-sql">raw</span>, <span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>.iata<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>) <span class="z-keyword z-operator z-assignment z-alias z-sql">as</span> origin_iata, </span><span class="z-source z-sql"> json_extract(<span class="z-constant z-other z-database-name z-sql">routes</span>.<span class="z-constant z-other z-table-name z-sql">value</span>, <span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>$[0]<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>) <span class="z-keyword z-operator z-assignment z-alias z-sql">as</span> dest_iata, </span><span class="z-source z-sql"> json_extract(<span class="z-constant z-other z-database-name z-sql">routes</span>.<span class="z-constant z-other z-table-name z-sql">value</span>, <span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>$[1]<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>) <span class="z-keyword z-operator z-assignment z-alias z-sql">as</span> distance, </span><span class="z-source z-sql"> json_extract(<span class="z-constant z-other z-database-name z-sql">routes</span>.<span class="z-constant z-other z-table-name z-sql">value</span>, <span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>$[2]<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>) <span class="z-keyword z-operator z-assignment z-alias z-sql">as</span> <span class="z-storage z-type z-sql">time</span>, </span><span class="z-source z-sql"> json_extract(<span class="z-constant z-other z-database-name z-sql">routes</span>.<span class="z-constant z-other z-table-name z-sql">value</span>, <span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>$[3]<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>) <span class="z-keyword z-operator z-assignment z-alias z-sql">as</span> carrier </span><span class="z-source z-sql"><span class="z-keyword z-other z-DML z-sql">from</span> raw_data, </span><span class="z-source z-sql">jq_each(<span class="z-constant z-other z-database-name z-sql">raw_data</span>.<span class="z-constant z-other z-table-name z-sql">raw</span>, <span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>.routes[] | . as $route | .carriers[] | [ $route.iata, $route.km, $route.min, . ]<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>) <span class="z-keyword z-operator z-assignment z-alias z-sql">as</span> routes; </span></code></pre> <p>Now we can use the data to find all of the routes from Glasgow Airport.</p> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-keyword z-other z-DML z-sql">select</span> <span class="z-constant z-other z-database-name z-sql">origin</span>.<span class="z-constant z-other z-table-name z-sql">iata</span>, <span class="z-constant z-other z-database-name z-sql">origin</span>.<span class="z-constant z-other z-table-name z-sql">name</span>, <span class="z-constant z-other z-database-name z-sql">destination</span>.<span class="z-constant z-other z-table-name z-sql">iata</span>, <span class="z-constant z-other z-database-name z-sql">destination</span>.<span class="z-constant z-other z-table-name z-sql">name</span>, routes.<span class="z-variable z-language z-star z-sql">*</span> </span><span class="z-source z-sql"><span class="z-keyword z-other z-DML z-sql">from</span> airports origin </span><span class="z-source z-sql"><span class="z-keyword z-other z-DML z-sql">left join</span> routes on <span class="z-constant z-other z-database-name z-sql">routes</span>.<span class="z-constant z-other z-table-name z-sql">origin_iata</span> <span class="z-keyword z-operator z-comparison z-sql">=</span> <span class="z-constant z-other z-database-name z-sql">origin</span>.<span class="z-constant z-other z-table-name z-sql">iata</span> </span><span class="z-source z-sql"><span class="z-keyword z-other z-DML z-sql">left join</span> airports <span class="z-keyword z-operator z-assignment z-alias z-sql">as</span> destination on <span class="z-constant z-other z-database-name z-sql">routes</span>.<span class="z-constant z-other z-table-name z-sql">destination_iata</span> <span class="z-keyword z-operator z-comparison z-sql">=</span> <span class="z-constant z-other z-database-name z-sql">destination</span>.<span class="z-constant z-other z-table-name z-sql">iata</span> </span><span class="z-source z-sql"><span class="z-keyword z-other z-DML z-sql">where</span> <span class="z-constant z-other z-database-name z-sql">origin</span>.<span class="z-constant z-other z-table-name z-sql">iata</span> <span class="z-keyword z-operator z-comparison z-sql">=</span> <span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>GLA<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>; </span><span class="z-source z-sql"> </span><span class="z-source z-sql"><span class="z-comment z-line z-double-dash z-sql"><span class="z-punctuation z-definition z-comment z-sql">--</span> GLA|Glasgow|LHR|Heathrow|GLA|LHR|555|90|British Airways </span></span><span class="z-source z-sql"><span class="z-comment z-line z-double-dash z-sql"><span class="z-punctuation z-definition z-comment z-sql">--</span> GLA|Glasgow|DUB|Dublin|GLA|DUB|295|60|Aer Lingus </span></span><span class="z-source z-sql"><span class="z-comment z-line z-double-dash z-sql"><span class="z-punctuation z-definition z-comment z-sql">--</span> GLA|Glasgow|DUB|Dublin|GLA|DUB|295|60|Ryanair </span></span><span class="z-source z-sql"><span class="z-comment z-line z-double-dash z-sql"><span class="z-punctuation z-definition z-comment z-sql">--</span> GLA|Glasgow|LGW|Gatwick|GLA|LGW|595|95|easyJet </span></span><span class="z-source z-sql"><span class="z-comment z-line z-double-dash z-sql"><span class="z-punctuation z-definition z-comment z-sql">--</span> ... </span></span></code></pre> <p>The complete script can be seen <a href="https://mgdm.net/weblog/using-jq-in-sqlite/load-airports.sql.txt">here</a>.</p> <h2 id="to-do">To-do<a class="zola-anchor" style="display: none;" href="#to-do" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>It’s very small, and I’ve got some work to do to add tests now that I understand how everything fits together. There’s probably some optimisation work to be done too, as this does do quite a lot of translation back and forth from JSON behind the scenes. I’m not sure if some form of caching might be appropriate, but I’ll look into it.</p> <p>There’s also a question about whether this is a good idea at all, but I’m happy with it.</p> 2022-01-12T00:00:00+00:00 2022-01-12T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/json-in-postgresql/ <p>I’ve been using SQLite a bit in some of my own projects recently, and I think it’s a great solution for small self-hosted applications, avoiding the need to run a full-blown RDBMS where it’s not really warranted. <a href="https://christine.website/blog/sqlite-json-munge-2022-01-04">Xe’s post about JSON in SQLite</a> talks a lot about how to use SQLite to handle JSON data and load it into other tables without having to have your application do it, which is great.</p> <p>On the other hand, in 2020 I spent a lot of time with a team demolishing an old enterprise system at work, with a fairly convoluted database structure. I learned a bit about PostgreSQL’s JSON functions in the process. I’ve been meaning to write up what I learned for some time now before I <del>banished</del> forgot it entirely, so Xe’s post was a good prompt.</p> <p>It turns out that similar capabilities exist in both databases, so I’m going to translate Xe’s post more or less directly into PostgreSQL equivalents. My site doesn’t have a JSONFeed endpoint (yet, perhaps <a href="https://getzola.org">Zola</a> will acquire one, one day), so I’m going to borrow Xe’s feed and Python script as the basis for what I do here. I’m using PostgreSQL 13.5 here. There are some <a href="https://www.postgresql.org/docs/release/14.0/">neat features in 14</a>, like array subscripting for JSON column types and things like that, which are worth reading up on.</p> <h2 id="setting-up-the-tables">Setting up the tables<a class="zola-anchor" style="display: none;" href="#setting-up-the-tables" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>SQLite uses text data types to store JSON in, and has operators which can work on that. However, PostgreSQL has a separate JSON datatype. In fact, it has two–<code>json</code> and <code>jsonb</code><sup class="footnote-reference"><a href="#1">1</a></sup>. The <code>json</code> type stores an exact copy of the input text, while <code>jsonb</code> transforms it into a binary form that is faster for many operations as well as being indexable. I’ve only ever used <code>jsonb</code>, and I’ll continue to do that here.</p> <p>Because PostgreSQL is a relational database server, there is some work around setting up a new database and its permissions that I’m going to skip past. There are plenty of tutorials for getting this up and running already elsewhere. Because I’m a Nix fan I used <code>nix-shell</code> to install PostgreSQL, as described in a <a href="https://mgdm.net/weblog/postgresql-in-a-nix-shell/">previous post</a>. For this example I’m going to assume the database is local, and the database, username and password are all <code>mydb</code>.</p> <p>Let’s create a table, and throw some data into it. The key differences are the column types I’m using (<code>timestamp</code> and <code>jsonb</code>), and the slightly different syntax required for the default timestamp.</p> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-comment z-line z-number-sign z-sql"><span class="z-punctuation z-definition z-comment z-sql">#</span> schema.sql </span></span><span class="z-source z-sql"><span class="z-meta z-create z-sql"><span class="z-keyword z-other z-create z-sql">CREATE</span> <span class="z-keyword z-other z-sql">TABLE</span> </span><span class="z-meta z-toc-list z-full-identifier z-sql">IF NOT EXISTS </span><span class="z-meta z-toc-list z-full-identifier z-sql"><span class="z-entity z-name z-function z-sql">jsonfeed_raw</span></span> </span><span class="z-source z-sql"> ( feed_url <span class="z-storage z-type z-sql">TEXT</span> <span class="z-storage z-modifier z-sql">PRIMARY KEY</span> </span><span class="z-source z-sql"> , scrape_date <span class="z-storage z-type z-sql">TIMESTAMP</span> <span class="z-keyword z-operator z-logical z-sql">NOT</span> <span class="z-constant z-language z-sql">NULL</span> <span class="z-storage z-modifier z-sql">DEFAULT</span> NOW<span class="z-meta z-block z-sql"><span class="z-punctuation z-section z-scope z-begin z-sql">(</span><span class="z-punctuation z-section z-scope z-end z-sql">)</span></span> </span><span class="z-source z-sql"> , raw JSONB <span class="z-keyword z-operator z-logical z-sql">NOT</span> <span class="z-constant z-language z-sql">NULL</span> </span><span class="z-source z-sql"> ); </span></code></pre> <aside> <p>I don't normally write SQL like this, but I'll go with the flow.</p> </aside> <p>There are a couple of Python libraries for communicating with PostgreSQL, but the one I’m most familar with is the dubiously-named <code>psycopg2</code>.</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span>!/usr/bin/env nix-shell </span></span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span>! nix-shell -p python39 python39Packages.psycopg2 -i python </span></span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">psycopg2</span></span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">urllib</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">request</span></span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">con</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">psycopg2</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">connect</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">host</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">localhost<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">dbname</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">mydb<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">user</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">mydb<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">password</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">mydb<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-function z-python"><span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">get_feed</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-variable z-parameter z-python">feed_url</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">req</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">urllib</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">request</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">Request</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">feed_url</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-variable z-parameter z-python">headers</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-section z-mapping z-begin z-python">{</span></span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">User-Agent<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">mgdm/feedfetch<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-section z-mapping z-end z-python">}</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-statement z-with z-python"><span class="z-keyword z-control z-flow z-with z-python">with</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">urllib</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">request</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">urlopen</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">req</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-meta z-statement z-with z-python"><span class="z-keyword z-control z-flow z-with z-as z-python">as</span></span></span><span class="z-meta z-statement z-with z-python"> <span class="z-meta z-generic-name z-python">response</span><span class="z-punctuation z-section z-block z-with z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">cur</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">con</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">cursor</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">body</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">response</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">read</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">decode</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">utf-8<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">cur</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">execute</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;&quot;&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"> </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-meta z-string z-python"></span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> <span class="z-keyword z-other z-DML z-sql">INSERT INTO</span> jsonfeed_raw </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> (feed_url, raw) </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> <span class="z-keyword z-other z-DML z-II z-sql">VALUES</span> </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> (<span class="z-constant z-other z-placeholder z-python">%s</span>, <span class="z-constant z-other z-placeholder z-python">%s</span>) </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> </span><span class="z-punctuation z-definition z-string z-end z-python">&quot;&quot;&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-sequence z-tuple z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">feed_url</span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">body</span></span><span class="z-punctuation z-section z-sequence z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">con</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">commit</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">print</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">got feed <span class="z-constant z-other z-placeholder z-python">%s</span><span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span> <span class="z-keyword z-operator z-arithmetic z-python">%</span> <span class="z-meta z-group z-python"><span class="z-punctuation z-section z-group z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">feed_url</span></span><span class="z-punctuation z-section z-group z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">get_feed</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">https://christine.website/blog.json<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <p>We can create the table from the schema above with the <code>psql</code> command:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> psql<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>U</span> mydb<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>f</span> schema.sql mydb</span> </span></code></pre> <p>Then use the Python script to load the data:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">$</span></span><span class="z-meta z-function-call z-arguments z-shell"> python jsonfeedfetch.py</span> </span></code></pre> <h2 id="json-operators">JSON operators<a class="zola-anchor" style="display: none;" href="#json-operators" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>PostgreSQL has <a href="https://www.postgresql.org/docs/current/functions-json.html">a few operators</a> that can be used for accessing data held in JSON columns. They look a little different from the ones in SQLite. The <code>-&gt;</code> operator returns a JSONB type, which can be queried further–we’ll see this in action later. However, the <code>-&gt;&gt;</code> operator will return a text representation of whatever value it retrieves. Note that the names of the JSON fields need to be quoted, otherwise PostgreSQL will think that you are referring to a column.</p> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql">mydb<span class="z-keyword z-operator z-comparison z-sql">=</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span> <span class="z-keyword z-other z-DML z-sql">select</span> raw<span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>title<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span> <span class="z-keyword z-operator z-assignment z-alias z-sql">as</span> title, pg_typeof(raw<span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>title<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>) <span class="z-keyword z-other z-DML z-sql">from</span> jsonfeed_raw; </span><span class="z-source z-sql"> title | pg_typeof </span><span class="z-source z-sql"><span class="z-comment z-line z-double-dash z-sql"><span class="z-punctuation z-definition z-comment z-sql">--</span>-----------+----------- </span></span><span class="z-source z-sql"> <span class="z-string z-quoted z-double z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&quot;</span>Xe&#39;s Blog<span class="z-punctuation z-definition z-string z-end z-sql">&quot;</span></span> | jsonb </span><span class="z-source z-sql">(<span class="z-constant z-numeric z-sql">1</span> row) </span><span class="z-source z-sql"> </span></code></pre> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql">mydb<span class="z-keyword z-operator z-comparison z-sql">=</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span> <span class="z-keyword z-other z-DML z-sql">select</span> raw<span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>title<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span> <span class="z-keyword z-operator z-assignment z-alias z-sql">as</span> title, pg_typeof(raw<span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>title<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>) <span class="z-keyword z-other z-DML z-sql">from</span> jsonfeed_raw; </span><span class="z-source z-sql"> title | pg_typeof </span><span class="z-source z-sql"><span class="z-comment z-line z-double-dash z-sql"><span class="z-punctuation z-definition z-comment z-sql">--</span>---------+----------- </span></span><span class="z-source z-sql"> Xe<span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>s Blog | text </span></span><span class="z-source z-sql"><span class="z-string z-quoted z-single z-sql">(1 row) </span></span></code></pre> <p>You can chain these operators, and also use integers to select array elements, so you can use something like this to select the title of the first post in the feed:</p> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql">mydb<span class="z-keyword z-operator z-comparison z-sql">=</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span> <span class="z-keyword z-other z-DML z-sql">select</span> raw<span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>items<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span><span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-constant z-numeric z-sql">0</span><span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>title<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span> <span class="z-keyword z-operator z-assignment z-alias z-sql">as</span> post_title <span class="z-keyword z-other z-DML z-sql">from</span> jsonfeed_raw; </span><span class="z-source z-sql"> post_title </span><span class="z-source z-sql"><span class="z-comment z-line z-double-dash z-sql"><span class="z-punctuation z-definition z-comment z-sql">--</span>----------------------------- </span></span><span class="z-source z-sql"> <span class="z-string z-quoted z-double z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&quot;</span>A Tool to Aid Forgetfulness<span class="z-punctuation z-definition z-string z-end z-sql">&quot;</span></span> </span><span class="z-source z-sql">(<span class="z-constant z-numeric z-sql">1</span> row) </span></code></pre> <p>Rather than using this syntax, it’s possible to use other operators to go more directly to the data you need. The query above could also be expressed as one of the following:</p> <pre class="z-code"><code><span class="z-text z-plain">-- Using a list of object keys and array indices </span><span class="z-text z-plain">select raw #&gt; &#39;{items,0,title}&#39; as post_title from jsonfeed_raw; </span><span class="z-text z-plain"> </span><span class="z-text z-plain">-- Using JSONPath </span><span class="z-text z-plain">select jsonb_path_query(raw, &#39;$.items[0].title&#39;) from jsonfeed_raw; </span></code></pre> <p>Let’s go on and create the metadata table and insert the data into it. I only had to make one change to the syntax here, to the default value on the <code>updated_at</code> column:</p> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-meta z-create z-sql"><span class="z-keyword z-other z-create z-sql">CREATE</span> <span class="z-keyword z-other z-sql">TABLE</span> </span><span class="z-meta z-toc-list z-full-identifier z-sql">IF NOT EXISTS </span><span class="z-meta z-toc-list z-full-identifier z-sql"><span class="z-entity z-name z-function z-sql">jsonfeed_metadata</span></span> </span><span class="z-source z-sql"> ( feed_url <span class="z-storage z-type z-sql">TEXT</span> <span class="z-storage z-modifier z-sql">PRIMARY KEY</span> </span><span class="z-source z-sql"> , title <span class="z-storage z-type z-sql">TEXT</span> <span class="z-keyword z-operator z-logical z-sql">NOT</span> <span class="z-constant z-language z-sql">NULL</span> </span><span class="z-source z-sql"> , description <span class="z-storage z-type z-sql">TEXT</span> </span><span class="z-source z-sql"> , home_page_url <span class="z-storage z-type z-sql">TEXT</span> </span><span class="z-source z-sql"> , updated_at <span class="z-storage z-type z-sql">TEXT</span> <span class="z-keyword z-operator z-logical z-sql">NOT</span> <span class="z-constant z-language z-sql">NULL</span> <span class="z-storage z-modifier z-sql">DEFAULT</span> NOW<span class="z-meta z-block z-sql"><span class="z-punctuation z-section z-scope z-begin z-sql">(</span><span class="z-punctuation z-section z-scope z-end z-sql">)</span></span> </span><span class="z-source z-sql"> ); </span></code></pre> <p>Populating the second table looks fairly similar, although note the different operators inside the <code>SELECT</code>:</p> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-keyword z-other z-DML z-sql">INSERT INTO</span> jsonfeed_metadata </span><span class="z-source z-sql"> ( feed_url </span><span class="z-source z-sql"> , title </span><span class="z-source z-sql"> , description </span><span class="z-source z-sql"> , home_page_url </span><span class="z-source z-sql"> , updated_at </span><span class="z-source z-sql"> ) </span><span class="z-source z-sql"><span class="z-keyword z-other z-DML z-sql">SELECT</span> <span class="z-constant z-other z-database-name z-sql">jsonfeed_raw</span>.<span class="z-constant z-other z-table-name z-sql">feed_url</span> <span class="z-keyword z-operator z-assignment z-alias z-sql">AS</span> feed_url </span><span class="z-source z-sql"> , raw<span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>title<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span> <span class="z-keyword z-operator z-assignment z-alias z-sql">AS</span> title </span><span class="z-source z-sql"> , raw<span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>description<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span> <span class="z-keyword z-operator z-assignment z-alias z-sql">AS</span> description </span><span class="z-source z-sql"> , raw<span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>home_page_url<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span> <span class="z-keyword z-operator z-assignment z-alias z-sql">AS</span> home_page_url </span><span class="z-source z-sql"> , NOW<span class="z-meta z-block z-sql"><span class="z-punctuation z-section z-scope z-begin z-sql">(</span><span class="z-punctuation z-section z-scope z-end z-sql">)</span></span> <span class="z-keyword z-operator z-assignment z-alias z-sql">AS</span> updated_at </span><span class="z-source z-sql"><span class="z-keyword z-other z-DML z-sql">FROM</span> jsonfeed_raw; </span></code></pre> <h2 id="triggers">Triggers<a class="zola-anchor" style="display: none;" href="#triggers" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>Triggers are a little more complex in PostgreSQL. It’s not currently possible to write triggers in plain SQL<sup class="footnote-reference"><a href="#2">2</a></sup>. Instead we have to write a function in another language and call that. The easiest choice in this case is to use <a href="https://www.postgresql.org/docs/current/plpgsql.html">PL/pgSQL</a>, which is a PostgreSQL-specific superset of SQL. That’s going to look something like this:</p> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-meta z-drop z-sql"><span class="z-keyword z-other z-create z-sql">DROP</span> <span class="z-keyword z-other z-sql">TRIGGER</span></span> IF <span class="z-keyword z-operator z-logical z-sql">EXISTS</span> jsonfeed_raw_ins ON jsonfeed_raw; </span><span class="z-source z-sql"><span class="z-meta z-drop z-sql"><span class="z-keyword z-other z-create z-sql">DROP</span> <span class="z-keyword z-other z-sql">FUNCTION</span></span> IF <span class="z-keyword z-operator z-logical z-sql">EXISTS</span> jsonfeed_update; </span><span class="z-source z-sql"> </span><span class="z-source z-sql"><span class="z-meta z-create z-sql"><span class="z-keyword z-other z-create z-sql">CREATE</span> <span class="z-keyword z-other z-sql">FUNCTION</span> </span><span class="z-meta z-toc-list z-full-identifier z-sql"><span class="z-entity z-name z-function z-sql">jsonfeed_update</span></span><span class="z-meta z-block z-sql"><span class="z-punctuation z-section z-scope z-begin z-sql">(</span><span class="z-punctuation z-section z-scope z-end z-sql">)</span></span> RETURNS trigger <span class="z-keyword z-operator z-assignment z-alias z-sql">AS</span> $jsonfeed_update$ </span><span class="z-source z-sql"> <span class="z-keyword z-other z-LUW z-sql">BEGIN</span> </span><span class="z-source z-sql"> <span class="z-keyword z-other z-DML z-sql">INSERT INTO</span> jsonfeed_metadata </span><span class="z-source z-sql"> ( feed_url </span><span class="z-source z-sql"> , title </span><span class="z-source z-sql"> , description </span><span class="z-source z-sql"> , home_page_url </span><span class="z-source z-sql"> ) </span><span class="z-source z-sql"> <span class="z-keyword z-other z-DML z-II z-sql">VALUES</span> ( <span class="z-constant z-other z-database-name z-sql">NEW</span>.<span class="z-constant z-other z-table-name z-sql">feed_url</span> </span><span class="z-source z-sql"> , <span class="z-constant z-other z-database-name z-sql">NEW</span>.<span class="z-constant z-other z-table-name z-sql">raw</span><span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>title<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span> </span><span class="z-source z-sql"> , <span class="z-constant z-other z-database-name z-sql">NEW</span>.<span class="z-constant z-other z-table-name z-sql">raw</span><span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>title<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span> </span><span class="z-source z-sql"> , <span class="z-constant z-other z-database-name z-sql">NEW</span>.<span class="z-constant z-other z-table-name z-sql">raw</span><span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>home_page_url<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span> </span><span class="z-source z-sql"> ) </span><span class="z-source z-sql"> ON CONFLICT (feed_url) DO </span><span class="z-source z-sql"> <span class="z-keyword z-other z-DML z-sql">UPDATE</span> <span class="z-keyword z-other z-DML z-sql">SET</span> </span><span class="z-source z-sql"> title <span class="z-keyword z-operator z-comparison z-sql">=</span> <span class="z-constant z-other z-database-name z-sql">NEW</span>.<span class="z-constant z-other z-table-name z-sql">raw</span><span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>title<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span> </span><span class="z-source z-sql"> , description <span class="z-keyword z-operator z-comparison z-sql">=</span> <span class="z-constant z-other z-database-name z-sql">NEW</span>.<span class="z-constant z-other z-table-name z-sql">raw</span><span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>description<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span> </span><span class="z-source z-sql"> , home_page_url <span class="z-keyword z-operator z-comparison z-sql">=</span> <span class="z-constant z-other z-database-name z-sql">NEW</span>.<span class="z-constant z-other z-table-name z-sql">raw</span><span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>home_page_url<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>; </span><span class="z-source z-sql"> RETURN <span class="z-constant z-language z-sql">NULL</span>; <span class="z-comment z-line z-double-dash z-sql"><span class="z-punctuation z-definition z-comment z-sql">--</span> Ignored because this is an AFTER trigger </span></span><span class="z-source z-sql"> <span class="z-keyword z-other z-DML z-sql">END</span>; </span><span class="z-source z-sql">$jsonfeed_update$ LANGUAGE plpgsql; </span><span class="z-source z-sql"> </span><span class="z-source z-sql"><span class="z-meta z-create z-sql"><span class="z-keyword z-other z-create z-sql">CREATE</span> <span class="z-keyword z-other z-sql">TRIGGER</span> </span><span class="z-meta z-toc-list z-full-identifier z-sql"><span class="z-entity z-name z-function z-sql">jsonfeed_raw_ins</span></span> </span><span class="z-source z-sql">AFTER <span class="z-keyword z-other z-DML z-sql">INSERT</span> <span class="z-keyword z-operator z-logical z-sql">OR</span> <span class="z-keyword z-other z-DML z-sql">UPDATE</span> <span class="z-keyword z-operator z-logical z-sql">OR</span> <span class="z-keyword z-other z-DML z-sql">DELETE</span> ON jsonfeed_raw </span><span class="z-source z-sql"> FOR EACH ROW EXECUTE FUNCTION jsonfeed_update<span class="z-meta z-block z-sql"><span class="z-punctuation z-section z-scope z-begin z-sql">(</span><span class="z-punctuation z-section z-scope z-end z-sql">)</span></span>; </span></code></pre> <p>This is a little more verbose, but does the same thing. Note the slightly different <code>ON CONFLICT</code> syntax, where we have to specify a column name unlike in the SQLite version. Speaking of conflicts, let’s also update the Python script so that it does an update on a conflict in the <code>jsonfeed_raw</code> table:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">cur</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">execute</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;&quot;&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"> </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-meta z-string z-python"></span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> <span class="z-keyword z-other z-DML z-sql">INSERT INTO</span> jsonfeed_raw </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> (feed_url, raw) </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> <span class="z-keyword z-other z-DML z-II z-sql">VALUES</span> </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> (<span class="z-constant z-other z-placeholder z-python">%s</span>, <span class="z-constant z-other z-placeholder z-python">%s</span>) </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> ON CONFLICT (feed_url) DO </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> <span class="z-keyword z-other z-DML z-sql">UPDATE</span> <span class="z-keyword z-other z-DML z-sql">SET</span> raw <span class="z-keyword z-operator z-comparison z-sql">=</span> <span class="z-constant z-other z-placeholder z-python">%s</span> </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> </span><span class="z-punctuation z-definition z-string z-end z-python">&quot;&quot;&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-sequence z-tuple z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">feed_url</span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">body</span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">body</span></span><span class="z-punctuation z-section z-sequence z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <p>Now, if we delete everything from each of the tables and run the script again, we should see data in <code>jsonfeed_metadata</code>.</p> <pre class="z-code"><code><span class="z-text z-plain">mydb=&gt; select * from jsonfeed_metadata; </span><span class="z-text z-plain"> feed_url | title | description | home_page_url | updated_at </span><span class="z-text z-plain">-------------------------------------+-------------+------------------------------------------------------------+-----------------------------+------------------------------- </span><span class="z-text z-plain"> https://christine.website/blog.json | &quot;Xe&#39;s Blog&quot; | &quot;My blog posts and rants about various technology things.&quot; | &quot;https://christine.website&quot; | 2022-01-12 22:15:36.920008+00 </span><span class="z-text z-plain">(1 row) </span><span class="z-text z-plain"> </span><span class="z-text z-plain">mydb=&gt; </span></code></pre> <h2 id="extracting-rows-from-the-json">Extracting rows from the JSON<a class="zola-anchor" style="display: none;" href="#extracting-rows-from-the-json" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>Like the <code>json_each</code> function in SQLite, PostgreSQL can unpack arrays of items in JSON and use those to create more rows using a function called <code>jsonb_array_elements</code>. Let’s create a table to put blog posts into:</p> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-meta z-create z-sql"><span class="z-keyword z-other z-create z-sql">CREATE</span> <span class="z-keyword z-other z-sql">TABLE</span> </span><span class="z-meta z-toc-list z-full-identifier z-sql">IF NOT EXISTS </span><span class="z-meta z-toc-list z-full-identifier z-sql"><span class="z-entity z-name z-function z-sql">jsonfeed_posts</span></span> </span><span class="z-source z-sql"> ( url <span class="z-storage z-type z-sql">TEXT</span> <span class="z-storage z-modifier z-sql">PRIMARY KEY</span> </span><span class="z-source z-sql"> , feed_url <span class="z-storage z-type z-sql">TEXT</span> <span class="z-keyword z-operator z-logical z-sql">NOT</span> <span class="z-constant z-language z-sql">NULL</span> </span><span class="z-source z-sql"> , title <span class="z-storage z-type z-sql">TEXT</span> <span class="z-keyword z-operator z-logical z-sql">NOT</span> <span class="z-constant z-language z-sql">NULL</span> </span><span class="z-source z-sql"> , date_published <span class="z-storage z-type z-sql">TIMESTAMP</span> <span class="z-keyword z-operator z-logical z-sql">NOT</span> <span class="z-constant z-language z-sql">NULL</span> </span><span class="z-source z-sql"> ); </span></code></pre> <p>We can populate this table as follows:</p> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-keyword z-other z-DML z-sql">INSERT INTO</span> jsonfeed_posts </span><span class="z-source z-sql"> ( url </span><span class="z-source z-sql"> , feed_url </span><span class="z-source z-sql"> , title </span><span class="z-source z-sql"> , date_published </span><span class="z-source z-sql"> ) </span><span class="z-source z-sql"><span class="z-keyword z-other z-DML z-sql">SELECT</span> </span><span class="z-source z-sql"> items<span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>url<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span> <span class="z-keyword z-operator z-assignment z-alias z-sql">AS</span> url </span><span class="z-source z-sql">, raw<span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>feed_url<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span> <span class="z-keyword z-operator z-assignment z-alias z-sql">AS</span> feed_url </span><span class="z-source z-sql">, items<span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>title<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span> <span class="z-keyword z-operator z-assignment z-alias z-sql">AS</span> title </span><span class="z-source z-sql">, (items<span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>date_published<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>)::<span class="z-storage z-type z-sql">timestamp</span> <span class="z-keyword z-operator z-assignment z-alias z-sql">AS</span> date_published </span><span class="z-source z-sql"><span class="z-keyword z-other z-DML z-sql">FROM</span> </span><span class="z-source z-sql"> jsonfeed_raw </span><span class="z-source z-sql">, jsonb_array_elements(<span class="z-constant z-other z-database-name z-sql">jsonfeed_raw</span>.<span class="z-constant z-other z-table-name z-sql">raw</span><span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>items<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>) <span class="z-keyword z-operator z-assignment z-alias z-sql">as</span> items; </span></code></pre> <p>We can do the same thing here as we did before, and turn this into a trigger. Again, we’ll need to use a slightly different language to do it, but it’s not going to be particularly difficult.</p> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-meta z-drop z-sql"><span class="z-keyword z-other z-create z-sql">DROP</span> <span class="z-keyword z-other z-sql">TRIGGER</span></span> IF <span class="z-keyword z-operator z-logical z-sql">EXISTS</span> jsonfeed_raw_ins_posts ON jsonfeed_raw; </span><span class="z-source z-sql"><span class="z-meta z-drop z-sql"><span class="z-keyword z-other z-create z-sql">DROP</span> <span class="z-keyword z-other z-sql">FUNCTION</span></span> IF <span class="z-keyword z-operator z-logical z-sql">EXISTS</span> jsonfeed_update_posts; </span><span class="z-source z-sql"> </span><span class="z-source z-sql"><span class="z-meta z-create z-sql"><span class="z-keyword z-other z-create z-sql">CREATE</span> <span class="z-keyword z-other z-sql">FUNCTION</span> </span><span class="z-meta z-toc-list z-full-identifier z-sql"><span class="z-entity z-name z-function z-sql">jsonfeed_update_posts</span></span><span class="z-meta z-block z-sql"><span class="z-punctuation z-section z-scope z-begin z-sql">(</span><span class="z-punctuation z-section z-scope z-end z-sql">)</span></span> RETURNS trigger <span class="z-keyword z-operator z-assignment z-alias z-sql">AS</span> $jsonfeed_update_posts$ </span><span class="z-source z-sql"> <span class="z-keyword z-other z-LUW z-sql">BEGIN</span> </span><span class="z-source z-sql"> <span class="z-keyword z-other z-DML z-sql">INSERT INTO</span> jsonfeed_posts </span><span class="z-source z-sql"> ( url </span><span class="z-source z-sql"> , feed_url </span><span class="z-source z-sql"> , title </span><span class="z-source z-sql"> , date_published </span><span class="z-source z-sql"> ) </span><span class="z-source z-sql"> <span class="z-keyword z-other z-DML z-sql">SELECT</span> </span><span class="z-source z-sql"> items<span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>url<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span> <span class="z-keyword z-operator z-assignment z-alias z-sql">AS</span> url </span><span class="z-source z-sql"> , raw<span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>feed_url<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span> <span class="z-keyword z-operator z-assignment z-alias z-sql">AS</span> feed_url </span><span class="z-source z-sql"> , items<span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>title<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span> <span class="z-keyword z-operator z-assignment z-alias z-sql">AS</span> title </span><span class="z-source z-sql"> , (items<span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>date_published<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>)::<span class="z-storage z-type z-sql">timestamp</span> <span class="z-keyword z-operator z-assignment z-alias z-sql">AS</span> date_published </span><span class="z-source z-sql"> <span class="z-keyword z-other z-DML z-sql">FROM</span> </span><span class="z-source z-sql"> jsonfeed_raw </span><span class="z-source z-sql"> , jsonb_array_elements(<span class="z-constant z-other z-database-name z-sql">jsonfeed_raw</span>.<span class="z-constant z-other z-table-name z-sql">raw</span><span class="z-keyword z-operator z-math z-sql">-</span><span class="z-keyword z-operator z-comparison z-sql">&gt;</span><span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>items<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>) <span class="z-keyword z-operator z-assignment z-alias z-sql">as</span> items </span><span class="z-source z-sql"> ON CONFLICT (url) DO </span><span class="z-source z-sql"> <span class="z-keyword z-other z-DML z-sql">UPDATE</span> <span class="z-keyword z-other z-DML z-sql">SET</span> title <span class="z-keyword z-operator z-comparison z-sql">=</span> <span class="z-constant z-other z-database-name z-sql">EXCLUDED</span>.<span class="z-constant z-other z-table-name z-sql">title</span> </span><span class="z-source z-sql"> , date_published <span class="z-keyword z-operator z-comparison z-sql">=</span> <span class="z-constant z-other z-database-name z-sql">EXCLUDED</span>.<span class="z-constant z-other z-table-name z-sql">date_published</span>; </span><span class="z-source z-sql"> RETURN <span class="z-constant z-language z-sql">NULL</span>; </span><span class="z-source z-sql"> <span class="z-keyword z-other z-DML z-sql">END</span>; </span><span class="z-source z-sql">$jsonfeed_update_posts$ LANGUAGE plpgsql; </span><span class="z-source z-sql"> </span><span class="z-source z-sql"><span class="z-meta z-create z-sql"><span class="z-keyword z-other z-create z-sql">CREATE</span> <span class="z-keyword z-other z-sql">TRIGGER</span> </span><span class="z-meta z-toc-list z-full-identifier z-sql"><span class="z-entity z-name z-function z-sql">jsonfeed_raw_ins_posts</span></span> </span><span class="z-source z-sql">AFTER <span class="z-keyword z-other z-DML z-sql">INSERT</span> <span class="z-keyword z-operator z-logical z-sql">OR</span> <span class="z-keyword z-other z-DML z-sql">UPDATE</span> <span class="z-keyword z-operator z-logical z-sql">OR</span> <span class="z-keyword z-other z-DML z-sql">DELETE</span> ON jsonfeed_raw </span><span class="z-source z-sql"> FOR EACH ROW EXECUTE FUNCTION jsonfeed_update_posts<span class="z-meta z-block z-sql"><span class="z-punctuation z-section z-scope z-begin z-sql">(</span><span class="z-punctuation z-section z-scope z-end z-sql">)</span></span>; </span></code></pre> <p>Next time you run <code>jsonfeedfetch.py</code>, you should have a fully-populated <code>jsonfeed_posts</code> table containing all of the posts from the feed.</p> <div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup> <p>There are also operators that can work on JSON stored in text fields, but I’ve not used those at all.</p> </div> <div class="footnote-definition" id="2"><sup class="footnote-definition-label">2</sup> <p>From the <a href="https://www.postgresql.org/docs/current/triggers.html">PostgreSQL documentation on triggers</a></p> </div> 2021-10-26T00:00:00+00:00 2021-10-26T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/firecracker-ignite-zfs/ <p>Now that I have my <a href="https://mgdm.net/weblog/nixos-on-raspberry-pi-4/">NixOS Pi 4</a> up and running and doing a couple of useful things, I thought I’d see what else I could do with it. So far it’s just running DHCP, DNS and Tailscale, so it’s got a bit of spare horsepower. I also wrote a <a href="https://github.com/mgdm/argonone-utils">fan control daemon for the Argon ONE case</a> because I had some different requirements from the one that Argon40 themselves ship.</p> <p>I decided to try out running Firecracker VMs on it, as I was aware Firecracker gained Pi 4 support some time ago and I’d been meaning to give it a go. I’d also recently read about the <a href="https://github.com/weaveworks/ignite">Ignite tool from Weaveworks</a> that lets you run VMs based on Docker container images without having to repack or rebuild them yourself. I expected that the quoted 125ms VM startup time was unlikely to be met on this hardware, but I figured it probably would still work.</p> <p>As I’m running NixOS, most of this was set up inside <code>configuration.nix</code>. I added <code>firecracker</code>, <code>ignite</code>, <code>git</code>, and <code>binutils-unwrapped</code> to <code>environment.systemPackages</code>, and also added a line to set <code>virtualisation.containerd.enable</code> to <code>true</code>. Ignite relies on <code>git</code> and the <code>strings</code> command from binutils, which I think should be explicit dependencies in the package, but I’ll come back to that later.</p> <p>A quick rebuild later, I had all the tools in place to be able to try and run a VM:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sudo</span></span><span class="z-meta z-function-call z-arguments z-shell"> ignite run weaveworks/ignite-ubuntu <span class="z-punctuation z-separator z-continuation z-line z-shell">\ </span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>cpus</span> 1 <span class="z-punctuation z-separator z-continuation z-line z-shell">\ </span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>memory</span> 512MB <span class="z-punctuation z-separator z-continuation z-line z-shell">\ </span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>ssh</span> <span class="z-punctuation z-separator z-continuation z-line z-shell">\ </span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>name</span> hello-world</span> </span></code></pre> <p>This didn’t give me a lot of output, but ultimately failed. Rerunning it with <code>--log-level debug</code> gave me something like this:</p> <pre class="z-code"><code><span class="z-text z-plain">DEBU[0001] Writing &quot;/var/lib/firecracker/vm/57b9e0bf6cf0228a/runtime.containerd.resolv.conf&quot; with new hash: &quot;fbe74b53a9d2380c8212bed5097146735021280bbc84d9e176139552a999fd25&quot;, old hash: &quot;&quot; </span><span class="z-text z-plain">FATA[0001] failed to start container for VM &quot;57b9e0bf6cf0228a&quot;: failed to mount /tmp/containerd-mount920509976: invalid argument </span></code></pre> <p>And digging into the journal gave me:</p> <pre class="z-code"><code><span class="z-text z-plain">Oct 26 20:13:12 nixpi kernel: overlayfs: upper fs does not support RENAME_WHITEOUT. </span><span class="z-text z-plain">Oct 26 20:13:12 nixpi kernel: overlayfs: upper fs missing required features. </span></code></pre> <p>Because I apparently enjoy making obtuse computing problems for myself to solve, I’m running this Pi on ZFS in order to learn more about how to operate it. It seems that ZFS doesn’t support something that <code>overlayfs</code> requires. I mentioned I had this problem on IRC and <a href="https://grahamc.com">Graham</a> suggested a <code>zvol</code>. This is not something I’d encountered before, but it’s a way of creating a block device that lives in your ZFS pool, which you can then format with whichever filesystem you like.</p> <p>It turns out after all I’m not the first person to try and work around this, as <a href="https://github.com/weaveworks/ignite/issues/630#issuecomment-763652140">this GitHub comment</a> in the Ignite repository proves, and I ended up using a very similar approach to the one described there.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sudo</span></span><span class="z-meta z-function-call z-arguments z-shell"> systemctl stop containerd.service</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sudo</span></span><span class="z-meta z-function-call z-arguments z-shell"> zfs create<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>s</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>V</span> 15G rpool/containerd</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sudo</span></span><span class="z-meta z-function-call z-arguments z-shell"> mkfs.ext4<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>b</span> 4096<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>L</span> containerd /dev/zvol/rpool/containerd</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sudo</span></span><span class="z-meta z-function-call z-arguments z-shell"> mkdir /tmp/migrate</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sudo</span></span><span class="z-meta z-function-call z-arguments z-shell"> mount /dev/zvol/rpool/containerd /tmp/migrate</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sudo</span></span><span class="z-meta z-function-call z-arguments z-shell"> cp<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>r</span> /var/lib/containerd/<span class="z-keyword z-operator z-regexp z-quantifier z-shell">*</span> /tmp/migrate</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sudo</span></span><span class="z-meta z-function-call z-arguments z-shell"> umount /tmp/migrate</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sudo</span></span><span class="z-meta z-function-call z-arguments z-shell"> mount /dev/zvol/rpool/containerd /var/lib/containerd</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sudo</span></span><span class="z-meta z-function-call z-arguments z-shell"> systemctl start containerd.service</span> </span></code></pre> <p>Running it again worked fine this time:</p> <img src="https:&#x2F;&#x2F;mgdm.net&#x2F;processed_images&#x2F;ignite.748c878a6ee8a52d.png" alt="A screenshot of the Ignite terminal session running the ignite run command from earlier" /> <p>I’ll make the new filesystem permanent by adding it into my <code>hardware-configuration.nix</code> using something like the below. It turns out that using <code>rpool/containerd</code> doesn’t work on boot, instead I need to use the device it’s exposed as (<code>/dev/zd0</code>), although a more robust way to do it would be to find its symlink in <code>/dev/disk/by-uuid</code>.</p> <pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">fileSystems</span><span class="z-keyword z-operator z-nix">.</span><span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>/var/lib/containerd<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span> <span class="z-invalid z-illegal">=</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">{</span> <span class="z-entity z-other z-attribute-name z-multipart z-nix">device</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>/dev/zvol/rpool/containerd<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">fsType</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>ext4<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-invalid z-illegal">;</span> </span></code></pre> <p>I’m pretty certain that this would have worked out of the box on a more conventional setup, but by doing it this way at I suppose I learned something. I don’t have any specific plans for this, but I’m going to see if it will handle things like the Unifi controller that I’d normally use Docker for.</p> 2021-09-12T00:00:00+00:00 2021-09-12T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/htmlq-revisited/ <p>A couple of years ago <a href="https://mgdm.net/weblog/htmlq/">I wrote about a tool I’d created</a> called <code>htmlq</code>. At the time, I’d been doing a bit of web scraping, and while it’s absolutely doable using Python and BeautifulSoup glued together with a shell script, it was a bit slow waiting for the Python startup time for each iteration. I ended up writing a bit more code to do the thing I wanted to do at an acceptable pace. Don’t get me wrong–both Python and BeautifulSoup are great tools, but sometimes I just want to be able to glue things together in an iterative way with <code>curl</code> and bash. I’m a fan of Rust, and knowing that the Mozilla Servo browser components were available, I looked to use those to make a tool to solve my problem (and hopefully learn something along the way).</p> <p>Last Monday I woke up to find that the tool I’d written was <a href="https://news.ycombinator.com/item?id=28441880">at the top of Hacker News</a>, where it had been submitted by <a href="https://twitter.com/jasonbosco">Jason Bosco</a>. It remained near the top for much of the day. This was great! However, at the time I’d not really thought too much about <code>htmlq</code> since I first wrote it. It turns out that a couple of people had been sending PRs and making suggestions in the time I’d not been paying attention, but I’d not spotted the notifications about them. I can only apologise for the oversight, but between some PRs that have been there for a little while and some more that came since, it’s in a better shape.</p> <p>I’ve <a href="https://crates.io/crates/htmlq">released a new version to crates.io</a>. <a href="https://github.com/NikolayMurha">Nikolay Murha</a> kindly added it to Homebrew. <a href="https://github.com/chrisdickinson">Chris Dickinson</a> kindly sent a PR last year which used GitHub Actions to create binary builds, which unfortunately sat around for a while before I got to merging it. Several people have also contributed to the documentation and to tweaking the defaults. It’s been really nice to have people do these things!</p> <p>At the moment I’m working on adding a couple of features that were requested, including one <a href="https://github.com/mgdm/htmlq/pull/21">to convert relative links to absolute ones</a> either using a <code>&lt;base&gt;</code> specified in the document or one supplied on the command line. I’m aiming to put a bit more testing around it too, and improve the pretty printing of the output.</p> <p>If anyone’s got any more suggestions, I’d be happy to hear about them.</p> 2021-09-05T00:00:00+00:00 2021-09-05T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/why-nix/ <p>I’ve written a lot about Nix and NixOS over the last couple of years, and I use Nix in some way most days that I’m on a computer. Sometimes when I mention this, people ask me what’s so interesting about it. I’m going to try and answer that here, by running through what I find useful about Nix.</p> <p>If you read the front page of the <a href="https://nixos.org">Nix website</a>, you’ll see a quote like this:</p> <blockquote> <p>Nix is a tool that takes a unique approach to package management and system configuration. Learn how to make reproducible, declarative and reliable systems.</p> </blockquote> <p>All of the above is true, and some folks may be able to see what is attractive about that immediately. I was not entirely one of those people when I started off, though I’d heard a lot about it from others. I’m still not an advanced Nix user by any stretch. I don’t really properly understand the Nix language yet, but this hasn’t particularly stopped me finding value in it.</p> <h2 id="background">Background<a class="zola-anchor" style="display: none;" href="#background" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>Normally, on a Linux machine, you’ve got a package manager that can install whatever you like, whenever you need it. If you’re on a Debian or Ubuntu machine, you might want to do some basic resizing of an image, and end up doing something like this:</p> <pre class="z-code"><code><span class="z-text z-plain">me@laptop:~$ convert -geometry 1024x big.jpg small.jpg </span><span class="z-text z-plain">bash: convert: command not found </span><span class="z-text z-plain"> </span><span class="z-text z-plain">me@laptop:~$ sudo apt install imagemagick </span><span class="z-text z-plain"># Much text ensues </span><span class="z-text z-plain"> </span><span class="z-text z-plain">me@laptop:~$ convert -geometry 1024x big.jpg small.jpg </span><span class="z-text z-plain"># Success! </span></code></pre> <p>This demonstrates one of the key points I want to get across here: on a conventional Linux distribution, <strong>these little changes accrete</strong>. If you need to wipe your machine, move to a new one, or just use a different computer for a little while, many of the things that you expect to be there are not going to be present. You might end up encountering these little paper cuts for some time as you continue working. Additionally, you end up with tools installed that you might use once and never again, taking up space.</p> <p>At work I generally use macOS, and on there I used to use Homebrew to install things. For 90% of what I do this was fine, but one thing tipped me over the edge into finally installing Nix. I was doing some data work which required me to use PostgreSQL. I’d just used <code>brew install postgresql</code> and got on with it at the time, without thinking too hard about what version was installed. It was a few weeks later that the version of PostgreSQL in Homebrew was upgraded as a side effect of installing something else, and the next time I looked at my Postgres DB it had been wiped and started from scratch. The data was still there, it was just in a different directory with the version number in the path, but the running daemon had been upgraded to a different one so it had started from an empty DB. There are other ways to solve this problem, but by this point I’d heard enough about “reproducibility” and “immutability” to wonder if this would be something that I could use.</p> <p>I installed Nix on macOS, and found out how to install things with <code>nix-env</code>.</p> <pre class="z-code"><code><span class="z-text z-plain">$ nix-env -iA nixpkgs.imagemagick </span></code></pre> <p>This is basically what I had before with <code>apt</code> or <code>brew</code>, except that it pulls from Nix’s own repository, called <code>nixpkgs</code>. You can install things, and they appear in your shell environment. It’s just different, not better.</p> <p>What is better is <code>nix-shell</code>. With this, you can create a new shell with any set of tools that you like present. For example:</p> <pre class="z-code"><code><span class="z-text z-plain">$ nix-shell -p imagemagick </span></code></pre> <p>This drops me into a new shell with ImageMagick present. When I close it again, it’s no longer there. If I don’t use it again eventually it’ll be garbage collected. I use this a bit for tools that I don’t need all of the time.</p> <h2 id="shell-nix">shell.nix<a class="zola-anchor" style="display: none;" href="#shell-nix" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>Shortly after this, I learned enough to discover <code>shell.nix</code> files. This is the part of Nix that I probably still use the most. These are files you can write and keep inside your projects that describe things you want to be available when you’re working on that bit of code. An example I have from the repository for this website looks like:</p> <pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-keyword z-other z-nix">with</span> <span class="z-support z-function z-nix">import</span> <span class="z-string z-unquoted z-spath z-nix">&lt;nixpkgs&gt;</span> <span class="z-punctuation z-definition z-attrset z-nix">{</span><span class="z-punctuation z-definition z-attrset z-nix">}</span>; </span><span class="z-source z-nix"> </span><span class="z-source z-nix"><span class="z-variable z-parameter z-name z-nix">pkgs</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">mkShell</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">buildInputs</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-list z-nix">[</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">pkgs</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">zola</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset z-nix">}</span> </span></code></pre> <p>This just means that if I change into the directory for the site, I can type <code>nix-shell</code> and that file will be loaded, and a new shell will result with <code>zola</code> available. Once I quit that shell, <code>zola</code> disappears from the environment. This is more important than it may appear, and results in next two key points: <strong>I can have different versions of tools available in different projects</strong>, and <strong>you can ensure you have only the tools you require</strong>. If I stop working on a project that has a particular dependency, it can get garbage collected and it’s no longer present on the system.</p> <p>I commit these <code>shell.nix</code> files to my projects, and I have blogged marginally more complex examples before for <a href="https://mgdm.net/weblog/postgresql-in-a-nix-shell/">keeping a PostgreSQL environment per-project</a> and also for <a href="https://mgdm.net/weblog/vscode-nix-go-tools/">the Go tools that VS Code wants</a>. For other projects at work where I’m the only person using Nix to install the build tools, I’ll sometimes keep them to myself by adding them to <code>.git/info/exclude</code>.</p> <p>A neat feature of this is that you can override how the packages you install are defined. For example, if you want to create a Python 3 environment with some packages from PyPI, you can do something like this:</p> <pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-keyword z-other z-nix">with</span> <span class="z-support z-function z-nix">import</span> <span class="z-string z-unquoted z-spath z-nix">&lt;nixpkgs&gt;</span> <span class="z-punctuation z-definition z-attrset z-nix">{</span><span class="z-punctuation z-definition z-attrset z-nix">}</span>; </span><span class="z-source z-nix"> </span><span class="z-source z-nix"><span class="z-keyword z-other z-nix">let</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">my-python-packages</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-variable z-parameter z-function z-4 z-nix">python-packages</span><span class="z-punctuation z-definition z-function z-nix">:</span> <span class="z-keyword z-other z-nix">with</span> <span class="z-variable z-parameter z-name z-nix">python-packages</span>; <span class="z-punctuation z-definition z-list z-nix">[</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">pillow</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">numpy</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">black</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">python-with-my-packages</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-variable z-parameter z-name z-nix">python3</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">withPackages</span> <span class="z-variable z-parameter z-name z-nix">my-python-packages</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"><span class="z-keyword z-other z-nix">in</span> </span><span class="z-source z-nix"><span class="z-variable z-parameter z-name z-nix">pkgs</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">mkShell</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">buildInputs</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-list z-nix">[</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">python-with-my-packages</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset z-nix">}</span> </span></code></pre> <h2 id="direnv-and-lorri">Direnv and Lorri<a class="zola-anchor" style="display: none;" href="#direnv-and-lorri" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p><code>nix-shell</code> is pretty neat, but it’s a bit like Python virtualenvs where you need to remember to activate and deactivate them as you go. You can use a tool called <a href="https://direnv.net">direnv</a> with its <a href="https://github.com/nix-community/nix-direnv">Nix addon</a> to automate this for you.</p> <p>Once you have that configured, setting up a project is pretty simple. You’ll need a <code>shell.nix</code> as above, and then you create a file to tell <code>direnv</code> what to do.</p> <pre class="z-code"><code><span class="z-text z-plain">$ echo &#39;use nix&#39; &gt; .envrc </span><span class="z-text z-plain">direnv: error .envrc is blocked. Run `direnv allow` to approve its content. </span></code></pre> <p>The error here demonstrates that you have to explicitly permit <code>direnv</code> to work in this directory. It prevents you accidentally running code in your shell directly from a Git checkout, or similar.</p> <pre class="z-code"><code><span class="z-text z-plain">$ direnv allow </span></code></pre> <p>Then all of your Nix dependencies will be installed, and you’ll return to your prompt in the same shell. It can take a little while, and if you do <code>nix-channel --update</code> or similar it will often do it again, but it’s pretty neat.</p> <p>To take some of the time out of this, you can combine direnv with <a href="https://github.com/nix-community/lorri">lorri</a>. It’s a daemon that runs in the background and can react live to updates in <code>shell.nix</code>. It also prevents your dependencies being cleared out if you run <code>nix-collect-garbage</code>–more on that command later. This also makes it easier to kick off new projects using <code>lorri init</code> instead of creating the files manually like I described.</p> <p>These days, at work, I’ve switched roles into one where I don’t do so much development as I used to, so I don’t actually use <code>lorri</code> any more. It’s still a pretty cool tool though and I’ll probably use it again if I return to a full-time software development role.</p> <h2 id="home-manager">Home Manager<a class="zola-anchor" style="display: none;" href="#home-manager" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>With Nix, I can create individual environments for projects I’m working on. It’d be cool if I could extend this idea to my entire login environment. With <a href="https://github.com/nix-community/home-manager">Home Manager</a>, I can. At a basic level, you can install it and do essentially the same thing as I described with <code>shell.nix</code> above, but it affects all of your shells. You can type <code>home-manager edit</code> and you’re dropped into an editor where you can make whatever changes you need, before using <code>home-manager switch</code> to apply them.</p> <p>Here’s a basic example from one of my machines:</p> <pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-punctuation z-definition z-entity z-function z-2 z-nix">{</span> <span class="z-variable z-parameter z-function z-1 z-nix">config</span><span class="z-keyword z-operator z-nix">,</span> <span class="z-variable z-parameter z-function z-1 z-nix">pkgs</span><span class="z-keyword z-operator z-nix">,</span> <span class="z-keyword z-operator z-nix">... </span><span class="z-punctuation z-definition z-entity z-function z-nix">}</span><span class="z-punctuation z-definition z-function z-nix">:</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"><span class="z-keyword z-other z-nix">let</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">pkgsUnstable</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-support z-function z-nix">import</span> <span class="z-string z-unquoted z-spath z-nix">&lt;nixpkgs-unstable&gt;</span> <span class="z-punctuation z-definition z-attrset z-nix">{</span><span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"><span class="z-keyword z-other z-nix">in</span> </span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">home</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">stateVersion</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>19.09<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">home</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">packages</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-keyword z-other z-nix">with</span> <span class="z-variable z-parameter z-name z-nix">pkgs</span>; <span class="z-punctuation z-definition z-list z-nix">[</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">awscli</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">direnv</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">git</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">htop</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">wget</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">ripgrep</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">jq</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">pkgsUnstable</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">asuka</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset z-nix">}</span> </span></code></pre> <p>Here I’m installing some of the tools I expect to be available all the time, including a newer version of <code>git</code> than is provided by macOS or Xcode. I normally use the stable version of a Nix release, but here I wanted to use a Gemini client called <code>asuka</code> which wasn’t available in a stable release when I set this up, so I installed that from the <code>unstable</code> release.</p> <p>If you keep your configuration in Git or somewhere like that, you can share it across machines, and have the same environment present everywhere you do work.</p> <p>Home Manager lets you do more than just install packages, though. You can configure them too, and it’s a convenient way to manage dotfiles. For example, I keep my Git configuration in there, and so Home Manager will write that to <code>~/.gitconfig</code> for me.</p> <pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-variable z-parameter z-name z-nix">programs</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">git</span> <span class="z-invalid z-illegal">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">enable</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">userName</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>Michael Maclean<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">userEmail</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>michael at thisdomain dot net<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-invalid z-illegal">;</span> </span></code></pre> <p>It can also start services for you at login. One example of this is <code>gpg-agent</code>:</p> <pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-variable z-parameter z-name z-nix">services</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">gpg-agent</span> <span class="z-invalid z-illegal">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">enable</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">defaultCacheTtl</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-numeric z-nix">1800</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">enableSshSupport</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-invalid z-illegal">;</span> </span></code></pre> <p>You’re also not limited to using it to install command line tools. On one of my Linux machines, I use it to configure VS Code and all the extensions I use. (I don’t use this on macOS–I’ll talk about that <a href="https://mgdm.net/weblog/why-nix/#things-i-don-t-use">later</a>.)</p> <p>I use this on every machine I have access to. The configuration is kept in Git so I can update it, and as long as I remember to pull the changes on my other machines, I end up having the same set of tools available.</p> <h2 id="nixos">NixOS<a class="zola-anchor" style="display: none;" href="#nixos" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>If you’ve read this far, you hopefully are able to see why I find Nix valuable. It lets me install the tools that I use and keep them configured the way I want in one place. It would be nice if I could build an entire system this way, which is where NixOS comes in. It’s a Linux distribution that fundamentally can be configured in the same way as I described above, using a Nix file to describe the state of the system you want. This differs from other configuration management tools, where sometimes you have to describe the set of actions that must be taken to make the system you want.</p> <p>The nicest thing about NixOS for me is that some thought has gone into how the services on your machine should be configured. Instead of having to write separate files in different languages and scatter them all across <code>/etc</code>, they’re all in one place (<code>/etc/nixos/configuration.nix</code>) and use the same language. I run a Miniflux RSS reader, and configuring that was really simple and clear:</p> <pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">services</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">miniflux</span> <span class="z-invalid z-illegal">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">enable</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-invalid z-illegal">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">services</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">nginx</span> <span class="z-invalid z-illegal">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">enable</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">recommendedGzipSettings</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">recommendedOptimisation</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">recommendedProxySettings</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">recommendedTlsSettings</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">virtualHosts</span>.<span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>rss-reader<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">forceSSL</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">enableACME</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">locations</span>.<span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>/<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">proxyPass</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>http://127.0.0.1:8080<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-invalid z-illegal">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">security</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">acme</span> <span class="z-invalid z-illegal">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">acceptTerms</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">email</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>blah<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-invalid z-illegal">;</span> </span></code></pre> <p>I’m not going to go into a lot more detail here as I’ve already covered some of what the rest of the system configuration looks like in previous posts. If you’d like to see what this might look like, check out these posts:</p> <ul> <li><a href="https://mgdm.net/weblog/nixos-on-raspberry-pi-4/">Installing NixOS on the Raspberry Pi 4</a></li> <li><a href="https://mgdm.net/weblog/nixos-on-kimsufi/">Installing NixOS on Kimsufi machines</a></li> <li><a href="https://mgdm.net/weblog/installing-nixos/">Installing NixOS on a desktop, with encrypted disks</a></li> <li><a href="https://mgdm.net/weblog/evading-broken-dns/">This post on setting up a DNS server</a> shows another example of how simple it can be to configure services on NixOS.</li> </ul> <p>Using NixOS doesn’t mean you need to throw out your Home Manager setup, either. I use NixOS’s own configuration to set up the hardware and the display manager, and to run services like OpenSSH on the machine, but all the software I want as a user (including Firefox, VS Code, that sort of thing) is set up using Home Manager. They coexist well.</p> <h2 id="things-i-don-t-use">Things I don’t use<a class="zola-anchor" style="display: none;" href="#things-i-don-t-use" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>Nix is a package manager, so obviously it can be used to package software for distribution. It can also build machine images, Docker images, Snaps, Flatpaks, and probably other things I’m forgetting. I don’t actually use it for much of that, which is related to me not doing full-time software development at the moment.</p> <p>I’ve also not tried to push the use of Nix for building software at work, partly due to the learning curve. I’m happy to help people as far as I can if they come to it themselves, but I’m not an evangelist for it there.</p> <p>Although I use Nix on macOS quite happily, I don’t use Nix to install GUI software there. I do on Linux. On macOS, most of the packages I’m interested in (things like Virtualbox) are marked as Linux-only and while I could override that, I found it didn’t really work, so I’ve not directly replaced <code>brew cask</code> with Nix. There is a tool called <code>nix-darwin</code> that I believe can do some of this, but I’ve not tried it. On my work computer I suspect it may upset the MDM, and on my home Mac I’m happy enough with Home Manager.</p> <p>In all three environments–<code>nix-shell</code>, Home Manager and NixOS–you can freeze the version of <code>nixpkgs</code> you are installing from, to specifically lock your environment to the versions of the dependencies that you know work. This is similar to a lock file in a package manager like Composer or NPM. I have used this a bit, but on the whole I’ve not had any trouble from just tracking the stable version of <code>nixpkgs</code>.</p> <h2 id="in-summary">In summary<a class="zola-anchor" style="display: none;" href="#in-summary" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>I get a lot of value out of Nix even at my current “advanced beginner” level of experience. I still don’t really know the Nix language. I’ve not used Flakes. I’ve not tried NixOps properly yet. All of these are things I am going to learn, but not knowing them has not held me back. I’ve been able to do all the things I’ve described, and do some contribution to <code>nixpkgs</code> itself, by learning from others and using the code and documentation that is already out there.</p> <p>If you’re a user of Nix already, you may note I’ve not talked about the core parts of how Nix works. I’ve not mentioned the store, nor immutability, nor really reproducibility–these are things that are important to how Nix works, but I find you don’t really need to understand them to make use of it.</p> <p>I have had a lot of assistance along the way. The Nix community has been pretty welcoming, although #nixos on Libera IRC is a pretty busy place. <a href="https://grahamc.com">Graham</a> has helped me and many others out a lot in the past, and I’ve learned a fair bit from discussions with <a href="https://seancoates.com/">Sean</a> and <a href="https://nerderati.com/">Joël</a> too. Thanks to all of them, and those in the community that have helped me in the past.</p> 2021-09-04T00:00:00+00:00 2021-09-04T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/nixos-on-raspberry-pi-4/ <p>I have been wanting to find a more permanent home for <a href="https://mgdm.net/weblog/evading-broken-dns/">the DNS server I built before</a> which was on a Pi 4 running Raspbian. As is evident by the number of posts I have on here about installing Nix on every computing device I can get my hands on, I wanted to put NixOS on it. Previously I’d not been keen doing that on an SD card because I still have a perception that Pis running Linux can chew them up pretty quickly, although I suppose reinstalling them from a reproducible OS would be less painful were this to happen.</p> <p>As it happens, last week a vendor sent me an email saying that they had a sale on the <a href="https://www.argon40.com/argon-one-m-2-case-for-raspberry-pi-4.html">Argon ONE m.2 case</a>, which would accept an m.2 SATA SSD. This would give me a bit more space and probably stand up to more use than the SD card would, so I got one.</p> <h2 id="requirements">Requirements<a class="zola-anchor" style="display: none;" href="#requirements" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>I could quite easily have just <code>dd</code>ed the SD card image onto the drive and booted that and called it a day, but I <del>wanted to make life difficult for myself</del> really wanted to use ZFS. I also wanted to use the AArch64 architecture instead of the armv7 that I believe Raspbian had been using.</p> <p>The first thing I did before getting started was boot Raspbian, update it, and then use it to update the firmware to a version that properly supports USB boot, as my Pi 4 was old enough not to have that out of the box.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sudo</span></span><span class="z-meta z-function-call z-arguments z-shell"> rpi-eeprom-update <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> Read the output of this before proceeding</span><span class="z-comment z-line z-number-sign z-shell"> </span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sudo</span></span><span class="z-meta z-function-call z-arguments z-shell"> rpi-eeprom-update<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>d</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>a <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> Actually do the update</span><span class="z-comment z-line z-number-sign z-shell"> </span></span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sudo</span></span><span class="z-meta z-function-call z-arguments z-shell"> raspi-config <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> Go to Advanced Options and enable USB boot</span><span class="z-comment z-line z-number-sign z-shell"> </span></span></span></code></pre> <h2 id="installation-process">Installation process<a class="zola-anchor" style="display: none;" href="#installation-process" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>I did consider trying to install it from the existing Raspbian OS that was on the SD card already, in a similar fashion to <a href="https://mgdm.net/weblog/nixos-on-kimsufi/">how I did so using the Kimsufi recovery OS</a>, but as I believe it was running a different architecture I wasn’t sure how that would work.</p> <p>I grabbed <a href="https://hydra.nixos.org/job/nixos/release-21.05/nixos.sd_image_new_kernel.aarch64-linux">the latest NixOS 21.05 <code>new_kernel</code> image I could find</a> on Hydra. This ended up being one built in July–images since then appear to have had some build failures due to something related to ZFS. I didn’t entirely pay attention here and ended up writing the compressed image to the SD card and trying to boot that–this predictably failed, and I re-read <a href="https://nixos.wiki/wiki/NixOS_on_ARM/Raspberry_Pi_4">the instructions</a> and uncompressed it first before trying again.</p> <p>Booting this results in it resizing to fit the SD card, and then drops you into a terminal. Normally here you’d probably create a <code>configuration.nix</code> and do <code>nixos-rebuild</code> to make the system you want, but I wanted to install to my SSD so I had a little more work to do.</p> <p>The procedure I followed was remarkably similar to that I used for the Kimsufi box. The SSD even presented it as <code>/dev/sda</code>, the same as on that machine. I’ll reproduce the commands here, as there are a couple of differences.</p> <h3 id="partitioning">Partitioning<a class="zola-anchor" style="display: none;" href="#partitioning" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">wipefs</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>a</span> /dev/sda</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> This sets a variable called DISK to save some typing later</span><span class="z-comment z-line z-number-sign z-shell"> </span></span><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> It should be set to the file starting with ata in the directory</span><span class="z-comment z-line z-number-sign z-shell"> </span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-storage z-modifier z-shell">export</span> <span class="z-variable z-other z-readwrite z-assignment z-shell">DISK</span><span class="z-keyword z-operator z-assignment z-shell">=</span><span class="z-string z-unquoted z-shell">/dev/disk/by-id/ata<span class="z-keyword z-operator z-regexp z-quantifier z-shell">*</span></span></span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">parted</span></span><span class="z-meta z-function-call z-arguments z-shell"> /dev/sda<span class="z-keyword z-operator z-end-of-options z-shell"> --</span></span><span class="z-meta z-function-call z-arguments z-shell"> mklabel msdos</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">parted</span></span><span class="z-meta z-function-call z-arguments z-shell"> /dev/sda<span class="z-keyword z-operator z-end-of-options z-shell"> --</span></span><span class="z-meta z-function-call z-arguments z-shell"> mkpart primary fat32 0MiB 512MiB <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> /dev/sda1 is /boot</span><span class="z-comment z-line z-number-sign z-shell"> </span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">parted</span></span><span class="z-meta z-function-call z-arguments z-shell"> /dev/sda<span class="z-keyword z-operator z-end-of-options z-shell"> --</span></span><span class="z-meta z-function-call z-arguments z-shell"> mkpart primary 512MiB -4GiB <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> This is the ZFS partition</span><span class="z-comment z-line z-number-sign z-shell"> </span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">parted</span></span><span class="z-meta z-function-call z-arguments z-shell"> /dev/sda<span class="z-keyword z-operator z-end-of-options z-shell"> --</span></span><span class="z-meta z-function-call z-arguments z-shell"> mkpart primary linux-swap -4GiB 100<span class="z-meta z-group z-expansion z-job z-shell"><span class="z-punctuation z-definition z-variable z-job z-shell">%</span></span> <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> Swap. This is optional</span><span class="z-comment z-line z-number-sign z-shell"> </span></span></span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">mkfs.vfat</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>F32</span> <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">DISK</span></span>-part1 <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> Format /boot</span><span class="z-comment z-line z-number-sign z-shell"> </span></span></span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> Create the zpool</span><span class="z-comment z-line z-number-sign z-shell"> </span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">zpool</span></span><span class="z-meta z-function-call z-arguments z-shell"> create<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>f</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>O</span> mountpoint=none rpool <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">DISK</span></span>-part2</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> Create some datasets inside the pool</span><span class="z-comment z-line z-number-sign z-shell"> </span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">zfs</span></span><span class="z-meta z-function-call z-arguments z-shell"> create<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>o</span> mountpoint=legacy rpool/root</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">zfs</span></span><span class="z-meta z-function-call z-arguments z-shell"> create<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>o</span> mountpoint=legacy rpool/root/nixos</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">zfs</span></span><span class="z-meta z-function-call z-arguments z-shell"> create<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>o</span> mountpoint=legacy rpool/home</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">mount</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>t</span> zfs rpool/root/nixos /mnt</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">mkdir</span></span><span class="z-meta z-function-call z-arguments z-shell"> /mnt/home</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">mount</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>t</span> zfs rpool/home /mnt/home</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">mkdir</span></span><span class="z-meta z-function-call z-arguments z-shell"> /mnt/boot</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">mount</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">DISK</span></span>-part1 /mnt/boot</span> </span></code></pre> <h3 id="nix-install">Nix install<a class="zola-anchor" style="display: none;" href="#nix-install" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <p>The installation process after this is a little cleaner than the one in the recovery system from previously.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">nixos-generate-config</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>root</span> /mnt</span> </span></code></pre> <p>At this point, I can edit the files in <code>/mnt/etc/nixos</code>. However, as the SD image I’m using is from July, it still has the same bug in it that I encountered last time, where the <code>nixos-install</code> process crashes out with errors similar to</p> <pre class="z-code"><code><span class="z-text z-plain">getting attributes of path /nix/store/dbri2d4r470fc6nrh95qa8bwcj54wh1q-zfs-kernel-2.0.5-5.10.52: No such file or directory </span></code></pre> <p>This has been fixed in a new version of Nix, but it wasn’t incorporated in the build I was using. I again used the workaround from <a href="https://github.com/NixOS/nixpkgs/issues/126141#issuecomment-861724383">this GitHub issue</a> to allow me to proceed, although as it was a native NixOS machine I didn’t have quite so much to type as before:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">nix-build</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-string z-quoted z-single z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&#39;</span>&lt;nixpkgs/nixos&gt;<span class="z-punctuation z-definition z-string z-end z-shell">&#39;</span></span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>A</span> config.system.build.toplevel<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>I</span> nixos-config=/mnt/etc/nixos/configuration.nix</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sudo</span></span><span class="z-meta z-function-call z-arguments z-shell"> nixos-install<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>root</span> /mnt</span> </span></code></pre> <h2 id="firmware">Firmware<a class="zola-anchor" style="display: none;" href="#firmware" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>The Raspberry Pi and/or U-Boot need some firmware files to be present in the first filesystem on the disk. The SD installer image keeps this in a partition at <code>/dev/mmcblk0p1</code> which is not actually mounted normally. I opted to mount the equivalent of that partition at <code>/boot</code> on my install, so it contains the generated <code>initrd</code>. However, NixOS doesn’t currently provide a neat way of copying these files into place.</p> <p>In the end, I just did something like:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">mkdir</span></span><span class="z-meta z-function-call z-arguments z-shell"> /firmware</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">mount</span></span><span class="z-meta z-function-call z-arguments z-shell"> /dev/mmcblk0p1 /firmware</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">cp</span></span><span class="z-meta z-function-call z-arguments z-shell"> /firmware/<span class="z-keyword z-operator z-regexp z-quantifier z-shell">*</span> /mnt/boot</span> </span></code></pre> <p>It took me a bit of effort to try and work out what partition type to use here, and then a little more to work out how to get the firmware on there. It’s not ideal, but you only need to do it once.</p> <h2 id="gotchas">Gotchas<a class="zola-anchor" style="display: none;" href="#gotchas" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>There were a couple of things worth noting that I found out as I was doing this.</p> <p>If the Pi’s firmware couldn’t work out what to do with the USB disk, it would fall back to booting off the SD card. This is fine in theory, but sometimes it would boot all the way past stage 2 and just at the point where you’d expect it to start a console the screen would go black, and then the monitor would switch off. The Pi wasn’t visible on the network at this point either so the only thing I could do was pull the power and give it a few moments before trying again. It normally worked from a cold boot.</p> <p>It took me a little while to work out what the first partition should look like. I had a few occasions where the machine would complain <code>Unable to read partition as FAT</code>. In the end, I needed to make it FAT32, start right at the beginning of the disk instead of 1MiB in like I had done before, and also flag it as type <code>0c</code>, which is <code>W95 FAT32 (LBA)</code> according to <code>fdisk</code>. I’m not sure which were the important parts of this, but it works now.</p> <p>Finally, some NixOS-on-RPi documentation suggests using the <code>boot.tmpOnTmpfs</code> option to put <code>/tmp</code> into a ramdisk. I found that doing that and asking Nix to install a few things at once made it run out of space very quickly and fail the build, so I turned that off. I could imagine that if I were to be installing on an SD card and not doing too much work I would consider enabling it again.</p> <h2 id="configuration-files">Configuration files<a class="zola-anchor" style="display: none;" href="#configuration-files" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>These are here in case they’re useful to anyone.</p> <p>My <code>hardware-configuration.nix</code> is pretty much the default, but there are some additions to include <code>zfs</code> in the supported filesystems.</p> <h3 id="hardware-configuration-nix">hardware-configuration.nix<a class="zola-anchor" style="display: none;" href="#hardware-configuration-nix" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-punctuation z-definition z-entity z-function z-2 z-nix">{</span> <span class="z-variable z-parameter z-function z-1 z-nix">config</span><span class="z-keyword z-operator z-nix">,</span> <span class="z-variable z-parameter z-function z-1 z-nix">lib</span><span class="z-keyword z-operator z-nix">,</span> <span class="z-variable z-parameter z-function z-1 z-nix">pkgs</span><span class="z-keyword z-operator z-nix">,</span> <span class="z-variable z-parameter z-function z-1 z-nix">modulesPath</span><span class="z-keyword z-operator z-nix">,</span> <span class="z-keyword z-operator z-nix">... </span><span class="z-punctuation z-definition z-entity z-function z-nix">}</span><span class="z-punctuation z-definition z-function z-nix">:</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">imports</span> <span class="z-keyword z-operator z-bind z-nix">=</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-list z-nix">[</span> <span class="z-punctuation z-definition z-expression z-nix">(</span><span class="z-variable z-parameter z-name z-nix">modulesPath</span> <span class="z-keyword z-operator z-nix">+</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>/installer/scan/not-detected.nix<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-definition z-expression z-nix">)</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">boot</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">initrd</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">availableKernelModules</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-list z-nix">[</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>xhci_pci<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>usbhid<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>uas<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>usb_storage<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">boot</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">initrd</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">kernelModules</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-list z-nix">[</span> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">boot</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">kernelModules</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-list z-nix">[</span> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">boot</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">extraModulePackages</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-list z-nix">[</span> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">boot</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">initrd</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">supportedFilesystems</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-list z-nix">[</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>zfs<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">boot</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">supportedFilesystems</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-list z-nix">[</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>zfs<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">fileSystems</span>.<span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>/<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span> <span class="z-keyword z-operator z-bind z-nix">=</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">{</span> <span class="z-entity z-other z-attribute-name z-multipart z-nix">device</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>rpool/root/nixos<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">fsType</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>zfs<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">fileSystems</span>.<span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>/home<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span> <span class="z-keyword z-operator z-bind z-nix">=</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">{</span> <span class="z-entity z-other z-attribute-name z-multipart z-nix">device</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>rpool/home<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">fsType</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>zfs<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">fileSystems</span>.<span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>/boot<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span> <span class="z-keyword z-operator z-bind z-nix">=</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">{</span> <span class="z-entity z-other z-attribute-name z-multipart z-nix">device</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>/dev/sda1<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">fsType</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>vfat<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">swapDevices</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-list z-nix">[</span> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">powerManagement</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">cpuFreqGovernor</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-variable z-parameter z-name z-nix">lib</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">mkDefault</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>ondemand<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-comment z-line z-number-sign z-nix"># high-resolution display</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">hardware</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">video</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">hidpi</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">enable</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-variable z-parameter z-name z-nix">lib</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">mkDefault</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset z-nix">}</span> </span></code></pre> <h3 id="configuration-nix">configuration.nix<a class="zola-anchor" style="display: none;" href="#configuration-nix" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <p>As an experiment, I’m running XFCE on this, just to see how it performs. You can comment all of that out.</p> <p>In this, I added an overlay, which you can see in the imports section. You can do this using the commands:</p> <pre class="z-code"><code><span class="z-text z-plain">sudo nix-channel --add https://github.com/NixOS/nixos-hardware/archive/master.tar.gz nixos-hardware </span><span class="z-text z-plain">sudo nix-channel --update </span></code></pre> <p>However, the only thing it provided for me was the option to do hardware acceleration for X (the option called <code>hardware.raspberry-pi."4".fkms-3d.enable</code>). This didn’t work out for me, and X just gave me a black screen. I intend to dig into this further and hopefully manage to reenable it in the future.</p> <pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-punctuation z-definition z-entity z-function z-2 z-nix">{</span> <span class="z-variable z-parameter z-function z-1 z-nix">config</span><span class="z-keyword z-operator z-nix">,</span> <span class="z-variable z-parameter z-function z-1 z-nix">pkgs</span><span class="z-keyword z-operator z-nix">,</span> <span class="z-keyword z-operator z-nix">... </span><span class="z-punctuation z-definition z-entity z-function z-nix">}</span><span class="z-punctuation z-definition z-function z-nix">:</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">imports</span> <span class="z-keyword z-operator z-bind z-nix">=</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-list z-nix">[</span> </span><span class="z-source z-nix"> <span class="z-string z-unquoted z-spath z-nix">&lt;nixos-hardware/raspberry-pi/4&gt;</span> </span><span class="z-source z-nix"> <span class="z-string z-unquoted z-path z-nix">./hardware-configuration.nix</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">boot</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">kernelPackages</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-variable z-parameter z-name z-nix">pkgs</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">linuxPackages_rpi4</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-comment z-line z-number-sign z-nix"># tmpOnTmpfs = true; # See note</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">kernelParams</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-list z-nix">[</span> </span><span class="z-source z-nix"> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>8250.nr_uarts=1<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span> </span><span class="z-source z-nix"> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>console=ttyAMA0,115200<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span> </span><span class="z-source z-nix"> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>console=tty1<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span> </span><span class="z-source z-nix"> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>cma=128M<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">boot</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">loader</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">raspberryPi</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">enable</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">version</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-numeric z-nix">4</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">boot</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">loader</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">grub</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">enable</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">false</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">boot</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">loader</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">generic-extlinux-compatible</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">enable</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">hardware</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">enableRedistributableFirmware</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">nixpkgs</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">config</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">allowUnfree</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">networking</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">hostName</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>nixpi<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">networking</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">hostId</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>3bc6efa8<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> <span class="z-comment z-line z-number-sign z-nix"># For ZFS</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">networking</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">networkmanager</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">enable</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-comment z-line z-number-sign z-nix"># Set your time zone.</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">time</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">timeZone</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>Europe/London<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">networking</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">useDHCP</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">false</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">networking</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">interfaces</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">eth0</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">useDHCP</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">networking</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">interfaces</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">wlan0</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">useDHCP</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-comment z-line z-number-sign z-nix"># Select internationalisation properties.</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">i18n</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">defaultLocale</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>en_GB.UTF-8<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">console</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">font</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>Lat2-Terminus16<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">keyMap</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>uk<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">services</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">xserver</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">enable</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">displayManager</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">lightdm</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">enable</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">desktopManager</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">xfce</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">enable</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-comment z-line z-number-sign z-nix"># Configure keymap in X11</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">services</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">xserver</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">layout</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>gb<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">services</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">xserver</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">xkbOptions</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>eurosign:e<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-comment z-line z-number-sign z-nix"># Enable sound.</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">sound</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">enable</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">hardware</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">pulseaudio</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">enable</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-comment z-line z-number-sign z-nix"># Enabling this caused X to have a black screen on boot</span> </span><span class="z-source z-nix"> <span class="z-comment z-line z-number-sign z-nix"># hardware.raspberry-pi.&quot;4&quot;.fkms-3d.enable = true;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">users</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">users</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">michael</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">isNormalUser</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">extraGroups</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-list z-nix">[</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>wheel<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> <span class="z-comment z-line z-number-sign z-nix"># Enable ‘sudo’ for the user.</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-comment z-line z-number-sign z-nix"># List packages installed in system profile. To search, run:</span> </span><span class="z-source z-nix"> <span class="z-comment z-line z-number-sign z-nix"># $ nix search wget</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">environment</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">systemPackages</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-keyword z-other z-nix">with</span> <span class="z-variable z-parameter z-name z-nix">pkgs</span>; <span class="z-punctuation z-definition z-list z-nix">[</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">vim</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"> <span class="z-comment z-line z-number-sign z-nix"># Enable the OpenSSH daemon.</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">services</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">openssh</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">enable</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">system</span>.<span class="z-entity z-other z-attribute-name z-multipart z-nix">stateVersion</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>21.05<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset z-nix">}</span> </span></code></pre> 2021-07-26T00:00:00+00:00 2021-07-26T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/nixos-on-kimsufi/ <div class="notice"> <p><strong>Note 2:</strong> This was fun at the time, but these days you probably want to use <a href="https://github.com/nix-community/nixos-anywhere">nixos-anywhere</a> and probably <a href="https://github.com/nix-community/disko">disko</a>.</p> </div> <p>I’ve been using a few EC2 and DigitalOcean machines for various small projects recently, but I was getting a little bored of the cost of all the little machines I’d ended up setting up for different purposes. I decided to consolidate them all into one hardware machine upon which I am going to run some containers (probably LXD, which I anticipate will be a future blog post). I opted to get a Kimsufi machine to see if I can persuade it to run NixOS. That way, the things I do which are Nix-native can live in NixOS containers, and all the other things that live on Ubuntu or similar can run inside LXD. There’s the theory. As it’s an experiment, I got one of the cheaper machines with a single disk to start with.</p> <p>Installing a deeply non-standard OS on a machine to which you have no console access is fun. I complicated my life further by deciding to use ZFS to learn more about that too. A couple of hours of research led me to two separate web pages that I decided to collide together:</p> <ol> <li><a href="https://lewo.abesis.fr/posts/2019-12-01-install-nixos-on-kimsufi.html">How to install NixOS on kimsufi</a></li> <li><a href="https://nixos.wiki/wiki/NixOS_on_ZFS">NixOS on ZFS on the NixOS Wiki</a></li> </ol> <h2 id="oops">Oops<a class="zola-anchor" style="display: none;" href="#oops" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>The major mistake I made through this process was assuming that the Kimsufi box would be happy enough booting using UEFI. Not so! I did all the setup and installed everything, and upon rebooting, my machine fell off the planet. It was unhappy to the point where it wouldn’t boot the rescue system any more and support had to intervene. Redoing it all assuming BIOS worked.</p> <h2 id="how-to">How-to<a class="zola-anchor" style="display: none;" href="#how-to" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>Set your machine to netboot into the rescue system using the Kimsufi control panel, and reboot it. You’ll get a password for the root user via email. The rescue system starts the machine up using a Linux distro provided over a read-only NFS, and a home directory which is in a <code>tmpfs</code> in memory. We’ll use this space to install the Nix tools later on. We’re logged in as root which gives us a fair amount of flexibility. The machine’s own hard disk is not mounted by default–we’ll do this after partitioning.</p> <p>The first steps are to log in via SSH, and flatten the disk, then set up the partition layout.</p> <h3 id="partitioning-and-formatting">Partitioning and formatting<a class="zola-anchor" style="display: none;" href="#partitioning-and-formatting" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <p>This layout is mostly inspired by the first post I mentioned above. It’s a 2TB disk and the machine has 8GB RAM, so I created a 512MB partition for <code>/boot</code>, then most of the disk is taken up by the ZFS partition, and the final 8GB is swap. I’m told it is theoretically possible to have <code>/boot</code> inside the ZFS pool but it can be hard to make it work reliably with <code>grub</code>, so I’m going to keep that one as <code>ext4</code>.</p> <p>I’m going to present all these commands as if it were a script, but I did it all interactively.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">wipefs</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>a</span> /dev/sda</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> This sets a variable called DISK to save some typing later</span><span class="z-comment z-line z-number-sign z-shell"> </span></span><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> It should be set to the file starting with ata in the directory</span><span class="z-comment z-line z-number-sign z-shell"> </span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-storage z-modifier z-shell">export</span> <span class="z-variable z-other z-readwrite z-assignment z-shell">DISK</span><span class="z-keyword z-operator z-assignment z-shell">=</span><span class="z-string z-unquoted z-shell">/dev/disk/by-id/ata<span class="z-keyword z-operator z-regexp z-quantifier z-shell">*</span></span></span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">parted</span></span><span class="z-meta z-function-call z-arguments z-shell"> /dev/sda<span class="z-keyword z-operator z-end-of-options z-shell"> --</span></span><span class="z-meta z-function-call z-arguments z-shell"> mklabel msdos</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">parted</span></span><span class="z-meta z-function-call z-arguments z-shell"> /dev/sda<span class="z-keyword z-operator z-end-of-options z-shell"> --</span></span><span class="z-meta z-function-call z-arguments z-shell"> mkpart primary 1MiB 512MiB <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> /dev/sda1 is /boot</span><span class="z-comment z-line z-number-sign z-shell"> </span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">parted</span></span><span class="z-meta z-function-call z-arguments z-shell"> /dev/sda<span class="z-keyword z-operator z-end-of-options z-shell"> --</span></span><span class="z-meta z-function-call z-arguments z-shell"> mkpart primary 512MiB -8GiB <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> This is the ZFS partition</span><span class="z-comment z-line z-number-sign z-shell"> </span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">parted</span></span><span class="z-meta z-function-call z-arguments z-shell"> /dev/sda<span class="z-keyword z-operator z-end-of-options z-shell"> --</span></span><span class="z-meta z-function-call z-arguments z-shell"> mkpart primary linux-swap -8GiB 100<span class="z-meta z-group z-expansion z-job z-shell"><span class="z-punctuation z-definition z-variable z-job z-shell">%</span></span> <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> Swap</span><span class="z-comment z-line z-number-sign z-shell"> </span></span></span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">mkfs.ext4</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>L</span> boot <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">DISK</span></span>-part1 <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> Format /boot</span><span class="z-comment z-line z-number-sign z-shell"> </span></span></span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> Create the zpool</span><span class="z-comment z-line z-number-sign z-shell"> </span></span><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> There are lots of options here: see the wiki page above</span><span class="z-comment z-line z-number-sign z-shell"> </span></span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">zpool</span></span><span class="z-meta z-function-call z-arguments z-shell"> create<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>f</span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>O</span> mountpoint=none rpool <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">DISK</span></span>-part2</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> Create some datasets inside the pool</span><span class="z-comment z-line z-number-sign z-shell"> </span></span><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> I&#39;m going to add another later for LXD but this is enough for now</span><span class="z-comment z-line z-number-sign z-shell"> </span></span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">zfs</span></span><span class="z-meta z-function-call z-arguments z-shell"> create<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>o</span> mountpoint=legacy rpool/root</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">zfs</span></span><span class="z-meta z-function-call z-arguments z-shell"> create<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>o</span> mountpoint=legacy rpool/root/nixos</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">zfs</span></span><span class="z-meta z-function-call z-arguments z-shell"> create<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>o</span> mountpoint=legacy rpool/home</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">mount</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>t</span> zfs rpool/root/nixos /mnt</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">mkdir</span></span><span class="z-meta z-function-call z-arguments z-shell"> /mnt/home</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">mount</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>t</span> zfs rpool/home /mnt/home</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">mkdir</span></span><span class="z-meta z-function-call z-arguments z-shell"> /mnt/boot</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">mount</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">DISK</span></span>-part1 /mnt/boot</span> </span></code></pre> <p>Now the partition layout is complete. We can move on to the NixOS parts.</p> <h3 id="getting-nix">Getting Nix<a class="zola-anchor" style="display: none;" href="#getting-nix" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <p>There are some prerequisites needed before we can use Nix. These are largely from the first post above again.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> Create a user for the Nix daemon</span><span class="z-comment z-line z-number-sign z-shell"> </span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">groupadd</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>g</span> 30000 nixbld</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">useradd</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>u</span> 30000<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>g</span> nixbld<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>G</span> nixbld nixbld</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">useradd</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>m</span> setupuser</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> Set this to something you will remember, you&#39;ll need it in a moment</span><span class="z-comment z-line z-number-sign z-shell"> </span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">passwd</span></span><span class="z-meta z-function-call z-arguments z-shell"> setupuser </span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sed</span></span><span class="z-meta z-function-call z-arguments z-shell"> s/root/setupuser/ /etc/sudoers<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>i</span></span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">su</span></span><span class="z-meta z-function-call z-arguments z-shell"> setupuser<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>s</span> /bin/bash</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"> <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> Install Nix</span><span class="z-comment z-line z-number-sign z-shell"> </span></span><span class="z-source z-shell z-bash"> <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> -L is required because the script is now a redirect</span><span class="z-comment z-line z-number-sign z-shell"> </span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">curl</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>L</span> https://nixos.org/nix/install</span> <span class="z-keyword z-operator z-logical z-pipe z-shell">|</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sh</span></span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> This next line makes the Nix tools available in our shell</span><span class="z-comment z-line z-number-sign z-shell"> </span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-support z-function z-source z-shell">source</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">HOME</span></span>/.nix-profile/etc/profile.d/nix.sh</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">nix-channel</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>add</span> https://nixos.org/channels/nixos-21.05 nixpkgs</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">nix-channel</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>update</span></span> </span></code></pre> <p>We need to take a minor detour here as <code>nixos-generate-config</code> is not installed by default.</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">nix-shell</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>p</span> nixos-generate-config</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sudo</span></span><span class="z-meta z-function-call z-arguments z-shell"> <span class="z-meta z-group z-expansion z-command z-backticks z-shell"><span class="z-punctuation z-section z-group z-begin z-shell">`</span><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">which</span></span><span class="z-meta z-function-call z-arguments z-shell"> nixos-generate-config</span><span class="z-punctuation z-section z-group z-end z-shell">`</span></span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>root</span> /mnt</span> </span></code></pre> <p>This will place the usual configuration in <code>/etc/nixos/configuration.nix</code> and <code>/etc/nixos/hardware-configuration.nix</code>.</p> <h4 id="hardware-configuration-nix">hardware-configuration.nix<a class="zola-anchor" style="display: none;" href="#hardware-configuration-nix" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h4> <p>I didn’t change very much in here, I only added these two lines:</p> <pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-variable z-parameter z-name z-nix">boot</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">initrd</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">supportedFilesystems</span> <span class="z-invalid z-illegal">=</span> <span class="z-punctuation z-definition z-list z-nix">[</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>zfs<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-invalid z-illegal">;</span> </span><span class="z-source z-nix"><span class="z-variable z-parameter z-name z-nix">boot</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">supportedFilesystems</span> <span class="z-invalid z-illegal">=</span> <span class="z-punctuation z-definition z-list z-nix">[</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>zfs<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-invalid z-illegal">;</span> </span></code></pre> <h4 id="configuration-nix">configuration.nix<a class="zola-anchor" style="display: none;" href="#configuration-nix" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h4> <p>I changed more in here. I’m not going to paste the entire config in here, but I will call out the key parts that were needed for this to work.</p> <pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"> </span><span class="z-source z-nix"><span class="z-comment z-line z-number-sign z-nix"># This machine is very insistent on BIOS boot, so configure that</span> </span><span class="z-source z-nix"><span class="z-variable z-parameter z-name z-nix">boot</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">loader</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">grub</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">enable</span> <span class="z-invalid z-illegal">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-invalid z-illegal">;</span> </span><span class="z-source z-nix"><span class="z-variable z-parameter z-name z-nix">boot</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">loader</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">grub</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">version</span> <span class="z-invalid z-illegal">=</span> <span class="z-constant z-numeric z-nix">2</span><span class="z-invalid z-illegal">;</span> </span><span class="z-source z-nix"><span class="z-variable z-parameter z-name z-nix">boot</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">loader</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">grub</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">device</span> <span class="z-invalid z-illegal">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>/dev/sda<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-invalid z-illegal">;</span> <span class="z-comment z-line z-number-sign z-nix"># or &quot;nodev&quot; for efi only</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"><span class="z-variable z-parameter z-name z-nix">networking</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">hostName</span> <span class="z-invalid z-illegal">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>metis<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-invalid z-illegal">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"><span class="z-comment z-line z-number-sign z-nix"># Make a random 8-character hex string for this. ZFS requires it.</span> </span><span class="z-source z-nix"><span class="z-variable z-parameter z-name z-nix">networking</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">hostId</span> <span class="z-invalid z-illegal">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>13371337<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-invalid z-illegal">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"><span class="z-comment z-line z-number-sign z-nix"># Check what the rescue system&#39;s IP, route and optionally nameservers are</span> </span><span class="z-source z-nix"><span class="z-comment z-line z-number-sign z-nix"># (or just use 8.8.8.8 for DNS if you like)</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"><span class="z-variable z-parameter z-name z-nix">networking</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">usePredictableInterfaceNames</span> <span class="z-invalid z-illegal">=</span> <span class="z-constant z-language z-nix">false</span><span class="z-invalid z-illegal">;</span> </span><span class="z-source z-nix"><span class="z-variable z-parameter z-name z-nix">networking</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">interfaces</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">eth0</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">ipv4</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">addresses</span> <span class="z-invalid z-illegal">=</span> <span class="z-punctuation z-definition z-list z-nix">[</span><span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">address</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>192.0.2.1<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">prefixLength</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-numeric z-nix">24</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-invalid z-illegal">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"><span class="z-variable z-parameter z-name z-nix">networking</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">defaultGateway</span> <span class="z-invalid z-illegal">=</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>192.0.2.254<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span><span class="z-invalid z-illegal">;</span> </span><span class="z-source z-nix"><span class="z-variable z-parameter z-name z-nix">networking</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">nameservers</span> <span class="z-invalid z-illegal">=</span> <span class="z-punctuation z-definition z-list z-nix">[</span> <span class="z-string z-quoted z-double z-nix"><span class="z-punctuation z-definition z-string z-double z-start z-nix">&quot;</span>8.8.8.8<span class="z-punctuation z-definition z-string z-double z-end z-nix">&quot;</span></span> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-invalid z-illegal">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"><span class="z-variable z-parameter z-name z-nix">services</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">openssh</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">enable</span> <span class="z-invalid z-illegal">=</span> <span class="z-constant z-language z-nix">true</span><span class="z-invalid z-illegal">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"><span class="z-comment z-line z-number-sign z-nix"># You can enable this once you&#39;ve proved the machine boots</span> </span><span class="z-source z-nix"><span class="z-variable z-parameter z-name z-nix">networking</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">firewall</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">enable</span> <span class="z-invalid z-illegal">=</span> <span class="z-constant z-language z-nix">false</span><span class="z-invalid z-illegal">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"><span class="z-variable z-parameter z-name z-nix">users</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">users</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">michael</span> <span class="z-invalid z-illegal">=</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-comment z-line z-number-sign z-nix"># Usual user config goes here. Be sure to include your SSH public key</span> </span><span class="z-source z-nix"><span class="z-invalid z-illegal">}</span> </span><span class="z-source z-nix"> </span></code></pre> <h3 id="installing-nixos">Installing NixOS<a class="zola-anchor" style="display: none;" href="#installing-nixos" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <p>I again had to take a detour here, because something is a little broken in NixOS 21.05 at the moment, resulting in errors like this:</p> <pre class="z-code"><code><span class="z-text z-plain">getting attributes of path /nix/store/dbri2d4r470fc6nrh95qa8bwcj54wh1q-zfs-kernel-2.0.5-5.10.52: No such file or directory </span></code></pre> <p>I have no idea what causes this, and perhaps it’s fixed by the time you read this, but it is documented in <a href="https://github.com/NixOS/nixpkgs/issues/126141#issuecomment-861724383">this GitHub issue</a>. The workaround described at that comment allowed me to continue, with this fairly contorted pair of commands:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sudo</span></span><span class="z-meta z-function-call z-arguments z-shell"> PATH=<span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">PATH</span></span><span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span> NIX_PATH=<span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">NIX_PATH</span></span><span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span> <span class="z-meta z-group z-expansion z-command z-backticks z-shell"><span class="z-punctuation z-section z-group z-begin z-shell">`</span><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">which</span></span><span class="z-meta z-function-call z-arguments z-shell"> nix-build</span><span class="z-punctuation z-section z-group z-end z-shell">`</span></span> <span class="z-string z-quoted z-single z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&#39;</span>&lt;nixpkgs/nixos&gt;<span class="z-punctuation z-definition z-string z-end z-shell">&#39;</span></span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>A</span> config.system.build.toplevel<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>I</span> nixos-config=/mnt/etc/nixos/configuration.nix</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sudo</span></span><span class="z-meta z-function-call z-arguments z-shell"> PATH=<span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">PATH</span></span><span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span> NIX_PATH=<span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span><span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">NIX_PATH</span></span><span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span> <span class="z-meta z-group z-expansion z-command z-backticks z-shell"><span class="z-punctuation z-section z-group z-begin z-shell">`</span><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">which</span></span><span class="z-meta z-function-call z-arguments z-shell"> nixos-install</span><span class="z-punctuation z-section z-group z-end z-shell">`</span></span><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>root</span> /mnt</span> </span></code></pre> <p>If all goes well, you should be asked to set a root password. You may encounter quite a lot of errors about locales being missing–I ignored these as they didn’t seem to matter.</p> <p>At this point, you should have a NixOS system ready to go on the machine’s internal disk. You’ll need to use the Kimsufi control panel to set the machine back to boot from its own hard disk, using the menu option marked Netboot.</p> <p>Now, all you can do is <code>reboot</code> and cross your fingers. It worked for me, although it’s worth noting that it took a good couple of minutes for the machine to come up and respond to SSH. You can connect as the user you set up in <code>configuration.nix</code>.</p> <p>You will probably encounter an error when you connect because the SSH host key will have changed. You can fix this with <code>ssh-keygen -R 192.0.2.1</code> (replacing the IP with the one for your machine), and then try again. Hopefully you’ve now got a new NixOS machine up and running and ready to do whatever you need it to do.</p> <h2 id="thanks">Thanks<a class="zola-anchor" style="display: none;" href="#thanks" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>Big thanks to <a href="https://grahamc.com">Graham Christensen</a> of Nix specialists <a href="https://determinate.systems/">Determinate Systems</a> for helping me out throughout this process.</p> 2021-07-19T00:00:00+00:00 2021-07-19T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/systemd-socket-activation/ <p>My <a href="https://mgdm.net/weblog/systemd/">last post on this topic</a> caught some attention, so I’m going to continue exploring some of the systemd features that may be useful to people writing network services. Here are some more things about socket activation I didn’t cover in the previous post.</p> <p>Firstly, it’s not something that needs a binding to a specific library to work (although systemd does make one available for this task). It uses conventions that already exist in Unix, and have done for a long time. The network sockets are presented as file descriptors, and some information about them as environment variables.</p> <h2 id="inetd">inetd<a class="zola-anchor" style="display: none;" href="#inetd" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>As I mentioned in the previous post, socket activation is pretty similar to how <code>inetd</code> operates. <code>inetd</code> is a “super-server”, which first appeared in 4.3BSD in 1986. There are other implementations of the same idea–<code>xinetd</code> is a common one using a different configuration format.</p> <p>The idea is that it listens on a set of ports normally configured in <code>/etc/inetd.conf</code>, and then any time a new connection arrives on that port, it will spawn a new instance of the configured process and send the incoming data to its standard input, and the process’s output back to the client. An example configuration for the <a href="https://datatracker.ietf.org/doc/html/rfc865">Quote of the Day service</a> in the original <code>inetd</code> may look like:</p> <pre class="z-code"><code><span class="z-text z-plain">qotd stream tcp nowait root /usr/sbin/tcpd /usr/games/fortune </span></code></pre> <p>The fields are:</p> <ol> <li>The service name. This references a service list in <code>/etc/services</code>. In this case, <code>qotd</code> maps to port 17.</li> <li>It’s a <code>stream</code> service operating over TCP, as opposed to a <code>dgram</code> service operating on UDP</li> <li><code>nowait</code> means to spawn a new instance of the process for every connection, instead of waiting for a single process to deal with each connection</li> <li>This service is configured to run as root, which seems like an awful idea but may possibly be required for <code>tcpd</code>.</li> <li>The final fields are the command to run on each incoming connection. <code>tcpd</code> is a program that does some basic filtering of incoming traffic. I’ll skip over this for the moment.</li> </ol> <h2 id="using-systemd-like-inetd">Using systemd like inetd<a class="zola-anchor" style="display: none;" href="#using-systemd-like-inetd" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>We can use systemd for the same purpose as <code>inetd</code>, by configuring it to create processes on demand and connecting the network socket to standard input and output.</p> <p>Let’s try it out, and use some of the features I talked about before to protect the service, as the <code>fortune</code> command isn’t really designed to be connected to a network so we don’t want it to get exploited. I’m on Ubuntu 21.04 here, but this should work on most distributions.</p> <p>Create a file called <code>/etc/systemd/system/fortune@.service</code>, and put the following in it:</p> <pre data-lang="ini" class="language-ini z-code"><code class="language-ini" data-lang="ini"><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Unit] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">Description</span><span class="z-keyword z-operator z-genconfig">=</span></span>QoTD </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Service] </span></span><span class="z-source z-genconfig"><span class="z-meta z-comment z-genconfig"><span class="z-comment z-line z-number-sign z-genconfig"># Note the - to make systemd ignore the exit code </span></span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">ExecStart</span><span class="z-keyword z-operator z-genconfig">=</span></span><span class="z-keyword z-operator z-genconfig">-</span><span class="z-keyword z-operator z-genconfig">/</span>usr<span class="z-keyword z-operator z-genconfig">/</span>games<span class="z-keyword z-operator z-genconfig">/</span>fortune </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-meta z-comment z-genconfig"><span class="z-comment z-line z-number-sign z-genconfig"># This is the part that makes it work like inetd </span></span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">StandardOutput</span><span class="z-keyword z-operator z-genconfig">=</span></span>socket </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-meta z-comment z-genconfig"><span class="z-comment z-line z-number-sign z-genconfig"># Run as a dynamic user </span></span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">DynamicUser</span><span class="z-keyword z-operator z-genconfig">=</span></span><span class="z-constant z-language z-genconfig">yes</span> </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-meta z-comment z-genconfig"><span class="z-comment z-line z-number-sign z-genconfig"># ProtectSystem=strict # Implied by DynamicUser=yes </span></span></span><span class="z-source z-genconfig"><span class="z-meta z-comment z-genconfig"><span class="z-comment z-line z-number-sign z-genconfig"># PrivateTmp=true # Also implied by DynamicUser=yes </span></span></span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-meta z-comment z-genconfig"><span class="z-comment z-line z-number-sign z-genconfig"># DynamicUser implies ProtectHome=read-only, but we do not need home directories </span></span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">ProtectHome</span><span class="z-keyword z-operator z-genconfig">=</span></span><span class="z-constant z-language z-genconfig">true</span> </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-meta z-comment z-genconfig"><span class="z-comment z-line z-number-sign z-genconfig"># We also do not need to see users </span></span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">PrivateUsers</span><span class="z-keyword z-operator z-genconfig">=</span></span><span class="z-constant z-language z-genconfig">true</span> </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Install] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">WantedBy</span><span class="z-keyword z-operator z-genconfig">=</span></span>multi<span class="z-keyword z-operator z-genconfig">-</span>user<span class="z-keyword z-operator z-genconfig">.</span>target </span></code></pre> <p>Note the comments–some of the features I talked about before are enabled automatically when choosing <code>DynamicUser=true</code>. Also note the <code>@</code> in the filename - this is significant as it indicates the service is a template, and that a new instance of the service will be run on every connection.</p> <p>Then put this in <code>/etc/systemd/system/fortune.socket</code>:</p> <pre data-lang="ini" class="language-ini z-code"><code class="language-ini" data-lang="ini"><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Socket] </span></span><span class="z-source z-genconfig"><span class="z-meta z-comment z-genconfig"><span class="z-comment z-line z-number-sign z-genconfig"># Listen on TCP port 17 </span></span></span><span class="z-source z-genconfig"><span class="z-meta z-comment z-genconfig"><span class="z-comment z-line z-number-sign z-genconfig"># Use ListenDatagram to listen on UDP </span></span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">ListenStream</span> <span class="z-keyword z-operator z-genconfig">=</span></span> <span class="z-constant z-numeric z-genconfig">17</span> </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-meta z-comment z-genconfig"><span class="z-comment z-line z-number-sign z-genconfig"># Call accept(3) on the socket before handing it to the process </span></span></span><span class="z-source z-genconfig"><span class="z-meta z-comment z-genconfig"><span class="z-comment z-line z-number-sign z-genconfig"># This is necessary because fortune knows nothing about the network </span></span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">Accept</span><span class="z-keyword z-operator z-genconfig">=</span></span><span class="z-constant z-language z-genconfig">yes</span> </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Install] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">WantedBy</span> <span class="z-keyword z-operator z-genconfig">=</span></span> sockets<span class="z-keyword z-operator z-genconfig">.</span>target </span></code></pre> <p>After this, you’ll need to do <code>systemctl daemon-reload</code> followed by <code>systemctl start fortune.socket</code> (and <code>systemctl enable fortune.socket</code> if you want it to start on boot).</p> <p>Now, we’ve built a QOTD service–port 17 is the IANA registered port number for this service. To try it out, you can use <code>nc</code> or perhaps <code>telnet</code>:</p> <pre class="z-code"><code><span class="z-text z-plain">mgdm@io:~$ nc localhost 17 </span><span class="z-text z-plain">Never trust an operating system. </span></code></pre> <p>It’s good advice!</p> <h2 id="socket-presentation-in-systemd">Socket presentation in systemd<a class="zola-anchor" style="display: none;" href="#socket-presentation-in-systemd" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>Socket activation using systemd’s own convention is similar to <code>inetd</code>, but has more room for expansion. The sockets themselves are presented to the process as file descriptors, similar to the inetd approach. The main difference that systemd does not use the standard streams, to allow it to supply more than one socket. It uses file descriptors numbered 3 and above. You can configure more than just the one I demonstrated above. If you specify more than one, they’ll be presented in the same order as they appear in the <code>.socket</code> file.</p> <p>In order for the process to be able to understand what to do with the sockets, a couple of environment variables are used. Firstly, <code>LISTEN_FDS</code> gives the number of sockets configured. For example, if this is set to <code>2</code>, the process will be able to work out that it has been given listening sockets on FDs 3 and 4.</p> <p>It’s handy that you can configure several listening ports, but you also need to be able to work out which is which. Some software may want to listen on more than one port, and it may or may not be obvious which protocol should be configured on each–consider a hypothetical web server that may want to present plaintext HTTP on port 80 and HTTPS on port 443. In this case it’s probably quite easy to work out which service needs to go where from the port numbers, but regardless, it is possible to supply a name for each socket to make this slightly more foolproof. A variable called <code>LISTEN_FDNAMES</code> can be set with a colon-separated list of the name to be given to each socket.</p> <h2 id="example">Example<a class="zola-anchor" style="display: none;" href="#example" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>Let’s extend <code>lunchd</code> from <a href="https://mgdm.net/weblog/systemd/">my previous post</a>, and in the process finish the exercise I left at the end, by making it listen on both HTTP and HTTPS. On the HTTP port, we’ll just redirect every request we get to HTTPS.</p> <p>The original <code>lunchd.socket</code> looked like this:</p> <pre data-lang="ini" class="language-ini z-code"><code class="language-ini" data-lang="ini"><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Socket] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">ListenStream</span> <span class="z-keyword z-operator z-genconfig">=</span></span> <span class="z-constant z-numeric z-genconfig">443</span> </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">BindIPv6Only</span> <span class="z-keyword z-operator z-genconfig">=</span></span> both </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Install] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">WantedBy</span> <span class="z-keyword z-operator z-genconfig">=</span></span> sockets<span class="z-keyword z-operator z-genconfig">.</span>target </span></code></pre> <p>We need to modify it slightly and then rename it. Add the extra <code>Service</code> and <code>FileDescriptorName</code> lines so it looks like below, and then rename it to <code>lunchd-https.socket</code>.</p> <pre data-lang="ini" class="language-ini z-code"><code class="language-ini" data-lang="ini"><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Socket] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">ListenStream</span> <span class="z-keyword z-operator z-genconfig">=</span></span> <span class="z-constant z-numeric z-genconfig">443</span> </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">BindIPv6Only</span> <span class="z-keyword z-operator z-genconfig">=</span></span> both </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">FileDescriptorName</span> <span class="z-keyword z-operator z-genconfig">=</span></span> https </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">Service</span><span class="z-keyword z-operator z-genconfig">=</span></span>lunchd<span class="z-keyword z-operator z-genconfig">.</span>service </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Install] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">WantedBy</span> <span class="z-keyword z-operator z-genconfig">=</span></span> sockets<span class="z-keyword z-operator z-genconfig">.</span>target </span></code></pre> <p>Then, copy it to <code>lunchd-http.socket</code> and change the port number there to 80 and the <code>FileDescriptorName</code> to <code>http</code>.</p> <pre data-lang="ini" class="language-ini z-code"><code class="language-ini" data-lang="ini"><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Socket] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">ListenStream</span> <span class="z-keyword z-operator z-genconfig">=</span></span> <span class="z-constant z-numeric z-genconfig">80</span> </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">BindIPv6Only</span> <span class="z-keyword z-operator z-genconfig">=</span></span> both </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">FileDescriptorName</span> <span class="z-keyword z-operator z-genconfig">=</span></span> http </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">Service</span><span class="z-keyword z-operator z-genconfig">=</span></span>lunchd<span class="z-keyword z-operator z-genconfig">.</span>service </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Install] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">WantedBy</span> <span class="z-keyword z-operator z-genconfig">=</span></span> sockets<span class="z-keyword z-operator z-genconfig">.</span>target </span></code></pre> <p>We now have two <code>.socket</code> files which is required if we want to present more than one different class of listening socket. By default, the names handed to the process for each socket type will be the same as the <code>.socket</code> files, but in this case we’re using the <code>FileDescriptorName</code> option to rename them.</p> <p>Next up, we need to make a little change to the actual <code>lunchd</code> code. The CoreOS library I’ve been using since the last post knows to look for the <code>LISTEN_FDS</code> variable to find out how many file descriptors have been presented, and also has the ability to look at <code>LISTEN_FDNAMES</code> to group them by name. To make use of this latter feature we’re going to switch to using <code>activation.ListenersWithNames()</code> instead of <code>activation.Listeners()</code>. This returns a <code>map[string][]net.Listener</code>. The keys in the map are the names configured in each <code>FileDescriptorName</code>, and the values are a slice of <code>net.Listener</code>s (as you can configure multiple <code>ListenStream</code> options in each file to listen on several interfaces or ports).</p> <p>This code is now reaching the point where I should be refactoring it a little into more than one file, but I’ll keep it as one here just so everything’s on the same page. The key changes are that instead of assuming it’s going to listen on one port, it now looks into the map to work out what to listen on, and spins up a new <code>http.ServeMux</code> for each. Any sockets presented in the key <code>http</code> will listen on plain HTTP, and those in <code>https</code> will run TLS. Any other keys in the map are ignored. In the case where the map is empty (indicating the process is not being run using systemd, or not with socket activation), it’ll default to an unprivileged port (8443). In this way the code still runs happily on other Unix systems such as macOS.</p> <p>I’ve added a little more logging to this code, so you can see what the environment variables look like when systemd starts the process. In this case, I had port 80 configured for HTTP and port 443 configured for HTTPS, so <code>LISTEN_FDS</code> and <code>LISTEN_FDNAMES</code> looked like the below. The first line indicates there are 2 sockets, and the second denotes the protocols for each.</p> <pre class="z-code"><code><span class="z-text z-plain">Jul 18 11:30:23 io lunchd[10425]: 2021/07/19 11:30:23 LISTEN_FDS is 2 </span><span class="z-text z-plain">Jul 18 11:30:23 io lunchd[10425]: 2021/07/19 11:30:23 LISTEN_FDNAMES is http:https </span></code></pre> <p>The code is <a href="https://mgdm.net/weblog/systemd-socket-activation/#final-code">below</a>, but it’s also available <a href="https://github.com/mgdm/lunchd/blob/v2/main.go">in a branch in the GitHub repo from last time</a>.</p> <h2 id="references">References<a class="zola-anchor" style="display: none;" href="#references" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <ul> <li>This post is quite heavily inspired by <a href="http://0pointer.de/blog/projects/socket-activation.html">systemd for Developers I</a> by Lennart Poettering</li> <li>Thanks to <a href="https://bigkevmcd.github.io/">Kevin McDermott</a> and <a href="https://twitter.com/jogbert/">James O’Gorman</a> for some advice on the Go code on both this and my last post</li> </ul> <h2 id="final-code">Final code<a class="zola-anchor" style="display: none;" href="#final-code" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <pre data-linenos data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><table><tbody><tr><td>1</td><td><span class="z-source z-go"><span class="z-keyword z-other z-package z-go">package</span> <span class="z-variable z-other z-go">main</span> </span></td></tr><tr><td>2</td><td><span class="z-source z-go"> </span></td></tr><tr><td>3</td><td><span class="z-source z-go"><span class="z-keyword z-other z-import z-go">import</span> <span class="z-punctuation z-section z-parens z-begin z-go">(</span> </span></td></tr><tr><td>4</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>crypto/tls<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>5</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>errors<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>6</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>flag<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>7</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>fmt<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>8</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>log<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>9</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>math/rand<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>10</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>net<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>11</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>net/http<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>12</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>sync<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>13</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>time<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>14</td><td><span class="z-source z-go"> </span></td></tr><tr><td>15</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>github.com/coreos/go-systemd/activation<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>16</td><td><span class="z-source z-go"><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></td></tr><tr><td>17</td><td><span class="z-source z-go"> </span></td></tr><tr><td>18</td><td><span class="z-source z-go"><span class="z-storage z-type z-keyword z-var z-go">var</span> <span class="z-variable z-declaration z-go">lunchOptions</span> <span class="z-keyword z-operator z-assignment z-go">=</span> <span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-punctuation z-section z-brackets z-end z-go">]</span><span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></td></tr><tr><td>19</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Sandwich<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> </span></span></td></tr><tr><td>20</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Soup<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> </span></span></td></tr><tr><td>21</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Salad<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> </span></span></td></tr><tr><td>22</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Burger<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> </span></span></td></tr><tr><td>23</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Sushi<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> </span></span></td></tr><tr><td>24</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></td></tr><tr><td>25</td><td><span class="z-source z-go"> </span></td></tr><tr><td>26</td><td><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">getRandomLunch</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></td></tr><tr><td>27</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-variable z-other z-go">lunchOptions</span><span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-variable z-other z-go">rand</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Intn</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-function z-go"><span class="z-support z-function z-builtin z-go">len</span></span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">lunchOptions</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-brackets z-end z-go">]</span> </span></span></td></tr><tr><td>28</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></td></tr><tr><td>29</td><td><span class="z-source z-go"> </span></td></tr><tr><td>30</td><td><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">getDefaultListeners</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-storage z-type z-keyword z-map z-go">map</span><span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span><span class="z-punctuation z-section z-brackets z-end z-go">]</span><span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-punctuation z-section z-brackets z-end z-go">]</span><span class="z-variable z-other z-go">net</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">Listener</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">error</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></td></tr><tr><td>31</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>32</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">listener</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-declaration z-go">err</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">net</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Listen</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>tcp<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>:8443<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>33</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>34</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">if</span> <span class="z-variable z-other z-go">err</span> <span class="z-keyword z-operator z-go">!=</span> <span class="z-constant z-language z-go">nil</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></td></tr><tr><td>35</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-constant z-language z-go">nil</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">err</span> </span></span></span></td></tr><tr><td>36</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span></td></tr><tr><td>37</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>38</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-storage z-type z-keyword z-map z-go">map</span><span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span><span class="z-punctuation z-section z-brackets z-end z-go">]</span><span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-punctuation z-section z-brackets z-end z-go">]</span><span class="z-variable z-other z-go">net</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">Listener</span><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></td></tr><tr><td>39</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>https<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">:</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span><span class="z-variable z-other z-go">listener</span><span class="z-punctuation z-section z-braces z-end z-go">}</span></span><span class="z-punctuation z-separator z-go">,</span> </span></span></span></td></tr><tr><td>40</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-constant z-language z-go">nil</span> </span></span></td></tr><tr><td>41</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></td></tr><tr><td>42</td><td><span class="z-source z-go"> </span></td></tr><tr><td>43</td><td><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">getListeners</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-storage z-type z-keyword z-map z-go">map</span><span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span><span class="z-punctuation z-section z-brackets z-end z-go">]</span><span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-punctuation z-section z-brackets z-end z-go">]</span><span class="z-variable z-other z-go">net</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">Listener</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">error</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></td></tr><tr><td>44</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">listeners</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-declaration z-go">err</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">activation</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">ListenersWithNames</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>45</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>46</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">if</span> <span class="z-variable z-other z-go">err</span> <span class="z-keyword z-operator z-go">!=</span> <span class="z-constant z-language z-go">nil</span> <span class="z-keyword z-operator z-go">||</span> <span class="z-variable z-function z-go"><span class="z-support z-function z-builtin z-go">len</span></span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">listeners</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-keyword z-operator z-go">==</span> <span class="z-constant z-numeric z-integer z-decimal z-go">0</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></td></tr><tr><td>47</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">log</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Printf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Received no listeners from socket activation, defaulting to HTTPS on port 8443<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>48</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">listeners</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">err</span> <span class="z-keyword z-operator z-assignment z-go">=</span> <span class="z-variable z-function z-go">getDefaultListeners</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>49</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span></td></tr><tr><td>50</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>51</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-variable z-other z-go">listeners</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">err</span> </span></span></td></tr><tr><td>52</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></td></tr><tr><td>53</td><td><span class="z-source z-go"> </span></td></tr><tr><td>54</td><td><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">getCertificatePaths</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">error</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></td></tr><tr><td>55</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">keyPath</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">flag</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">String</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>key<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>The path to the private key<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>56</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">certPath</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">flag</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">String</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>certificate<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>The path to the certificate<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>57</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">flag</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Parse</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>58</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>59</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">if</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">keyPath</span> <span class="z-keyword z-operator z-go">==</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> <span class="z-keyword z-operator z-go">||</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">certPath</span> <span class="z-keyword z-operator z-go">==</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></td></tr><tr><td>60</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">errors</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">New</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Either or both of -key or -certificate not set<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>61</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span></td></tr><tr><td>62</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>63</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">keyPath</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">certPath</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-constant z-language z-go">nil</span> </span></span></td></tr><tr><td>64</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></td></tr><tr><td>65</td><td><span class="z-source z-go"> </span></td></tr><tr><td>66</td><td><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">getTLSConfig</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-parameter z-go">keyPath</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-parameter z-go">certPath</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">tls</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">Config</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">error</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></td></tr><tr><td>67</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">config</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-keyword z-operator z-go">&amp;</span><span class="z-variable z-other z-go">tls</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">Config</span><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></td></tr><tr><td>68</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">Certificates</span><span class="z-punctuation z-separator z-go">:</span> <span class="z-variable z-function z-go"><span class="z-support z-function z-builtin z-go">make</span></span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-punctuation z-section z-brackets z-end z-go">]</span><span class="z-variable z-other z-go">tls</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">Certificate</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-constant z-numeric z-integer z-decimal z-go">1</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-separator z-go">,</span> </span></span></span></td></tr><tr><td>69</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">NextProtos</span><span class="z-punctuation z-separator z-go">:</span> <span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-punctuation z-section z-brackets z-end z-go">]</span><span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>h2<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>http/1.1<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-section z-braces z-end z-go">}</span></span><span class="z-punctuation z-separator z-go">,</span> </span></span></span></td></tr><tr><td>70</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">PreferServerCipherSuites</span><span class="z-punctuation z-separator z-go">:</span> <span class="z-constant z-language z-go">true</span><span class="z-punctuation z-separator z-go">,</span> </span></span></span></td></tr><tr><td>71</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span></td></tr><tr><td>72</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>73</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-storage z-type z-keyword z-var z-go">var</span> <span class="z-variable z-declaration z-go">err</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">error</span></span> </span></span></td></tr><tr><td>74</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>75</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">log</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Printf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Loading certs from key: <span class="z-constant z-other z-placeholder z-go">%s</span> and cert: <span class="z-constant z-other z-placeholder z-go">%s</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">keyPath</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">certPath</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>76</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>77</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">config</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">Certificates</span><span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-constant z-numeric z-integer z-decimal z-go">0</span><span class="z-punctuation z-section z-brackets z-end z-go">]</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">err</span> <span class="z-keyword z-operator z-assignment z-go">=</span> <span class="z-variable z-other z-go">tls</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">LoadX509KeyPair</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span> </span></span></td></tr><tr><td>78</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">certPath</span><span class="z-punctuation z-separator z-go">,</span> </span></span></td></tr><tr><td>79</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">keyPath</span><span class="z-punctuation z-separator z-go">,</span> </span></span></td></tr><tr><td>80</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>81</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>82</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">if</span> <span class="z-variable z-other z-go">err</span> <span class="z-keyword z-operator z-go">!=</span> <span class="z-constant z-language z-go">nil</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></td></tr><tr><td>83</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">log</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Printf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Failed to configure TLS: <span class="z-constant z-other z-placeholder z-go">%s</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">err</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>84</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-constant z-language z-go">nil</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">err</span> </span></span></span></td></tr><tr><td>85</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span></td></tr><tr><td>86</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>87</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-variable z-other z-go">config</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-constant z-language z-go">nil</span> </span></span></td></tr><tr><td>88</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></td></tr><tr><td>89</td><td><span class="z-source z-go"> </span></td></tr><tr><td>90</td><td><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">createWebServers</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">ServeMux</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">ServeMux</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></td></tr><tr><td>91</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">httpMux</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">NewServeMux</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>92</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">httpMux</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">HandleFunc</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>/<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-storage z-type z-keyword z-function z-go">func</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-parameter z-go">w</span> <span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">ResponseWriter</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-parameter z-go">req</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">Request</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></td></tr><tr><td>93</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">log</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Printf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>HTTP request from <span class="z-constant z-other z-placeholder z-go">%s</span><span class="z-constant z-character z-escape z-go">\n</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">req</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">RemoteAddr</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>94</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">hostname</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">fmt</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Sprintf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>https://<span class="z-constant z-other z-placeholder z-go">%s</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">req</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">Host</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>95</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Redirect</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">w</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">req</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">hostname</span><span class="z-keyword z-operator z-go">+</span><span class="z-variable z-other z-go">req</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">RequestURI</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">StatusMovedPermanently</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>96</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>97</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>98</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">httpsMux</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">NewServeMux</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>99</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">httpsMux</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">HandleFunc</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>/<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-storage z-type z-keyword z-function z-go">func</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-parameter z-go">w</span> <span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">ResponseWriter</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-parameter z-go">req</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">Request</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></td></tr><tr><td>100</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">log</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Printf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>HTTPS request from <span class="z-constant z-other z-placeholder z-go">%s</span><span class="z-constant z-character z-escape z-go">\n</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">req</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">RemoteAddr</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>101</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">fmt</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Fprintf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">w</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>&lt;h1&gt;<span class="z-constant z-other z-placeholder z-go">%s</span>&lt;/h1&gt;<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-function z-go">getRandomLunch</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>102</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>103</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>104</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-variable z-other z-go">httpMux</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">httpsMux</span> </span></span></td></tr><tr><td>105</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></td></tr><tr><td>106</td><td><span class="z-source z-go"> </span></td></tr><tr><td>107</td><td><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">main</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></td></tr><tr><td>108</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">rand</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Seed</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">time</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Now</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">UnixNano</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>109</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>110</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">log</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Printf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>LISTEN_FDS is <span class="z-constant z-other z-placeholder z-go">%s</span><span class="z-constant z-character z-escape z-go">\n</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">os</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Getenv</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>LISTEN_FDS<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>111</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">log</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Printf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>LISTEN_FDNAMES is <span class="z-constant z-other z-placeholder z-go">%s</span><span class="z-constant z-character z-escape z-go">\n</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">os</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Getenv</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>LISTEN_FDNAMES<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>112</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>113</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">listeners</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-declaration z-go">err</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-function z-go">getListeners</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>114</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">httpMux</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-declaration z-go">httpsMux</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-function z-go">createWebServers</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>115</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>116</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">if</span> <span class="z-variable z-other z-go">err</span> <span class="z-keyword z-operator z-go">!=</span> <span class="z-constant z-language z-go">nil</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></td></tr><tr><td>117</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">log</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Fatalf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Could not set up listeners: <span class="z-constant z-other z-placeholder z-go">%s</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">err</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>118</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span></td></tr><tr><td>119</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>120</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-comment z-line z-go"><span class="z-punctuation z-definition z-comment z-go">//</span> This is used to block the main goroutine and wait for the others </span></span></span></td></tr><tr><td>121</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-storage z-type z-keyword z-var z-go">var</span> <span class="z-variable z-declaration z-go">wg</span> <span class="z-variable z-other z-go">sync</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">WaitGroup</span> </span></span></td></tr><tr><td>122</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>123</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">if</span> <span class="z-variable z-declaration z-go">tlsListeners</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-declaration z-go">ok</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">listeners</span><span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>https<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-section z-brackets z-end z-go">]</span><span class="z-punctuation z-terminator z-go">;</span> <span class="z-variable z-other z-go">ok</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></td></tr><tr><td>124</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">keyPath</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-declaration z-go">certPath</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-declaration z-go">err</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-function z-go">getCertificatePaths</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>125</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> </span></span></span></td></tr><tr><td>126</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">if</span> <span class="z-variable z-other z-go">err</span> <span class="z-keyword z-operator z-go">!=</span> <span class="z-constant z-language z-go">nil</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></span></td></tr><tr><td>127</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">log</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Fatalf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Could not load certificates: <span class="z-constant z-other z-placeholder z-go">%s</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">err</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></span></td></tr><tr><td>128</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span></span></td></tr><tr><td>129</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> </span></span></span></td></tr><tr><td>130</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">tlsConfig</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-declaration z-go">err</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-function z-go">getTLSConfig</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">keyPath</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">certPath</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>131</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> </span></span></span></td></tr><tr><td>132</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">for</span> <span class="z-variable z-declaration z-go">i</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-keyword z-other z-go">range</span> <span class="z-variable z-other z-go">tlsListeners</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></span></td></tr><tr><td>133</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">wg</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Add</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-constant z-numeric z-integer z-decimal z-go">1</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></span></td></tr><tr><td>134</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> </span></span></span></span></td></tr><tr><td>135</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">go</span> <span class="z-storage z-type z-keyword z-function z-go">func</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-parameter z-go">l</span> <span class="z-variable z-other z-go">net</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">Listener</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></span></span></td></tr><tr><td>136</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">log</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Printf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Starting secure web server on port <span class="z-constant z-other z-placeholder z-go">%s</span><span class="z-constant z-character z-escape z-go">\n</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">l</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Addr</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></span></span></td></tr><tr><td>137</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">tl</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">tls</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">NewListener</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">l</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">tlsConfig</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></span></span></td></tr><tr><td>138</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">log</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Fatal</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Serve</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">tl</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">httpsMux</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></span></span></td></tr><tr><td>139</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">tlsListeners</span><span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-variable z-other z-go">i</span><span class="z-punctuation z-section z-brackets z-end z-go">]</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></span></td></tr><tr><td>140</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span></span></td></tr><tr><td>141</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span></td></tr><tr><td>142</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>143</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">if</span> <span class="z-variable z-declaration z-go">plainListeners</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-declaration z-go">ok</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">listeners</span><span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>http<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-section z-brackets z-end z-go">]</span><span class="z-punctuation z-terminator z-go">;</span> <span class="z-variable z-other z-go">ok</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></td></tr><tr><td>144</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">for</span> <span class="z-variable z-declaration z-go">i</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-keyword z-other z-go">range</span> <span class="z-variable z-other z-go">plainListeners</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></span></td></tr><tr><td>145</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">wg</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Add</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-constant z-numeric z-integer z-decimal z-go">1</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></span></td></tr><tr><td>146</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> </span></span></span></span></td></tr><tr><td>147</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">go</span> <span class="z-storage z-type z-keyword z-function z-go">func</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-parameter z-go">l</span> <span class="z-variable z-other z-go">net</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">Listener</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></span></span></td></tr><tr><td>148</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">log</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Printf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Starting plaintext web server on port <span class="z-constant z-other z-placeholder z-go">%s</span><span class="z-constant z-character z-escape z-go">\n</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">l</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Addr</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></span></span></td></tr><tr><td>149</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">log</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Fatal</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Serve</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">l</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">httpMux</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></span></span></td></tr><tr><td>150</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">plainListeners</span><span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-variable z-other z-go">i</span><span class="z-punctuation z-section z-brackets z-end z-go">]</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></span></td></tr><tr><td>151</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span></span></td></tr><tr><td>152</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span></td></tr><tr><td>153</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>154</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-comment z-line z-go"><span class="z-punctuation z-definition z-comment z-go">//</span> The code never calls `wg.Done()` so this will block forever </span></span></span></td></tr><tr><td>155</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-comment z-line z-go"><span class="z-punctuation z-definition z-comment z-go">//</span> Letting the other goroutines carry on serving </span></span></span></td></tr><tr><td>156</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">wg</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Wait</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>157</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></td></tr></tbody></table></code></pre> 2021-07-11T00:00:00+00:00 2021-07-11T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/vscode-nix-go-tools/ <p>I use both Nix on macOS and NixOS on different machines, so most of my projects have acquired <code>shell.nix</code> files at their root so that I can have the same set of tools and dependencies available to me regardless of which machine I’m on. I also use VS Code at least some of the time, and the Go extension there requires a number of command line tools for much of its functionality. It’s quite happy to go off and install these itself, but that I’d prefer to maintain those using Nix.</p> <p>After some experimentation, here’s a skeleton <code>shell.nix</code> that provides everything VS Code’s Go extension looks for:</p> <pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-punctuation z-definition z-entity z-function z-2 z-nix">{</span> <span class="z-variable z-parameter z-function z-1 z-nix">pkgs</span> <span class="z-keyword z-operator z-nix">?</span> <span class="z-support z-function z-nix">import</span> <span class="z-string z-unquoted z-spath z-nix">&lt;nixpkgs&gt;</span> <span class="z-punctuation z-definition z-attrset z-nix">{</span> <span class="z-punctuation z-definition z-attrset z-nix">}</span> <span class="z-punctuation z-definition z-entity z-function z-nix">}</span><span class="z-punctuation z-definition z-function z-nix">:</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"><span class="z-keyword z-other z-nix">with</span> <span class="z-variable z-parameter z-name z-nix">pkgs</span>; </span><span class="z-source z-nix"> </span><span class="z-source z-nix"><span class="z-variable z-parameter z-name z-nix">mkShell</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">buildInputs</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-list z-nix">[</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">go</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">gotools</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">gopls</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">go-outline</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">gocode</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">gopkgs</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">gocode-gomod</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">godef</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">golint</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset z-nix">}</span> </span></code></pre> <p>I find it works best if I switch to the project directory, enter <code>nix-shell</code> (or <a href="https://hardselius.github.io/2020/nix-shell-and-direnv/">use direnv</a> if you prefer), then launch Code using <code>code .</code>.</p> <p>I have tried the <code>nix-env-selector</code> extension in the past, but I’ve found that not every VS Code language extension plays well with that, and previously Go was one of those which didn’t. I will give it another shot later, but for now this works for me.</p> 2021-06-26T00:00:00+00:00 2021-06-26T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/systemd/ <p>Unix machines, since the early days of the operating system, have been designed for multiple users to use concurrently. Traditionally there is a set of “unprivileged” users used by people and system services, and the root account which can generally do anything. Because of the concept that most things in Unix are represented by a file, users could be allowed to perform various operations by adding them to groups and using filesystem permissions. There were also other functions which could not be delegated in this way–notably, binding to certain IP ports. Various operating systems developed over the years have blurred these lines a little, particularly on Linux which has features like capabilities and ACLs that allow more control than the standard Unix permission model provides.</p> <aside> Linux goes much further than this. There are mandatory access control systems like SELinux or AppArmor that let you apply restrictions at the kernel level outside of the software you're running. Features like `cgroups` and namespaces combine to provide what we now call containers. Other features like `seccomp` allow software to opt-in to limits on its own ability to use various system calls. <p>BSD operating systems have similar features, notably <code>pledge</code> and <code>unveil</code> on OpenBSD.</p> </aside> <p>Much of the original permission model remains to this day. If you want to run a service that listens on a “well-known” port, those numbered less than 1024, you generally need to be root to bind to the port. There are other ways to allow this on Linux, such as applying a capability called <code>CAP_NET_BIND_SERVICE</code> to the program you want to run, but most server software that is designed to be portable among Unix systems implements a feature called privilege dropping. The service initially starts as root, binds to the ports that it requires, and then calls some functions to set its own user and group IDs to an unprivileged user. Ideally it does this before doing any significant work in order to minimise the potential for an exploit to occur while running as root.</p> <p>Even if the service isn’t going to bind to a privileged port, sometimes there are files only readable by root that are required for the service to operate. A good example of these are the private keys for TLS certificates, which often live in a root-owned directory under <code>/etc</code>. It’s common to do a similar trick to access these–start the service as root, open the files, and then drop the privileges once they’ve been read. Ideally the service will drop the privileges before parsing anything. Parser code needs to be written very carefully, especially in languages with manual memory management, and it has historically been a source of security vulnerabilities.</p> <p>All of this adds a bit of complexity to the services we write, which it would be nice to avoid. It also adds to the attack surface: privilege dropping code has been a source of vulnerabilities, notably on a couple of occasions in Bash. Avoiding writing it at all, or at least delegating it to other software with more testing than our own, would be good.</p> <p>It’s fairly common these days to write a service that runs on an unprivileged port and then run some other software in front of it as a reverse proxy–often nginx or Apache are used for this purpose. Depending on the use case these may provide some advantages, but they do require additional configuration and will use some resources on the machines operating the service.</p> <h1 id="an-example">An example<a class="zola-anchor" style="display: none;" href="#an-example" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h1> <p>I’m working on my new startup idea–Lunch as a Service, a new SaaS application to solve the complex problem of deciding what to have for lunch. While I’m working on getting the VC funding together, I’m not going to be deploying it on a huge Kubernetes setup with a service mesh and a GitOps CI/CD platform. While that would be nice one day, for now I’m going to make do with a VM.</p> <p>The core of the new service is <code>lunchd</code>. It looks like this.</p> <pre data-linenos data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><table><tbody><tr><td>1</td><td><span class="z-source z-go"><span class="z-keyword z-other z-package z-go">package</span> <span class="z-variable z-other z-go">main</span> </span></td></tr><tr><td>2</td><td><span class="z-source z-go"> </span></td></tr><tr><td>3</td><td><span class="z-source z-go"><span class="z-keyword z-other z-import z-go">import</span> <span class="z-punctuation z-section z-parens z-begin z-go">(</span> </span></td></tr><tr><td>4</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>fmt<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>5</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>math/rand<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>6</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>net/http<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>7</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>time<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>8</td><td><span class="z-source z-go"><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></td></tr><tr><td>9</td><td><span class="z-source z-go"> </span></td></tr><tr><td>10</td><td><span class="z-source z-go"><span class="z-storage z-type z-keyword z-var z-go">var</span> <span class="z-variable z-declaration z-go">lunchOptions</span> <span class="z-keyword z-operator z-assignment z-go">=</span> <span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-punctuation z-section z-brackets z-end z-go">]</span><span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></td></tr><tr><td>11</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Sandwich<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> </span></span></td></tr><tr><td>12</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Soup<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> </span></span></td></tr><tr><td>13</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Salad<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> </span></span></td></tr><tr><td>14</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Burger<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> </span></span></td></tr><tr><td>15</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Sushi<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> </span></span></td></tr><tr><td>16</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></td></tr><tr><td>17</td><td><span class="z-source z-go"> </span></td></tr><tr><td>18</td><td><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">getRandomLunch</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></td></tr><tr><td>19</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-variable z-other z-go">lunchOptions</span><span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-variable z-other z-go">rand</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Intn</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-function z-go"><span class="z-support z-function z-builtin z-go">len</span></span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">lunchOptions</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-brackets z-end z-go">]</span> </span></span></td></tr><tr><td>20</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></td></tr><tr><td>21</td><td><span class="z-source z-go"> </span></td></tr><tr><td>22</td><td><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">main</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></td></tr><tr><td>23</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">rand</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Seed</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">time</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Now</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">UnixNano</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>24</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>25</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">HandleFunc</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>/<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-storage z-type z-keyword z-function z-go">func</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-parameter z-go">w</span> <span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">ResponseWriter</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-parameter z-go">req</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">Request</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></td></tr><tr><td>26</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">fmt</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Fprintf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">w</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>&lt;h1&gt;<span class="z-constant z-other z-placeholder z-go">%s</span>&lt;/h1&gt;<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-function z-go">getRandomLunch</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>27</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>28</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>29</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">fmt</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Println</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Starting web server on port 8080<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>30</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">ListenAndServe</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>:8080<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-constant z-language z-go">nil</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>31</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></td></tr></tbody></table></code></pre> <p>I’m using a reasonably recent Linux distribution with <code>systemd</code>, so I’m going to use that to start the service when the machine boots. The basic unit file looks like this.</p> <pre data-lang="ini" class="language-ini z-code"><code class="language-ini" data-lang="ini"><span class="z-source z-genconfig"><span class="z-meta z-comment z-genconfig"><span class="z-comment z-line z-number-sign z-genconfig"># This goes in /etc/systemd/system/lunchd.service </span></span></span><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Unit] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">Description</span><span class="z-keyword z-operator z-genconfig">=</span></span>Lunch as a Service </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Service] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">ExecStart</span><span class="z-keyword z-operator z-genconfig">=</span></span><span class="z-keyword z-operator z-genconfig">/</span>usr<span class="z-keyword z-operator z-genconfig">/</span>local<span class="z-keyword z-operator z-genconfig">/</span>bin<span class="z-keyword z-operator z-genconfig">/</span>lunchd </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">User</span><span class="z-keyword z-operator z-genconfig">=</span></span>lunchd </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Install] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">WantedBy</span><span class="z-keyword z-operator z-genconfig">=</span></span>multi<span class="z-keyword z-operator z-genconfig">-</span>user<span class="z-keyword z-operator z-genconfig">.</span>target </span></code></pre> <p>I’ve added a user called <code>lunchd</code> for the the service. After running <code>sudo systemctl daemon-reload</code>, I can then do <code>sudo systemctl start lunchd</code>. The service is hard coded to run on port 8080 right now, which is not a privileged port so it’ll start quite happily and run as a standard user. The final line makes sure it starts when the system boots normally.</p> <h2 id="getting-it-ready-for-production">Getting it ready for production<a class="zola-anchor" style="display: none;" href="#getting-it-ready-for-production" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>I can boot the application and have it run as a user other than root, which is a good start. However, I want it to be able to run on port 443 and use a TLS certificate. I would also like to use some of the filesystem protection features that systemd advertises. Can I use all of these to avoid writing code to drop privileges?</p> <h3 id="filesystem-sandboxing">Filesystem sandboxing<a class="zola-anchor" style="display: none;" href="#filesystem-sandboxing" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <p>Our service doesn’t require much in the way of filesystem access. <code>systemd</code> provides ways to restrict the parts of the filesystem the service can see. By doing so, we can limit some of the opportunities for an attacker if the service is compromised.</p> <ul> <li><code>ProtectSystem</code> can be set to <code>true</code> to make <code>/usr</code> and <code>/boot</code> or <code>/efi</code> read-only for this process. If set to <code>full</code>, <code>/etc</code> is read-only too. <code>strict</code> makes the entire filesystem hierarchy read-only. This is fine for this service as it doesn’t read anything, so we’ll enable that.</li> <li><code>ProtectHome</code> can be set to <code>true</code> to make <code>/home</code>, <code>/root</code> and <code>/run/user</code> empty and inaccessible from the point of view of the service.</li> <li><code>PrivateTmp</code> makes sure that the process’s temp directories are only visible to itself, and not another process. Additionally, they’ll be emptied once the process finishes.</li> </ul> <p>The complete list of options can be found <a href="https://www.freedesktop.org/software/systemd/man/systemd.exec.html">in the systemd documentation</a>.</p> <p>Let’s add some of these features to our unit file.</p> <pre data-lang="ini" class="language-ini z-code"><code class="language-ini" data-lang="ini"><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Unit] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">Description</span><span class="z-keyword z-operator z-genconfig">=</span></span>Lunch as a Service </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Service] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">ExecStart</span><span class="z-keyword z-operator z-genconfig">=</span></span><span class="z-keyword z-operator z-genconfig">/</span>usr<span class="z-keyword z-operator z-genconfig">/</span>local<span class="z-keyword z-operator z-genconfig">/</span>bin<span class="z-keyword z-operator z-genconfig">/</span>lunchd </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">ProtectSystem</span><span class="z-keyword z-operator z-genconfig">=</span></span>strict </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">ProtectHome</span><span class="z-keyword z-operator z-genconfig">=</span></span><span class="z-constant z-language z-genconfig">true</span> </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">PrivateUsers</span><span class="z-keyword z-operator z-genconfig">=</span></span><span class="z-constant z-language z-genconfig">true</span> </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">PrivateTmp</span><span class="z-keyword z-operator z-genconfig">=</span></span><span class="z-constant z-language z-genconfig">true</span> </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Install] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">WantedBy</span><span class="z-keyword z-operator z-genconfig">=</span></span>multi<span class="z-keyword z-operator z-genconfig">-</span>user<span class="z-keyword z-operator z-genconfig">.</span>target </span></code></pre> <p>There are far more options available here than I really want to get into describing. The documentation linked above is a good reference. To get some ideas for a service of your own, you can try a command similar to</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">sudo</span></span><span class="z-meta z-function-call z-arguments z-shell"> systemd-analyze security lunchd.service<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>no-pager</span></span> </span></code></pre> <p>This gives you an overview of what security features systemd has, and which are enabled. It is probably not a matter of just enabling everything it mentions, as then your service may not be able to do very much–instead, you should consider these as suggestions and see if they will work for you.</p> <h3 id="dynamic-users">Dynamic users<a class="zola-anchor" style="display: none;" href="#dynamic-users" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <p>Traditionally, many services have expected to run as a defined user and group, usually specified in their configuration file. Some will start and run as that user if they don’t need to do anything privileged, but others will start as root and switch. Many common internet services such as web and mail servers operate in this way.</p> <p>With systemd we can have a brand new user and group allocated for us when the service starts. These users are prevented from changing any state in the system except in the directories detailed above. Once the service exits, the user is removed–it never actually exists in <code>/etc/passwd</code>. This makes it harder to create a persistent route in to a system after an exploit.</p> <pre data-lang="ini" class="language-ini z-code"><code class="language-ini" data-lang="ini"><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Unit] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">Description</span><span class="z-keyword z-operator z-genconfig">=</span></span>Lunch as a Service </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Service] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">ExecStart</span><span class="z-keyword z-operator z-genconfig">=</span></span><span class="z-keyword z-operator z-genconfig">/</span>usr<span class="z-keyword z-operator z-genconfig">/</span>local<span class="z-keyword z-operator z-genconfig">/</span>bin<span class="z-keyword z-operator z-genconfig">/</span>lunchd </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">ProtectSystem</span><span class="z-keyword z-operator z-genconfig">=</span></span>strict </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">ProtectHome</span><span class="z-keyword z-operator z-genconfig">=</span></span><span class="z-constant z-language z-genconfig">true</span> </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">PrivateUsers</span><span class="z-keyword z-operator z-genconfig">=</span></span><span class="z-constant z-language z-genconfig">true</span> </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">PrivateTmp</span><span class="z-keyword z-operator z-genconfig">=</span></span><span class="z-constant z-language z-genconfig">true</span> </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">DynamicUser</span><span class="z-keyword z-operator z-genconfig">=</span></span><span class="z-constant z-language z-genconfig">yes</span> </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Install] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">WantedBy</span><span class="z-keyword z-operator z-genconfig">=</span></span>multi<span class="z-keyword z-operator z-genconfig">-</span>user<span class="z-keyword z-operator z-genconfig">.</span>target </span></code></pre> <p>For this I’ve removed the system user I added earlier. After another <code>daemon-reload</code> and restarting the service, we can see that the service is still running as a user called <code>lunchd</code>, but it has a very high user ID:</p> <pre class="z-code"><code><span class="z-text z-plain">mgdm@lunchbox:~$ ps aux | grep lunchd </span><span class="z-text z-plain">lunchd 33146 0.2 0.4 1002788 4312 ? Ssl 20:28 0:00 /usr/local/bin/lunchd </span><span class="z-text z-plain">mgdm@lunchbox:~$ id lunchd </span><span class="z-text z-plain">uid=62840(lunchd) gid=62840(lunchd) groups=62840(lunchd) </span></code></pre> <p>None of these changes have, so far, required much in the way of modification to our source code. At the most, we just need to make sure our service can accept some configuration for the directories it writes to.</p> <h3 id="using-port-80">Using port 80<a class="zola-anchor" style="display: none;" href="#using-port-80" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <p>Although we’ve added some security features, our service is still running on port 8080. That’s fine for development purposes, but we’ve run into the first of the problems I described at the start. We want to bind to a privileged port, but the service is now definitely running as an unprivileged user.</p> <p><code>systemd</code> has a feature called socket activation, which allows it to bind to a port and then hand a file descriptor for that port to a process it launches. It can either launch the process once per connection, in a similar fashion to the old <a href="https://en.wikipedia.org/wiki/Inetd">inetd daemon</a>, or it can bind to the port once and hand it to a persistent process. This latter option is what we’ll do. It does require some modification to the code, but none of it will ever run as root which further reduces the attack surface.</p> <p>It’s certainly possible to hand-write this code but in this case I’m going to use a package from CoreOS. If you’d like to see a minimal hand-written version, check out <a href="https://github.com/systemd/portable-walkthrough-go/blob/master/main.go#L15-L31">this one from Lennart Poettering</a>.</p> <p>Here’s the modified service.</p> <pre data-linenos data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><table><tbody><tr><td>1</td><td><span class="z-source z-go"><span class="z-keyword z-other z-package z-go">package</span> <span class="z-variable z-other z-go">main</span> </span></td></tr><tr><td>2</td><td><span class="z-source z-go"> </span></td></tr><tr><td>3</td><td><span class="z-source z-go"><span class="z-keyword z-other z-import z-go">import</span> <span class="z-punctuation z-section z-parens z-begin z-go">(</span> </span></td></tr><tr><td>4</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>fmt<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>5</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>log<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>6</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>math/rand<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>7</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>net<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>8</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>net/http<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>9</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>time<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>10</td><td><span class="z-source z-go"> </span></td></tr><tr><td>11</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>github.com/coreos/go-systemd/activation<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>12</td><td><span class="z-source z-go"><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></td></tr><tr><td>13</td><td><span class="z-source z-go"> </span></td></tr><tr><td>14</td><td><span class="z-source z-go"><span class="z-storage z-type z-keyword z-var z-go">var</span> <span class="z-variable z-declaration z-go">lunchOptions</span> <span class="z-keyword z-operator z-assignment z-go">=</span> <span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-punctuation z-section z-brackets z-end z-go">]</span><span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></td></tr><tr><td>15</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Sandwich<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> </span></span></td></tr><tr><td>16</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Soup<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> </span></span></td></tr><tr><td>17</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Salad<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> </span></span></td></tr><tr><td>18</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Burger<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> </span></span></td></tr><tr><td>19</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Sushi<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> </span></span></td></tr><tr><td>20</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></td></tr><tr><td>21</td><td><span class="z-source z-go"> </span></td></tr><tr><td>22</td><td><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">getRandomLunch</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></td></tr><tr><td>23</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-variable z-other z-go">lunchOptions</span><span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-variable z-other z-go">rand</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Intn</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-function z-go"><span class="z-support z-function z-builtin z-go">len</span></span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">lunchOptions</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-brackets z-end z-go">]</span> </span></span></td></tr><tr><td>24</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></td></tr><tr><td>25</td><td><span class="z-source z-go"> </span></td></tr><tr><td>26</td><td><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">getListener</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">net</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">Listener</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">error</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></td></tr><tr><td>27</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">listeners</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-declaration z-go">err</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">activation</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Listeners</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>28</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>29</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">if</span> <span class="z-variable z-other z-go">err</span> <span class="z-keyword z-operator z-go">!=</span> <span class="z-constant z-language z-go">nil</span> <span class="z-keyword z-operator z-go">||</span> <span class="z-variable z-function z-go"><span class="z-support z-function z-builtin z-go">len</span></span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">listeners</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-keyword z-operator z-go">!=</span> <span class="z-constant z-numeric z-integer z-decimal z-go">1</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></td></tr><tr><td>30</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">log</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Printf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Excpected one listener, got <span class="z-constant z-other z-placeholder z-go">%d</span>: <span class="z-constant z-other z-placeholder z-go">%s</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-function z-go"><span class="z-support z-function z-builtin z-go">len</span></span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">listeners</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">err</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>31</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> </span></span></span></td></tr><tr><td>32</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">listener</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-declaration z-go">err</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">net</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Listen</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>tcp<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>:8080<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>33</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-variable z-other z-go">listener</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">err</span> </span></span></span></td></tr><tr><td>34</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span></td></tr><tr><td>35</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>36</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-variable z-other z-go">listeners</span><span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-constant z-numeric z-integer z-decimal z-go">0</span><span class="z-punctuation z-section z-brackets z-end z-go">]</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">err</span> </span></span></td></tr><tr><td>37</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></td></tr><tr><td>38</td><td><span class="z-source z-go"> </span></td></tr><tr><td>39</td><td><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">main</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></td></tr><tr><td>40</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">rand</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Seed</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">time</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Now</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">UnixNano</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>41</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>42</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">listener</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-declaration z-go">err</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-function z-go">getListener</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>43</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>44</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">if</span> <span class="z-variable z-other z-go">err</span> <span class="z-keyword z-operator z-go">!=</span> <span class="z-constant z-language z-go">nil</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></td></tr><tr><td>45</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">log</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Panicf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Could not set up listener: <span class="z-constant z-other z-placeholder z-go">%s</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">err</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>46</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span></td></tr><tr><td>47</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>48</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">HandleFunc</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>/<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-storage z-type z-keyword z-function z-go">func</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-parameter z-go">w</span> <span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">ResponseWriter</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-parameter z-go">req</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">Request</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></td></tr><tr><td>49</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">fmt</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Fprintf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">w</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>&lt;h1&gt;<span class="z-constant z-other z-placeholder z-go">%s</span>&lt;/h1&gt;<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-function z-go">getRandomLunch</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>50</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>51</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>52</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">log</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Printf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Starting web server on port <span class="z-constant z-other z-placeholder z-go">%s</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">listener</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Addr</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">String</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>53</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Serve</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">listener</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-constant z-language z-go">nil</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>54</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></td></tr></tbody></table></code></pre> <p>Once we’ve built the new version, we can test the freshly-built version before deploying it:</p> <pre class="z-code"><code><span class="z-text z-plain">mgdm@lunchbox:~$ systemd-socket-activate -l 8080 ./main </span><span class="z-text z-plain">Listening on [::]:8080 as 3. </span><span class="z-text z-plain">Communication attempt on fd 3. </span><span class="z-text z-plain">Execing ./main (./main) </span><span class="z-text z-plain">Starting web server on port 8080 </span></code></pre> <p>This runs fine. To make it work this way permanently, we need to create a socket file for systemd in addition to the service file. It describes the port you want the systemd to bind to and pass to the service. This can go in <code>/etc/systemd/system/lunchd.socket</code>.</p> <pre data-lang="ini" class="language-ini z-code"><code class="language-ini" data-lang="ini"><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Socket] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">ListenStream</span> <span class="z-keyword z-operator z-genconfig">=</span></span> <span class="z-constant z-numeric z-genconfig">80</span> </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">BindIPv6Only</span> <span class="z-keyword z-operator z-genconfig">=</span></span> both </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">Accept</span><span class="z-keyword z-operator z-genconfig">=</span></span><span class="z-constant z-language z-genconfig">no</span> </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Install] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">WantedBy</span> <span class="z-keyword z-operator z-genconfig">=</span></span> sockets<span class="z-keyword z-operator z-genconfig">.</span>target </span></code></pre> <p>After another <code>systemctl daemon-reload</code>, you can then type <code>systemctl start lunchd.socket</code> which will make it listen to the port.</p> <p>The <code>Accept</code> option provides a choice between systemd starting the service once and handing it the listening sockets (<code>Accept=no</code>), or starting a new instance of the service on each request (<code>Accept=yes</code>). This latter mode is quite like how <code>inetd</code> operates.</p> <p>Another advantage of having systemd listen on these ports is that if the service crashes, requests can still come in on port 80 and systemd will take care of starting a new one for you. Theoretically at least, this means you shouldn’t drop those requests.</p> <h3 id="adding-https">Adding HTTPS<a class="zola-anchor" style="display: none;" href="#adding-https" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <p>For this we’ll need a certificate, which most of the time we can get from LetsEncrypt. However, most LetsEncrypt clients will place the private key somewhere in <code>/etc</code> in a directory only accessible by root. This is fine, as having the private key owned by another user makes it harder to steal in the event our service is compromised. (We’ll just have to hope we don’t have <a href="https://en.wikipedia.org/wiki/Cloudbleed">some kind of memory leak attack</a>). We want to be able to use the certificate, but we don’t want to use privilege dropping here.</p> <p>Normally for this, we’d use code something like the following, <a href="https://golang.org/src/crypto/tls/example_test.go#L114">taken from the Go documentation</a>:</p> <pre data-linenos data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><table><tbody><tr><td>1</td><td><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">ExampleLoadX509KeyPair</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></td></tr><tr><td>2</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">cert</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-declaration z-go">err</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">tls</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">LoadX509KeyPair</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>example-cert.pem<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>example-key.pem<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>3</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>4</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">if</span> <span class="z-variable z-other z-go">err</span> <span class="z-keyword z-operator z-go">!=</span> <span class="z-constant z-language z-go">nil</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></td></tr><tr><td>5</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">log</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Fatal</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">err</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>6</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span></td></tr><tr><td>7</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>8</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">cfg</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-keyword z-operator z-go">&amp;</span><span class="z-variable z-other z-go">tls</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">Config</span><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span><span class="z-variable z-other z-go">Certificates</span><span class="z-punctuation z-separator z-go">:</span> <span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-punctuation z-section z-brackets z-end z-go">]</span><span class="z-variable z-other z-go">tls</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">Certificate</span><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span><span class="z-variable z-other z-go">cert</span><span class="z-punctuation z-section z-braces z-end z-go">}</span></span><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span></td></tr><tr><td>9</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">listener</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-declaration z-go">err</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">tls</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Listen</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>tcp<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>:2000<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">cfg</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>10</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>11</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">if</span> <span class="z-variable z-other z-go">err</span> <span class="z-keyword z-operator z-go">!=</span> <span class="z-constant z-language z-go">nil</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></td></tr><tr><td>12</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">log</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Fatal</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">err</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>13</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span></td></tr><tr><td>14</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>15</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-language z-blank z-go">_</span> <span class="z-keyword z-operator z-assignment z-go">=</span> <span class="z-variable z-other z-go">listener</span> </span></span></td></tr><tr><td>16</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>17</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></td></tr></tbody></table></code></pre> <p>We can pretty much use this, although the paths are hard-coded so this assumes the process is going to be able to open the files from whatever user it’s running as. I’ve used <code>certbot</code> to create a LetsEncrypt certificate, which is stored in <code>/etc/letsencrypt/live/lunch.mgdm.net</code> and is only readable by root.</p> <p>We can work around this using systemd. It can open the files and present them in a modified filesystem view to the service. We’ll need to make some more changes to our code, but we were going to have to do this anyway to add the TLS. The only thing to remember is to make the paths to those files configurable. In this case, I’ve added flags called <code>key</code> and <code>certificate</code> which provide the paths to the TLS private key and the certificate with its chain.</p> <p>The key thing here is the <code>LoadCredential</code> setting. It takes both a key and a path, separated by <code>:</code>. systemd will load the contents of the file at the specified path, and expose it in a directory as a file named after the key. This directory is given to the process as an environment variable called <code>CREDENTIALS_DIRECTORY</code>, which can also be used in the <code>ExecStart</code> line of the unit file. You can see this in action in the modified unit file below. We’ve also made a change to the socket file in order to make it listen on port 443 instead.</p> <h4 id="lunchd-service">lunchd.service<a class="zola-anchor" style="display: none;" href="#lunchd-service" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h4> <pre data-lang="ini" class="language-ini z-code"><code class="language-ini" data-lang="ini"><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Unit] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">Description</span><span class="z-keyword z-operator z-genconfig">=</span></span>Lunch as a Service </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Service] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">ExecStart</span><span class="z-keyword z-operator z-genconfig">=</span></span><span class="z-keyword z-operator z-genconfig">/</span>usr<span class="z-keyword z-operator z-genconfig">/</span>local<span class="z-keyword z-operator z-genconfig">/</span>bin<span class="z-keyword z-operator z-genconfig">/</span>lunchd <span class="z-keyword z-operator z-genconfig">-</span>key<span class="z-keyword z-operator z-genconfig">=</span><span class="z-storage z-source z-genconfig">${CREDENTIALS_DIRECTORY}</span><span class="z-keyword z-operator z-genconfig">/</span>key<span class="z-keyword z-operator z-genconfig">.</span>pem <span class="z-keyword z-operator z-genconfig">-</span>certificate<span class="z-keyword z-operator z-genconfig">=</span><span class="z-storage z-source z-genconfig">${CREDENTIALS_DIRECTORY}</span><span class="z-keyword z-operator z-genconfig">/</span>chain<span class="z-keyword z-operator z-genconfig">.</span>pem </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">LoadCredential</span><span class="z-keyword z-operator z-genconfig">=</span></span>key<span class="z-keyword z-operator z-genconfig">.</span>pem<span class="z-keyword z-operator z-genconfig">:</span><span class="z-keyword z-operator z-genconfig">/</span>etc<span class="z-keyword z-operator z-genconfig">/</span>letsencrypt<span class="z-keyword z-operator z-genconfig">/</span>live<span class="z-keyword z-operator z-genconfig">/</span>lunch<span class="z-keyword z-operator z-genconfig">.</span><span class="z-support z-type z-genconfig">mgdm.net/privkey.pem</span> </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">LoadCredential</span><span class="z-keyword z-operator z-genconfig">=</span></span>chain<span class="z-keyword z-operator z-genconfig">.</span>pem<span class="z-keyword z-operator z-genconfig">:</span><span class="z-keyword z-operator z-genconfig">/</span>etc<span class="z-keyword z-operator z-genconfig">/</span>letsencrypt<span class="z-keyword z-operator z-genconfig">/</span>live<span class="z-keyword z-operator z-genconfig">/</span>lunch<span class="z-keyword z-operator z-genconfig">.</span><span class="z-support z-type z-genconfig">mgdm.net/fullchain.pem</span> </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">ProtectSystem</span><span class="z-keyword z-operator z-genconfig">=</span></span>strict </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">ProtectHome</span><span class="z-keyword z-operator z-genconfig">=</span></span><span class="z-constant z-language z-genconfig">true</span> </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">PrivateUsers</span><span class="z-keyword z-operator z-genconfig">=</span></span><span class="z-constant z-language z-genconfig">true</span> </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">PrivateTmp</span><span class="z-keyword z-operator z-genconfig">=</span></span><span class="z-constant z-language z-genconfig">true</span> </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">DynamicUser</span><span class="z-keyword z-operator z-genconfig">=</span></span><span class="z-constant z-language z-genconfig">yes</span> </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Install] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">WantedBy</span><span class="z-keyword z-operator z-genconfig">=</span></span>multi<span class="z-keyword z-operator z-genconfig">-</span>user<span class="z-keyword z-operator z-genconfig">.</span>target </span></code></pre> <h4 id="lunchd-socket">lunchd.socket<a class="zola-anchor" style="display: none;" href="#lunchd-socket" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h4> <pre data-lang="ini" class="language-ini z-code"><code class="language-ini" data-lang="ini"><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Socket] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">ListenStream</span> <span class="z-keyword z-operator z-genconfig">=</span></span> <span class="z-constant z-numeric z-genconfig">443</span> </span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">BindIPv6Only</span> <span class="z-keyword z-operator z-genconfig">=</span></span> both </span><span class="z-source z-genconfig"> </span><span class="z-source z-genconfig"><span class="z-storage z-type z-genconfig">[Install] </span></span><span class="z-source z-genconfig"><span class="z-meta z-param z-genconfig"><span class="z-variable z-parameter z-genconfig">WantedBy</span> <span class="z-keyword z-operator z-genconfig">=</span></span> sockets<span class="z-keyword z-operator z-genconfig">.</span>target </span></code></pre> <h4 id="main-go">main.go<a class="zola-anchor" style="display: none;" href="#main-go" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h4> <pre data-linenos data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><table><tbody><tr><td>1</td><td><span class="z-source z-go"><span class="z-keyword z-other z-package z-go">package</span> <span class="z-variable z-other z-go">main</span> </span></td></tr><tr><td>2</td><td><span class="z-source z-go"> </span></td></tr><tr><td>3</td><td><span class="z-source z-go"><span class="z-keyword z-other z-import z-go">import</span> <span class="z-punctuation z-section z-parens z-begin z-go">(</span> </span></td></tr><tr><td>4</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>crypto/tls<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>5</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>flag<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>6</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>fmt<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>7</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>log<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>8</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>math/rand<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>9</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>net<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>10</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>net/http<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>11</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>time<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>12</td><td><span class="z-source z-go"> </span></td></tr><tr><td>13</td><td><span class="z-source z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>github.com/coreos/go-systemd/activation<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span></td></tr><tr><td>14</td><td><span class="z-source z-go"><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></td></tr><tr><td>15</td><td><span class="z-source z-go"> </span></td></tr><tr><td>16</td><td><span class="z-source z-go"><span class="z-storage z-type z-keyword z-var z-go">var</span> <span class="z-variable z-declaration z-go">lunchOptions</span> <span class="z-keyword z-operator z-assignment z-go">=</span> <span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-punctuation z-section z-brackets z-end z-go">]</span><span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></td></tr><tr><td>17</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Sandwich<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> </span></span></td></tr><tr><td>18</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Soup<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> </span></span></td></tr><tr><td>19</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Salad<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> </span></span></td></tr><tr><td>20</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Burger<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> </span></span></td></tr><tr><td>21</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Sushi<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> </span></span></td></tr><tr><td>22</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></td></tr><tr><td>23</td><td><span class="z-source z-go"> </span></td></tr><tr><td>24</td><td><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">getRandomLunch</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></td></tr><tr><td>25</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-variable z-other z-go">lunchOptions</span><span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-variable z-other z-go">rand</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Intn</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-function z-go"><span class="z-support z-function z-builtin z-go">len</span></span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">lunchOptions</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-brackets z-end z-go">]</span> </span></span></td></tr><tr><td>26</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></td></tr><tr><td>27</td><td><span class="z-source z-go"> </span></td></tr><tr><td>28</td><td><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">getListener</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">net</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">Listener</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">error</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></td></tr><tr><td>29</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">listeners</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-declaration z-go">err</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">activation</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Listeners</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>30</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>31</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">if</span> <span class="z-variable z-other z-go">err</span> <span class="z-keyword z-operator z-go">!=</span> <span class="z-constant z-language z-go">nil</span> <span class="z-keyword z-operator z-go">||</span> <span class="z-variable z-function z-go"><span class="z-support z-function z-builtin z-go">len</span></span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">listeners</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-keyword z-operator z-go">!=</span> <span class="z-constant z-numeric z-integer z-decimal z-go">1</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></td></tr><tr><td>32</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">log</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Printf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Excpected one listener, got <span class="z-constant z-other z-placeholder z-go">%d</span>: <span class="z-constant z-other z-placeholder z-go">%s</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-function z-go"><span class="z-support z-function z-builtin z-go">len</span></span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">listeners</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">err</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>33</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> </span></span></span></td></tr><tr><td>34</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">listener</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-declaration z-go">err</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">net</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Listen</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>tcp<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>:8080<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>35</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-variable z-other z-go">listener</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">err</span> </span></span></span></td></tr><tr><td>36</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span></td></tr><tr><td>37</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>38</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-variable z-other z-go">listeners</span><span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-constant z-numeric z-integer z-decimal z-go">0</span><span class="z-punctuation z-section z-brackets z-end z-go">]</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">err</span> </span></span></td></tr><tr><td>39</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></td></tr><tr><td>40</td><td><span class="z-source z-go"> </span></td></tr><tr><td>41</td><td><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">getCertificates</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">error</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></td></tr><tr><td>42</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">keyPath</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">flag</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">String</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>key<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>The path to the private key<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>43</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">certPath</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">flag</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">String</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>certificate<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>The path to the certificate<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>44</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">flag</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Parse</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>45</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>46</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">if</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">keyPath</span> <span class="z-keyword z-operator z-go">==</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> <span class="z-keyword z-operator z-go">||</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">certPath</span> <span class="z-keyword z-operator z-go">==</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></td></tr><tr><td>47</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">fmt</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Errorf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Either or both of -key or -cert not set<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>48</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span></td></tr><tr><td>49</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>50</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">keyPath</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">certPath</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-constant z-language z-go">nil</span> </span></span></td></tr><tr><td>51</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></td></tr><tr><td>52</td><td><span class="z-source z-go"> </span></td></tr><tr><td>53</td><td><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">tlsSetup</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-parameter z-go">keyPath</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-parameter z-go">certPath</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-parameter z-go">listener</span> <span class="z-variable z-other z-go">net</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">Listener</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">net</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">Listener</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">error</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></td></tr><tr><td>54</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">config</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-keyword z-operator z-go">&amp;</span><span class="z-variable z-other z-go">tls</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">Config</span><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></td></tr><tr><td>55</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">Certificates</span><span class="z-punctuation z-separator z-go">:</span> <span class="z-variable z-function z-go"><span class="z-support z-function z-builtin z-go">make</span></span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-punctuation z-section z-brackets z-end z-go">]</span><span class="z-variable z-other z-go">tls</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">Certificate</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-constant z-numeric z-integer z-decimal z-go">1</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-separator z-go">,</span> </span></span></span></td></tr><tr><td>56</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">NextProtos</span><span class="z-punctuation z-separator z-go">:</span> <span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-punctuation z-section z-brackets z-end z-go">]</span><span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">string</span></span><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>h2<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>http/1.1<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-section z-braces z-end z-go">}</span></span><span class="z-punctuation z-separator z-go">,</span> </span></span></span></td></tr><tr><td>57</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">PreferServerCipherSuites</span><span class="z-punctuation z-separator z-go">:</span> <span class="z-constant z-language z-go">true</span><span class="z-punctuation z-separator z-go">,</span> </span></span></span></td></tr><tr><td>58</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span></td></tr><tr><td>59</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>60</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-storage z-type z-keyword z-var z-go">var</span> <span class="z-variable z-declaration z-go">err</span> <span class="z-storage z-type z-go"><span class="z-support z-type z-builtin z-go">error</span></span> </span></span></td></tr><tr><td>61</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>62</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">log</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Printf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Loading certs from key: <span class="z-constant z-other z-placeholder z-go">%s</span> and cert: <span class="z-constant z-other z-placeholder z-go">%s</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">keyPath</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">certPath</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>63</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>64</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">config</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">Certificates</span><span class="z-punctuation z-section z-brackets z-begin z-go">[</span><span class="z-constant z-numeric z-integer z-decimal z-go">0</span><span class="z-punctuation z-section z-brackets z-end z-go">]</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">err</span> <span class="z-keyword z-operator z-assignment z-go">=</span> <span class="z-variable z-other z-go">tls</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">LoadX509KeyPair</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span> </span></span></td></tr><tr><td>65</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">certPath</span><span class="z-punctuation z-separator z-go">,</span> </span></span></td></tr><tr><td>66</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">keyPath</span><span class="z-punctuation z-separator z-go">,</span> </span></span></td></tr><tr><td>67</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>68</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>69</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">if</span> <span class="z-variable z-other z-go">err</span> <span class="z-keyword z-operator z-go">!=</span> <span class="z-constant z-language z-go">nil</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></td></tr><tr><td>70</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">log</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Printf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Failed to configure TLS: <span class="z-constant z-other z-placeholder z-go">%s</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">err</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>71</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-constant z-language z-go">nil</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">err</span> </span></span></span></td></tr><tr><td>72</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span></td></tr><tr><td>73</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>74</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">return</span> <span class="z-variable z-other z-go">tls</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">NewListener</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">listener</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">config</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-constant z-language z-go">nil</span> </span></span></td></tr><tr><td>75</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></td></tr><tr><td>76</td><td><span class="z-source z-go"> </span></td></tr><tr><td>77</td><td><span class="z-source z-go"><span class="z-storage z-type z-keyword z-function z-go">func</span> <span class="z-entity z-name z-function z-go">main</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></td></tr><tr><td>78</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">rand</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Seed</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">time</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Now</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">UnixNano</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>79</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>80</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">listener</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-declaration z-go">err</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-function z-go">getListener</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>81</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>82</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">if</span> <span class="z-variable z-other z-go">err</span> <span class="z-keyword z-operator z-go">!=</span> <span class="z-constant z-language z-go">nil</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></td></tr><tr><td>83</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">log</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Fatalf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Could not set up listener: <span class="z-constant z-other z-placeholder z-go">%s</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">err</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>84</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span></td></tr><tr><td>85</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>86</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">keyPath</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-declaration z-go">certPath</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-declaration z-go">err</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-function z-go">getCertificates</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>87</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>88</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">if</span> <span class="z-variable z-other z-go">err</span> <span class="z-keyword z-operator z-go">!=</span> <span class="z-constant z-language z-go">nil</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></td></tr><tr><td>89</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">log</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Fatalf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Could not load certificates: <span class="z-constant z-other z-placeholder z-go">%s</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">err</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>90</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span></td></tr><tr><td>91</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>92</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-declaration z-go">tlsListener</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-declaration z-go">err</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-function z-go">tlsSetup</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">keyPath</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">certPath</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">listener</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>93</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>94</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">HandleFunc</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>/<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-storage z-type z-keyword z-function z-go">func</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-parameter z-go">w</span> <span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">ResponseWriter</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-parameter z-go">req</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">Request</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span></td></tr><tr><td>95</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">fmt</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Fprintf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">w</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>&lt;h1&gt;<span class="z-constant z-other z-placeholder z-go">%s</span>&lt;/h1&gt;<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-function z-go">getRandomLunch</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span></td></tr><tr><td>96</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>97</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> </span></span></td></tr><tr><td>98</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">fmt</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Printf</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Starting web server on port <span class="z-constant z-other z-placeholder z-go">%s</span><span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">tlsListener</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Addr</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">String</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>99</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-variable z-other z-go">http</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">Serve</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">tlsListener</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-constant z-language z-go">nil</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></td></tr><tr><td>100</td><td><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></td></tr></tbody></table></code></pre> <p>Once we make these changes, compile it, and deploy it, we can see the following in the journal by running <code>sudo journalctl -xe</code>:</p> <pre class="z-code"><code><span class="z-text z-plain">Jun 26 12:05:35 lunchbox lunchd[943]: 2021/06/26 12:05:35 Loading certs from key: /run/credentials/lunchd.service/key.pem and cert: /run/credentials/lunchd.service/chain.pem </span></code></pre> <p>This shows that the <code>CREDENTIALS_DIRECTORY</code> has been set to <code>/run/credentials/lunchd.service</code> in this case, and the private key and chain are exposed there. To make this work, no changes have been made to the code beyond that required to enable TLS, which would be common to any platform I ran this code on. The only thing that I’ve done is made sure that I can specify the paths to the key and certificate on the command line. The systemd-specific <code>CREDENTIALS_DIRECTORY</code> is only referenced in the unit file.</p> <p>I could modify the service further so that systemd sets up listeners on both ports 80 and 443, so the service deals with both HTTP and HTTPS itself, but I think I’ll leave that as an exercise for the reader.</p> <h1 id="references">References<a class="zola-anchor" style="display: none;" href="#references" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h1> <p>I found the following articles helpful.</p> <ol> <li><a href="https://www.freedesktop.org/software/systemd/man/systemd.exec.html">The documentation for systemd.exec</a></li> <li><a href="https://gist.github.com/ageis/f5595e59b1cddb1513d1b425a323db04">Options for hardening systemd service units</a></li> <li><a href="https://vincent.bernat.ch/en/blog/2018-systemd-golang-socket-activation">Integration of a Go service with systemd: socket activation by Vincent Bernat</a></li> <li><a href="https://wiki.debian.org/ServiceSandboxing">The Debian docs on Service Sandboxing</a></li> <li><a href="http://0pointer.net/blog/walkthrough-for-portable-services-in-go.html">Walkthrough for Portable Services in Go</a></li> </ol> <p><a href="https://github.com/mgdm/lunchd">The code is on GitHub</a> in case it’s useful to anyone.</p> 2021-03-23T00:00:00+00:00 2021-03-23T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/how-to-accidentally-mysql/ <h2 id="a-long-time-ago-in-a-company-far-far-away">A long time ago, in a company far, far away<a class="zola-anchor" style="display: none;" href="#a-long-time-ago-in-a-company-far-far-away" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>Our site ran on dedicated hardware, rented from a managed hosting provider who we never really got around to doing a lot of managing. As such, we all had root on all the machines, and so were free to do what we wanted. We also didn’t really know a lot about configuration management at that point, so changes were largely ad-hoc and unrepeatable.</p> <p>At some point we found that the DB server was a bit slow for some queries, and that it had a lot more RAM than it appeared to be using. While nowadays I would probably assume that I could bump the <code>innodb_buffer_pool_size</code> up by some gigabytes, at that point what appeared to be a really great idea to past-Michael was to stick the MySQL temporary tables into a RAM disk. This made things <em>fly</em> and seemed like a totally great idea with no possible future negative consequences.</p> <h2 id="in-the-mean-time">In the mean time<a class="zola-anchor" style="display: none;" href="#in-the-mean-time" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>We were using the site to show some media that required us to implement a geographic filter for the viewers. This was implemented by either using the users’ device’s geolocation to supply a latitude and longitude, or by having the user enter a postal code. Our site could look up whether it was within a certain bounding box, and if not it would redirect to another provider.</p> <h2 id="eventually">Eventually<a class="zola-anchor" style="display: none;" href="#eventually" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>We came to move to AWS. We had everything built out with configuration management, deployed our site to EC2, used <code>memcached</code> provided by Elasticache, and finally put our MySQL database on RDS.</p> <p>We had a couple of runs at switching it over from the dedicated host. The second time it finally stuck, after bumping the database size up a bit. That evening we had one of these media events that needed the geographic filter. I had hoped that we wouldn’t need to use it, but it turned out we had to. I hung around in the office to watch what ensued.</p> <p>About 10 minutes prior to the event, the site fell all the way over. Something was thrashing the database in a way that hadn’t been happening before.</p> <h2 id="skip-to-the-end">Skip to the end<a class="zola-anchor" style="display: none;" href="#skip-to-the-end" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>It turned out our geographic filter was implemented like this:</p> <ol> <li>User provides their latitude and longitude via the geolocation API on a device like an iPad</li> <li>The site does a query against a table containing a snapshot of the Ordnance Survey Codepoint database. This is a list of every known UK postal code and the coordinates of its centre point, which consisted of something like 350 000 rows.</li> <li>This query implements the <a href="https://en.wikipedia.org/wiki/Haversine_formula">Haversine formula</a>, which uses spherical trigonometry to calculate a pretty accurate <a href="https://en.wikipedia.org/wiki/Great-circle_distance">great circle distance</a> between the user-supplied point, and each of the 350 000 rows in the Codepoint database.</li> <li>The query finished up with <code>ORDER BY distance LIMIT 1</code>, which threw away 349 999 of those rows and returned the distance to the nearest. It could then see if this row was within the area for which we held the rights.</li> </ol> <p>The query looked like the below, and I suspect it was taken from <a href="https://stackoverflow.com/questions/13250645/sql-syntax-error-haversine-formula">this StackOverflow answer</a> which is from around the right time period.</p> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-keyword z-other z-DML z-sql">SELECT</span> </span><span class="z-source z-sql">postcode, </span><span class="z-source z-sql">( <span class="z-constant z-numeric z-sql">6371</span> <span class="z-variable z-language z-star z-sql">*</span> acos( cos( radians(<span class="z-constant z-numeric z-sql">18</span>.<span class="z-constant z-numeric z-sql">204540500000</span>) ) </span><span class="z-source z-sql"> <span class="z-variable z-language z-star z-sql">*</span> cos( radians( latitude ) ) </span><span class="z-source z-sql"> <span class="z-variable z-language z-star z-sql">*</span> cos( radians( longitude ) </span><span class="z-source z-sql"> <span class="z-keyword z-operator z-math z-sql">-</span> radians(<span class="z-keyword z-operator z-math z-sql">-</span><span class="z-constant z-numeric z-sql">66</span>.<span class="z-constant z-numeric z-sql">450958500000</span>) ) </span><span class="z-source z-sql"> <span class="z-keyword z-operator z-math z-sql">+</span> sin( radians(<span class="z-constant z-numeric z-sql">18</span>.<span class="z-constant z-numeric z-sql">204540500000</span> ) ) </span><span class="z-source z-sql"> <span class="z-variable z-language z-star z-sql">*</span> sin( radians( latitude ) ) </span><span class="z-source z-sql"> ) </span><span class="z-source z-sql"> ) <span class="z-keyword z-operator z-assignment z-alias z-sql">AS</span> distance </span><span class="z-source z-sql"> <span class="z-keyword z-other z-DML z-sql">FROM</span> codepoint </span><span class="z-source z-sql"> <span class="z-keyword z-other z-DML z-sql">ORDER BY</span> distance </span><span class="z-source z-sql"> <span class="z-keyword z-other z-DML z-sql">LIMIT</span> <span class="z-constant z-numeric z-sql">1</span> </span></code></pre> <p>On our dedicated hardware, with the oh-so-clever RAM disk trick, this query took 0.2 seconds. On RDS, where you are prevented from doing bullshit like that, it took 14 seconds. As it was not even slightly cacheable the traffic brought down the DB and hence the site in pretty short order.</p> <h2 id="the-fix">The fix<a class="zola-anchor" style="display: none;" href="#the-fix" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>We moved it to Elasticsearch, where the same query executed in about 0.4ms.</p> <h2 id="postscript">Postscript<a class="zola-anchor" style="display: none;" href="#postscript" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>This was probably only the third-worst failure we experienced while I was there.</p> 2020-12-29T00:00:00+00:00 2020-12-29T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/evading-broken-dns/ <p>Once upon a time I had quite a good ISP. They were fairly technically adept, had decent uptime, and would let you use whatever equipment you liked. Unfortunately a couple of years ago they were bought by my current ISP. They’re less friendly, require you to use their own equipment, do not provide a bridge mode on their routers, and finally they do an unfortunate thing where they will redirect any DNS queries I make on my internet connection to their own DNS resolvers. It doesn’t matter if you specify different ones, the resolver that responds is always going to be theirs.</p> <p>This isn’t great from a privacy point of view; as DNS queries are generally transmitted in the clear, they can use their resolvers to get analytics on who’s doing what. I’ve been thinking about looking into their T&amp;Cs to see what’s going on there but, despite 2020, I haven’t been sufficiently bored enough yet.</p> <p>If you’re concerned that this might be happening to you, there are ways to test for it. One that’s quite easy to use is the <a href="https://www.dnsleaktest.com/">DNS leak test</a>.</p> <p>As I’m using my connection for working from home I haven’t been in a hurry to go and invite the possibility of a couple of days of downtime in a failed switchover.</p> <p>A couple of days ago everything seemed to be grinding to a halt. It seemed like there was something consuming a lot of bandwidth, but having unplugged everything it became apparent that the ISP’s DNS servers were just failing intermittently. As I write this, it’s been about 48 hours and they’re still doing it. Sadly, because of their cheeky firewall implementation, even if I configure my DHCP server to point DNS elsewhere, it still ends up on the failing resolvers. As is in-vogue these days, the status pages have not been updated to indicate that there’s any fault.</p> <h2 id="the-fixes">The fixes<a class="zola-anchor" style="display: none;" href="#the-fixes" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>So how can I work around this, if there’s no way to use another DNS service? There are two possibilities.</p> <h3 id="browsers-dns-over-https">Browsers - DNS over HTTPS<a class="zola-anchor" style="display: none;" href="#browsers-dns-over-https" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <p>Modern browsers (I’m using Firefox) have a feature built-in called DNS over HTTPS, or DoH. This routes DNS queries over an HTTPS-based transport, which appear to the ISP as standard HTTPS traffic, so they can’t really do much about it if you force it on in the browser’s settings.</p> <p>Generally speaking, browsers will default to Cloudflare (1.1.1.1) or Google (8.8.8.8) for their DoH services. There is obviously a question of trust for using either of those, though for me it was better than what was available to me already.</p> <p><a href="https://developers.cloudflare.com/1.1.1.1/dns-over-https/web-browser">Cloudflare’s pages on enabling DoH</a> cover how to do it for desktop browsers.</p> <p>Apparently DoH has made it into iOS 14 and macOS Big Sur. <a href="https://paulmillr.com/posts/encrypted-dns/">Paul Miller has a post on enabling it in iOS</a>.</p> <h3 id="everything-else-dns-over-tls">Everything else - DNS over TLS<a class="zola-anchor" style="display: none;" href="#everything-else-dns-over-tls" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <p>That covers some web browsers, but it won’t fix the other devices on my network. As well as DNS over HTTPS, there is a specification for DNS over TLS. This, to me at least, feels like the kind of thing that should have been done a while ago. Rather than sending queries over an HTTP-based API, DoT just sends the same traditional DNS queries over a TLS-based encrypted and authenticated connection.</p> <p>There’s a little bit of a chicken-and-egg thing with DoT. Because it’s TLS, and we want to use the trust features of the protocol, we want to be able to specify a hostname in the connection setup so that we can verify it in the same way as we do for HTTPS. However, we can’t resolve the hostname ourselves, so we need to specify the server’s IP and also the hostname we expect from it.</p> <p>In practice this turned out to be pretty simple. I used CoreDNS. The configuration to create a new resolver forwarding everything to Cloudflare’s 1.1.1.1 service and caching it looks like this:</p> <pre class="z-code"><code><span class="z-text z-plain"> . { </span><span class="z-text z-plain"> forward . tls://1.1.1.1 { </span><span class="z-text z-plain"> tls_servername tls.cloudflare-dns.com </span><span class="z-text z-plain"> } </span><span class="z-text z-plain"> cache </span><span class="z-text z-plain"> } </span></code></pre> <p>As I’m running this on NixOS, the changes required to make it work amounted to adding the following to <code>configuration.nix</code>:</p> <pre class="z-code"><code><span class="z-text z-plain"> services.coredns.enable = true; </span><span class="z-text z-plain"> services.coredns.config = &#39;&#39; </span><span class="z-text z-plain"> . { </span><span class="z-text z-plain"> forward . tls://1.1.1.1 { </span><span class="z-text z-plain"> tls_servername tls.cloudflare-dns.com </span><span class="z-text z-plain"> } </span><span class="z-text z-plain"> cache </span><span class="z-text z-plain"> } </span><span class="z-text z-plain"> &#39;&#39;; </span><span class="z-text z-plain"> </span></code></pre> <p>Followed by a quick <code>sudo nixos-rebuild switch</code>. I then had a functioning DNS resolver that could not be intercepted by my ISP, and I could slot its IP into my router’s DHCP options.</p> <p>This all works great, although I’m probably going to move it onto another device for a more permanent solution. I’ve not yet played with Pi-Hole, and I do expect this could use the same trick.</p> <p><strong>NB</strong>: I don’t have a good reference for this, but I’ve seen people mention that lot of IoT devices such as set top boxes do not actually pay any attention to the DNS servers issued to them by DHCP. Some of them just use 8.8.8.8 all the time regardless of what the network is telling them to do. In this particular case with an actively hostile ISP, we couldn’t solve the problem for those devices without a router or similar doing something more clever than I’ve talked about here.</p> 2020-06-13T00:00:00+00:00 2020-06-13T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/postgresql-in-a-nix-shell/ <p>Sometimes (well, quite often of late, at work) I want to load some data into a PostgreSQL database and do some work with it. I use a Mac at work, and used to do this by installing PostgreSQL using Homebrew and running it system-wide. This stung me a couple of times; Homebrew would upgrade the package to a new minor release (9.5 -&gt; 9.6, etc) and create a new data directory with the version number in it, meaning that a <code>brew upgrade</code> would sometimes make the data I loaded previously inaccessible. I never lost anything through this route, but it did slow me down.</p> <p>Nowadays, I’ve switched to using Nix for everything I can get away with, which means I can install PostgreSQL as my own user. Using Nix shells, I can even have many of them, one for each different project.</p> <p>In each directory, I have the simplest <code>shell.nix</code> I can get away with:</p> <pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-keyword z-other z-nix">with</span> <span class="z-support z-function z-nix">import</span> <span class="z-string z-unquoted z-spath z-nix">&lt;nixpkgs&gt;</span> <span class="z-punctuation z-definition z-attrset z-nix">{</span><span class="z-punctuation z-definition z-attrset z-nix">}</span>; </span><span class="z-source z-nix"><span class="z-variable z-parameter z-name z-nix">mkShell</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">buildInputs</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-list z-nix">[</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">postgresql</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset z-nix">}</span> </span></code></pre> <p>Creating this and then running <code>nix-shell</code> means you’ll end up with a new shell with PostgreSQL available, but not actually running. To make it work, you can do this the first time:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> Create a database with the data stored in the current directory</span><span class="z-comment z-line z-number-sign z-shell"> </span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">initdb</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>D</span> .tmp/mydb</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> Start PostgreSQL running as the current user</span><span class="z-comment z-line z-number-sign z-shell"> </span></span><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> and with the Unix socket in the current directory</span><span class="z-comment z-line z-number-sign z-shell"> </span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">pg_ctl</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>D</span> .tmp/mydb<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>l</span> logfile<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>o</span> <span class="z-string z-quoted z-double z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&quot;</span>--unix_socket_directories=&#39;<span class="z-meta z-group z-expansion z-parameter z-shell"><span class="z-punctuation z-definition z-variable z-shell">$</span><span class="z-variable z-other z-readwrite z-shell">PWD</span></span>&#39;<span class="z-punctuation z-definition z-string z-end z-shell">&quot;</span></span> start</span> </span><span class="z-source z-shell z-bash"> </span><span class="z-source z-shell z-bash"><span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> Create a database</span><span class="z-comment z-line z-number-sign z-shell"> </span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">createdb</span></span><span class="z-meta z-function-call z-arguments z-shell"> mydb</span> </span></code></pre> <p>Then every other time you re-enter that shell, you can just run the part that starts the database. It will keep running until you reboot, or stop it like this:</p> <pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">pg_ctl</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>D</span> .tmp/mydb stop</span> </span></code></pre> <p>Using Nix’s tools you can pin PostgreSQL to a particular version if you like, or use a more specific dependency like <code>postgresql_9_6</code> if you need it.</p> <p>Eventually, I will figure out if I can automate a lot of this with <code>shellHook</code> or similar, but it hasn’t annoyed me enough yet to make me look into doing that.</p> 2020-01-21T00:00:00+00:00 2020-01-21T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/csp-logging-with-nginx/ <h2 id="background">Background<a class="zola-anchor" style="display: none;" href="#background" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>The <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy">Content-Security-Policy HTTP response header</a> has a huge number of ways in which you can limit the kinds of resources that your site is permitted to load. This can defend against XSS attacks and other kinds of content injection. It’s very powerful and, depending on how your site is built, can be fairly simple to implement by adding a header to every response that you send, either in your application or in the web server configuration.</p> <p>CSP headers can include a directive to report to an API when one of these restrictions has been enforced. Also, if you’re working on implementing CSP, you don’t need to go straight to enforcement–<a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy-Report-Only">the Content-Security-Policy-Report-Only header</a> lets you try out a policy and see what would happen if you deployed it for real. The <code>report-uri</code> or <code>report-to</code> directives allow you to specify where to send reports when a policy has, or would have been, enforced. These are simple JSON payloads sent via an HTTP POST to a URL that you specify.</p> <p>There’s a migration going on from the original <code>report-uri</code> specification to the new <code>report-to</code> one. In the former, you just specified a URL to POST to along the lines of this, taken from the above Mozilla page:</p> <pre class="z-code"><code><span class="z-text z-plain">Content-Security-Policy: default-src https:; report-uri /csp-violation-report-endpoint/ </span></code></pre> <p>However the <code>report-to</code> specification is much more flexible and so a bit more complex to set up. It works in conjunction with the new <code>Report-To</code> header, and so would look like this:</p> <pre class="z-code"><code><span class="z-text z-plain">Report-To: { &quot;group&quot;: &quot;csp-reports&quot;, </span><span class="z-text z-plain"> &quot;max_age&quot;: 10886400, </span><span class="z-text z-plain"> &quot;endpoints&quot;: [ </span><span class="z-text z-plain"> { &quot;url&quot;: &quot;https://example.com/reports&quot; }, </span><span class="z-text z-plain"> { &quot;url&quot;: &quot;https://backup.com/reports&quot; } </span><span class="z-text z-plain"> ] } </span><span class="z-text z-plain"> </span><span class="z-text z-plain">Content-Security-Policy: default-src https:; report-to csp-reports; report-uri https://example.com/reports </span></code></pre> <p>As you can see, you can include the older <code>report-uri</code> directive as well, for browsers which only support that. Those that support <code>report-to</code> will ignore <code>report-uri</code> when both are specified.</p> <h2 id="logging-csp-reports-via-nginx">Logging CSP reports via nginx<a class="zola-anchor" style="display: none;" href="#logging-csp-reports-via-nginx" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>While there are a number of third-party services which can be used to log these reports, it’s not all that difficult to log them yourself. With nginx, you can use a custom log format to do so. This looks something like the below. I’m using a Debian-packaged nginx and have listed the files where I’ve placed this configuration.</p> <h3 id="conf-d-csp-conf">conf.d/csp.conf<a class="zola-anchor" style="display: none;" href="#conf-d-csp-conf" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <pre class="z-code"><code><span class="z-text z-plain">log_format CSP escape=json &#39;{&quot;date&quot;:&quot;$time_local&quot;, &quot;IP address&quot;:&quot;$remote_addr&quot;, &quot;http_x_forwarded_for&quot;:&quot;$http_x_forwarded_for&quot;, &quot;status&quot;:&quot;$status&quot;, &quot;http_user_agent&quot;:&quot;$http_user_agent&quot;, &quot;body_bytes_sent&quot;:&quot;$body_bytes_sent&quot;, &quot;request&quot;:&quot;$request&quot;,&quot;request_body&quot;: &quot;$request_body&quot;}&#39;; </span></code></pre> <h3 id="sites-available-default">sites-available/default<a class="zola-anchor" style="display: none;" href="#sites-available-default" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <pre class="z-code"><code><span class="z-text z-plain">server { </span><span class="z-text z-plain"> listen 443 ssl http2 default_server; </span><span class="z-text z-plain"> server_name example.org; </span><span class="z-text z-plain"> </span><span class="z-text z-plain"> # More config here </span><span class="z-text z-plain"> </span><span class="z-text z-plain"> add_header Content-Security-Policy-Report-Only &quot;default-src &#39;self&#39;; script-src &#39;self&#39; &#39;unsafe-inline&#39; fonts.googleapis.com ssl.google-analytics.com; font-src &#39;self&#39; fonts.gstatic.com; style-src &#39;self&#39; &#39;unsafe-inline&#39; fonts.googleapis.com; img-src &#39;self&#39; ssl.google-analytics.com; report-uri https://example.org/_csp&quot;; </span><span class="z-text z-plain"> </span><span class="z-text z-plain"> location = /_csp { </span><span class="z-text z-plain"> access_log /var/log/nginx/csp.log CSP; </span><span class="z-text z-plain"> proxy_pass http://127.0.0.1/_csp_response; </span><span class="z-text z-plain"> } </span><span class="z-text z-plain">} </span><span class="z-text z-plain"> </span><span class="z-text z-plain">server { </span><span class="z-text z-plain"> listen 80 default_server; </span><span class="z-text z-plain"> server_name example.org; </span><span class="z-text z-plain"> </span><span class="z-text z-plain"> # More config here </span><span class="z-text z-plain"> </span><span class="z-text z-plain"> location /_csp_response { </span><span class="z-text z-plain"> access_log off; </span><span class="z-text z-plain"> return 204; </span><span class="z-text z-plain"> } </span><span class="z-text z-plain">} </span></code></pre> <p>At first glance, the <code>proxy_pass</code> directive may look a bit suspicious. The reason it’s there is because if you just do <code>return 204</code> directly from the <code>/_csp</code> location, the request body is not logged in the <code>csp.log</code> file. By using the <code>proxy_pass</code> hack, it is. You may also notice in this example I’m only configuring the older <code>report-uri</code> directive.</p> <p>Something that would be good to do would be to limit the above requests to only HTTP POSTs, and also ensure that the requests have the correct <code>Content-Type</code> header for the CSP report (which would be <code>application/csp-report</code>). I’m not doing that in this case to keep the example clearer, but I’d certainly think about it in production.</p> <p>The actual reports look like this in the log file. If you’re using an older nginx, before <code>1.11.8</code>, it may not support the <code>escape=json</code> part of the log format and you may see all of the <code>"</code> characters in your log being replaced with <code>\x22</code>. You’ll probably want to upgrade to fix that, not least because that version is quite old!</p> <pre class="z-code"><code><span class="z-text z-plain">{&quot;date&quot;:&quot;21/Jan/2020:21:58:46 +0000&quot;, &quot;IP address&quot;:&quot;198.51.100.4&quot;, &quot;http_x_forwarded_for&quot;:&quot;&quot;, &quot;status&quot;:&quot;204&quot;, &quot;http_user_agent&quot;:&quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36&quot;, &quot;body_bytes_sent&quot;:&quot;0&quot;, &quot;request&quot;:&quot;POST /_csp HTTP/2.0&quot;,&quot;request-body&quot;: {\&quot;csp-report\&quot;:{\&quot;document-uri\&quot;:\&quot;https://example.org/\&quot;,\&quot;referrer\&quot;:\&quot;\&quot;,\&quot;violated-directive\&quot;:\&quot;font-src\&quot;,\&quot;effective-directive\&quot;:\&quot;font-src\&quot;,\&quot;original-policy\&quot;:\&quot;default-src &#39;self&#39;; script-src &#39;self&#39; &#39;unsafe-inline&#39; fonts.googleapis.com ssl.google-analytics.com gist.github.com; font-src &#39;self&#39; fonts.gstatic.com; style-src &#39;self&#39; &#39;unsafe-inline&#39; fonts.googleapis.com github.githubassets.com; img-src &#39;self&#39; ssl.google-analytics.com; report-uri https://example.org/_csp\&quot;,\&quot;disposition\&quot;:\&quot;report\&quot;,\&quot;blocked-uri\&quot;:\&quot;\&quot;,\&quot;line-number\&quot;:366,\&quot;column-number\&quot;:179,\&quot;source-file\&quot;:\&quot;https://example.org/assets/js/modernizr.js\&quot;,\&quot;status-code\&quot;:0,\&quot;script-sample\&quot;:\&quot;\&quot;}}} </span></code></pre> <p>From here you can use tools like <code>jq</code> to inspect what you find. You’ll note that the actual report is recorded in the <code>request_body</code> property in the JSON, and that this contents is escaped. You can recover this like so:</p> <pre class="z-code"><code><span class="z-text z-plain">$ cat /var/log/nginx/csp.log | jq -r &#39;.request_body | fromjson&#39; </span><span class="z-text z-plain"> </span><span class="z-text z-plain">{ </span><span class="z-text z-plain"> &quot;csp-report&quot;: { </span><span class="z-text z-plain"> &quot;document-uri&quot;: &quot;https://example.org/&quot;, </span><span class="z-text z-plain"> &quot;referrer&quot;: &quot;&quot;, </span><span class="z-text z-plain"> &quot;violated-directive&quot;: &quot;font-src&quot;, </span><span class="z-text z-plain"> &quot;effective-directive&quot;: &quot;font-src&quot;, </span><span class="z-text z-plain"> &quot;original-policy&quot;: &quot;default-src &#39;self&#39;; script-src &#39;self&#39; &#39;unsafe-inline&#39; fonts.googleapis.com ssl.google-analytics.com gist.github.com; font-src &#39;self&#39; fonts.gstatic.com; style-src &#39;self&#39; &#39;unsafe-inline&#39; fonts.googleapis.com github.githubassets.com; img-src &#39;self&#39; ssl.google-analytics.com; report-uri https://example.org/_csp&quot;, </span><span class="z-text z-plain"> &quot;disposition&quot;: &quot;report&quot;, </span><span class="z-text z-plain"> &quot;blocked-uri&quot;: &quot;&quot;, </span><span class="z-text z-plain"> &quot;line-number&quot;: 366, </span><span class="z-text z-plain"> &quot;column-number&quot;: 179, </span><span class="z-text z-plain"> &quot;source-file&quot;: &quot;https://example.org/assets/js/modernizr.js&quot;, </span><span class="z-text z-plain"> &quot;status-code&quot;: 0, </span><span class="z-text z-plain"> &quot;script-sample&quot;: &quot;&quot; </span><span class="z-text z-plain"> } </span><span class="z-text z-plain">} </span></code></pre> <p>From this I can tell that Chrome is not entirely happy with how Modernizr is doing some feature detection around web fonts. I’ll need to look into that, but that is for another day. Later work could also include implementing the other parts of the <code>report-to</code> spec, which include neat things like <a href="https://scotthelme.co.uk/network-error-logging-deep-dive/">network error logging, as explained here by Scott Helme</a>.</p> <p><strong>NB:</strong> You will almost certainly want to add the <code>csp.log</code> file to your <code>logrotate</code> configuration, or something similar, otherwise you may end up with a chunky file eating a lot of disk space if your site doesn’t quite comply with your policy!</p> 2019-08-22T00:00:00+00:00 2019-08-22T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/flask-on-nixos/ <p>I recently acquired a second hand ThinkPad to run NixOS on, and I’ve been using it recently to mess around with a couple of small web projects. I’m prototyping one of these in Flask, and have been using NixOS’s own Flask package in order to do so.</p> <p>I found that I could quite happily do <code>flask run</code>, but as soon as I tried to enable development mode, I’d end up with an error like this:</p> <pre class="z-code"><code><span class="z-text z-plain">$ FLASK_APP=myapp FLASK_ENV=development flask run </span><span class="z-text z-plain"> * Serving Flask app &quot;myapp&quot; (lazy loading) </span><span class="z-text z-plain"> * Environment: development </span><span class="z-text z-plain"> * Debug mode: on </span><span class="z-text z-plain"> * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit) </span><span class="z-text z-plain"> * Restarting with stat </span><span class="z-text z-plain"> File &quot;/nix/store/dpdfxfjf6azdgnka78fhsch3a8y3d46g-python3.7-Flask-1.0.2/bin/flask&quot;, line 2 </span><span class="z-text z-plain"> export PATH=&#39;/nix/store/96p42644i76jqgdkxjgyl3f0c8n8k16j-python3-3.7.4/bin:/nix/store/dpdfxfjf6azdgnka78fhsch3a8y3d46g-python3.7-Flask-1.0.2/bin:/nix/store/dvzig77cx97rdxn5i2pdmyblm8f6pf7d-python3.7-setuptools-40.8.0/bin&#39;${PATH:+&#39;:&#39;}$PATH </span><span class="z-text z-plain"> ^ </span><span class="z-text z-plain">SyntaxError: invalid syntax </span></code></pre> <p>This looks a bit like something is attempting to run a Bash script through the Python interpreter. I didn’t understand why this was happening until I encountered <a href="https://github.com/NixOS/nixpkgs/issues/42924">this issue on Nix’s github</a> which suggests that it ought to be fixed upstream, but that it might be possible to override it in Nix. The code given there to do so didn’t neatly copy and paste into <code>shell.nix</code> as I had hoped, but after some tinkering I got it to work. Here’s what I ended up with:</p> <pre data-lang="nix" class="language-nix z-code"><code class="language-nix" data-lang="nix"><span class="z-source z-nix"><span class="z-keyword z-other z-nix">with</span> <span class="z-support z-function z-nix">import</span> <span class="z-string z-unquoted z-spath z-nix">&lt;nixpkgs&gt;</span> <span class="z-punctuation z-definition z-attrset z-nix">{</span><span class="z-punctuation z-definition z-attrset z-nix">}</span>; </span><span class="z-source z-nix"> </span><span class="z-source z-nix"><span class="z-keyword z-other z-nix">let</span> </span><span class="z-source z-nix"><span class="z-entity z-other z-attribute-name z-multipart z-nix">myWerkzeug</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-variable z-parameter z-name z-nix">python37Packages</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">werkzeug</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">overrideAttrs</span> <span class="z-punctuation z-definition z-expression z-nix">(</span><span class="z-variable z-parameter z-function z-4 z-nix">oldAttrs</span><span class="z-punctuation z-definition z-function z-nix">:</span> <span class="z-keyword z-other z-nix">rec</span> <span class="z-punctuation z-definition z-attrset z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">postPatch</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-string z-quoted z-other z-nix"><span class="z-punctuation z-definition z-string z-other z-start z-nix">&#39;&#39;</span> </span></span><span class="z-source z-nix"><span class="z-string z-quoted z-other z-nix"> substituteInPlace werkzeug/_reloader.py \ </span></span><span class="z-source z-nix"><span class="z-string z-quoted z-other z-nix"> --replace &quot;rv = [sys.executable]&quot; &quot;return sys.argv&quot; </span></span><span class="z-source z-nix"><span class="z-string z-quoted z-other z-nix"> <span class="z-punctuation z-definition z-string z-other z-end z-nix">&#39;&#39;</span></span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">doCheck</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-constant z-language z-nix">false</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-definition z-expression z-nix">)</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"><span class="z-entity z-other z-attribute-name z-multipart z-nix">myFlask</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-variable z-parameter z-name z-nix">python37Packages</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">flask</span><span class="z-keyword z-operator z-nix">.</span><span class="z-variable z-parameter z-name z-nix">override</span> <span class="z-punctuation z-definition z-expression z-nix">(</span><span class="z-punctuation z-definition z-attrset z-nix">{</span> <span class="z-entity z-other z-attribute-name z-multipart z-nix">werkzeug</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-variable z-parameter z-name z-nix">myWerkzeug</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> <span class="z-punctuation z-definition z-attrset z-nix">}</span><span class="z-punctuation z-definition z-expression z-nix">)</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"> </span><span class="z-source z-nix"><span class="z-keyword z-other z-nix">in</span> <span class="z-variable z-parameter z-name z-nix">mkShell</span> <span class="z-punctuation z-definition z-attrset-or-function z-nix">{</span> </span><span class="z-source z-nix"> <span class="z-entity z-other z-attribute-name z-multipart z-nix">buildInputs</span> <span class="z-keyword z-operator z-bind z-nix">=</span> <span class="z-punctuation z-definition z-list z-nix">[</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">python37</span> </span><span class="z-source z-nix"> <span class="z-variable z-parameter z-name z-nix">myFlask</span> </span><span class="z-source z-nix"> <span class="z-punctuation z-definition z-list z-nix">]</span><span class="z-punctuation z-terminator z-bind z-nix">;</span> </span><span class="z-source z-nix"><span class="z-punctuation z-definition z-attrset z-nix">}</span> </span></code></pre> <p>Probably not hard to anyone who properly understands Nix, but I learned a couple of things doing it. Hopefully someone else will find this useful.</p> 2019-05-20T00:00:00+00:00 2019-05-20T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/htmlq/ <p>Continuing my journey into learning Rust, I have been looking around for small, useful projects that might be interesting to build rather than doing exercises. In the mean time I was doing some web scraping using Python and BeautifulSoup, which is great but it can be a little slow. I am aware of Servo, the browser written by Mozilla in Rust, and had the idea to try and recycle parts of it to make a small CLI tool to extract data from HTML documents.</p> <p>The obvious choice for me for the query language was CSS selectors. While XPath works fine for XML, the more specific selector language seems like a better fit for what I wanted to do here. I intended for the CLI to be fairly simple, and fast.</p> <p>Servo’s HTML rendering library is called <code>html5ever</code>. I initially looked at using this directly, and integrating with the CSS selector parsing engine, but it turns out that someone else has done most of the work I needed here and wrapped both in a library called <a href="https://github.com/kuchiki-rs/kuchiki">Kuchiki</a>. Most of the hard work is actually done in here. Parsing the HTML document becomes as simple as something like this:</p> <pre class="z-code"><code><span class="z-text z-plain">let document = kuchiki::parse_html() </span><span class="z-text z-plain"> .from_utf8() </span><span class="z-text z-plain"> .read_from(&amp;mut input) </span><span class="z-text z-plain"> .unwrap(); </span></code></pre> <p>I’m very grateful to Kuchiki’s authors for doing most of the hard work!</p> <h3 id="examples">Examples<a class="zola-anchor" style="display: none;" href="#examples" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <p>For example, if I wanted to get the HTML for the section with ID <code>posts</code> on this blog, I could do:</p> <pre class="z-code"><code><span class="z-text z-plain">curl -s https://mgdm.net | htmlq &#39;#posts&#39; </span></code></pre> <p>Or to find all the links inside <code>&lt;a&gt;</code> elements on a page:</p> <pre class="z-code"><code><span class="z-text z-plain">curl -s https://www.rust-lang.org/ | htmlq -a href a </span></code></pre> <p>The <code>-a</code> option takes the name of an attribute to return, and the final <code>a</code> is the CSS selector. Elements that do not have the matching attribute are ignored.</p> <p>The <code>-t</code> option allows you to just return the text content, ignoring the rest of the markup. There’s also a <code>-p</code> which attempts to pretty-print the HTML, but this is a bit of a work in progress.</p> <p><a href="https://crates.io/crates/htmlq">It’s now on crates.io</a> and <a href="https://github.com/mgdm/htmlq">the code is on GitHub</a>. You should be able to install it on Unix-like systems with <code>cargo install htmlq</code>. I’d be happy to hear any feedback people may have, and I hope that it might be useful.</p> 2019-04-29T00:00:00+00:00 2019-04-29T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/nettop/ <p>Recently I’ve been learning Rust, and by far the simplest way to install that is to use <a href="https://rustup.rs/">Rustup</a>. That works really well on my home computer, but I wanted to try solving a problem I had at work using Rust, so I tried it on my work Mac. My company’s network has a fairly restrictive set of firewalls, and I found that rustup was somehow failing to connect to the network to download the indexes it needs. The verbose output didn’t give me any clues, and I found it hard to narrow down what it was trying to do using <code>tcpdump</code>.</p> <p>After some searching, I came across <code>nettop</code>. It seems to a tool originating from BSD that is mostly used for monitoring network activity across the system, but it’s also possible to narrow it down to a single process:</p> <pre class="z-code"><code><span class="z-text z-plain">sudo nettop -p rustup -L 0 </span></code></pre> <p>This shows all the network connections being made processes called <code>rustup</code>, in this sort of format:</p> <pre class="z-code"><code><span class="z-text z-plain">21:40:52.233621,tcp4 192.168.33.10:50963&lt;-&gt;server-99-84-8-62.lhr62.r.cloudfront.net:443,utun1,SynSent,0,0,0,0,0,0.00 ms,131072,0,BE,-,cubic,-,-,-,-, </span></code></pre> <p>This indicates that <code>rustup</code> is trying to connect to port 443 on a server in Cloudfront.</p> <h2 id="the-rustup-issue">The rustup issue<a class="zola-anchor" style="display: none;" href="#the-rustup-issue" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>In this case, it turns out that for some reason it was trying to connect to our corporate web proxy on port 8080 instead of port 80. The proxy is set in my environment as <code>http://corporate-proxy.example.org:80</code>. I tweeted about this, and <a href="https://blog.digital-scurf.org/">Daniel Silverstone</a> found that there was an edge case in the <code>url</code> crate used by the <code>env_proxy</code> crate, in turn used by Rustup to parse the <code>https_proxy</code> environment variable–it would omit the port 80 as that is the default for HTTP, and then <code>env_proxy</code> would assume that because the port had been omitted it should try 8080. This should be fixed shortly! Thanks again to Daniel for the help.</p> 2019-01-05T00:00:00+00:00 2019-01-05T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/installing-nixos/ <p>It’s been a while since I’ve had a machine running Linux, and even longer since I’ve had actual desktop hardware. Recently my brother upgraded his gaming PC, and I acquired the old motherboard, CPU, RAM and graphics card. It’s a couple of years old but definitely quick enough for my purposes.</p> <p>I decided to install NixOS on it to give it a proper go. I’d tried Nix on my work OSX laptop for a while, but a lot of tools at work assume you’re using Homebrew so I had to revert back to that. <a href="https://nixos.org/releases/nix/nix-1.9/manual/#chap-installation">The manual</a> is pretty informative, but as I wanted to run with full disk encryption I used <a href="https://twitter.com/grhmc">Graham Christensen</a>’s <a href="https://grahamc.com/blog/nixos-on-dell-9560">instructions for installing on a Dell machine</a> as the base. All that I had to change were the references to the drive names, as my SATA SSD appeared as <code>/dev/sda</code>.</p> <h2 id="things-to-look-out-for">Things to look out for<a class="zola-anchor" style="display: none;" href="#things-to-look-out-for" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <h3 id="uefi-vs-bios">UEFI vs BIOS<a class="zola-anchor" style="display: none;" href="#uefi-vs-bios" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <p>The installation was relatively straightforward following Graham’s instructions, but when I got to the point where it was going to install Grub, it came up with errors like this:</p> <pre class="z-code"><code><span class="z-text z-plain">nix/store/6czhmdkzs04y1bdlq53yvw6hn1bvl1zr-grub-2.x-2015-07-27/sbin/grub-install: warning: this GPT partition label contains no BIOS Boot Partition; embedding won&#39;t be possible. </span><span class="z-text z-plain">/nix/store/6czhmdkzs04y1bdlq53yvw6hn1bvl1zr-grub-2.x-2015-07-27/sbin/grub-install: warning: Embedding is not possible. GRUB can only be installed in this setup by using blocklists. However, blocklists are UNRELIABLE and their use is discouraged.. </span><span class="z-text z-plain">/nix/store/6czhmdkzs04y1bdlq53yvw6hn1bvl1zr-grub-2.x-2015-07-27/sbin/grub-install: error: will not proceed with blocklists. </span><span class="z-text z-plain">/nix/store/w345fzhsr2ckzhazmxr3lc906glnxqiy-install-grub.pl: installation of GRUB on /dev/sda failed </span></code></pre> <p>(These are not the precise errors I got as the paths are different, but they’re the same messages). I had partitioned the SSD with a GPT, as I knew the machine had UEFI, but it turned out that for some reason the installer had opted to boot in BIOS mode. Some messing around in the firmware followed, until I scrolled and found the option to disable the “Compatibility Support Module”, or CSM. Rebooting with that switched off made the installer boot in UEFI mode, which was evident by it having a very much smaller font. I copied the existing <code>hardware-configuration.nix</code> and <code>configuration.nix</code> files and re-ran <code>nixos-generate-config</code>, then put back the <code>boot.initrd.luks.devices</code> section and finally ran <code>nixos-install</code> again.</p> <h3 id="usb-keyboards">USB keyboards<a class="zola-anchor" style="display: none;" href="#usb-keyboards" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <p>I rebooted, and got to the prompt that asked for the passphrase to unlock the disks. I found out quite quickly that the kernel didn’t know what to do with my USB keyboard. I rebooted back into the installer, mounted the disks, and added <code>hid_microsoft</code> to the list in <code>boot.initrd.availableKernelModules</code> and re-ran <code>nixos-install</code>. Rebooting then got me a keyboard. I do have a Microsoft keyboard attached to this machine–I suspect I could have got away with the <code>usbhid</code> module, but this one seemed to make sense.</p> <h3 id="fun-with-uuids">Fun with UUIDs<a class="zola-anchor" style="display: none;" href="#fun-with-uuids" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <p><strong>Make absolutely certain that the UUIDs you use for the various drives are correct</strong>. I got quite mixed up here, as when I wrote the <code>boot.initrd.luks.devices</code> section in <code>hardware-configuration.nix</code> I was using the UUIDs for the mapped devices (<code>dm-1</code> and <code>dm-2</code>) which appear after the disks have been unlocked. If you look at <code>hardware-configuration.nix</code> and find the UUIDs in <code>fileSystems."/"</code> or <code>swapDevices</code> are the same as those in your LUKS devices config, then you’ve got the LUKS config wrong, and will want to have a look carefully at the output of <code>ls -l /dev/disks/by-uuid/</code> to find the ones that reference actual physical disk partitions like in my case <code>/dev/sda3</code>, and not mapped devices like <code>dm-1</code> or something under <code>/dev/mapper</code>.</p> <p>Fixing this, and one more <code>nixos-install</code>, got me to reboot into a working machine.</p> 2015-06-25T00:00:00+00:00 2015-06-25T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/mffi/ <p>In PHP, if you want to make use of a library written in C, you generally have to go and write a new PHP extension. This isn’t that hard if you’ve done it before, but it does require a bit of work and some getting used to. There’s also a fair bit of boilerplate code that is required to create functions, classes, and methods, and to handle parameter parsing. The documentation for this side of things isn’t that great right now, though there are <a href="http://www.phpinternalsbook.com/">efforts to change this</a>. One advantage is that there are quite a number of people who have done it before, and you can use tools like <a href="http://lxr.php.net">PHP’s OpenGrok</a> to find examples and inspiration.</p> <p>It would be nice to make this sort of thing simpler to do. Perhaps, we could avoid having to write any C code. As I mentioned in <a href="http://mgdm.net/weblog/php-at-the-speed-of-c/">my previous post</a>, there are other tools in the works to make accelerating code written in PHP faster. I mentioned Anthony Ferrara’s <a href="https://github.com/google/recki-ct">Recki-CT</a>, which takes PHP code and accelerates that. There is also the <a href="http://zephir-lang.com/">Zephir project</a> which takes code written in a PHP-like language and can create extensions for you from that.</p> <p>Both of these require a bit of time to set up and operate, though. They’re also not really aimed at using existing libraries – their goal is to accelerate self-contained high-level code rather than to provide access to lower-level libraries.</p> <p>Some languages have a feature that allows the user to load C shared libraries and call functions from them in the same way you’d call any other function in that language. These are known as <a href="https://en.wikipedia.org/wiki/Foreign_function_interface"><em>foreign function interfaces</em></a>. Python has a number of these – <a href="https://docs.python.org/2/library/ctypes.html">ctypes</a> is one example, and <a href="https://cffi.readthedocs.org/en/latest/">CFFI</a> is another. Similarly, Ruby has the <a href="https://github.com/ffi/ffi">FFI</a> library. Many other languages possess these. There has been one for PHP for a while, called <a href="http://pecl.php.net/package/ffi">ext/ffi</a>, but it hasn’t had a release in a number of years.</p> <h2 id="mffi">MFFI<a class="zola-anchor" style="display: none;" href="#mffi" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>I decided it sounded like an interesting project, so I would like to present to the world <a href="https://github.com/mgdm/MFFI">MFFI</a>.</p> <p>It’s an attempt at making a new, intuitive, FFI for PHP. Rather than take the approach of attempting to parse C headers, like <code>ext/ffi</code> and Python’s CFFI do, it uses PHP to declare the types. This is easiest to show in an example:</p> <pre class="z-code"><code><span class="z-text z-plain">&lt;?php </span><span class="z-text z-plain">$library = new MFFI\Library(); </span><span class="z-text z-plain">$puts = $library-&gt;bind(&#39;puts&#39;, [ MFFI\Type::TYPE_STRING ], MFFI\Type::TYPE_INT); </span><span class="z-text z-plain">$puts(&#39;Hello world&#39;); </span></code></pre> <p>The code above binds to the PHP process itself, meaning that any functions available within the process can be called. It then creates a new <code>MFFI\Func</code> object to represent <code>libc</code>’s <code>puts</code> function. The second parameter to <code>MFFI\Library::bind()</code> is an array representing the types of the arguments (in this case just one, a string), and the final argument is the return type (an int in this case). Finally, it calls the function with a standard PHP string. Running this code does what you’d expect:</p> <pre class="z-code"><code><span class="z-text z-plain">michael@morbo:mffi% php7 test.php </span><span class="z-text z-plain">Hello world </span><span class="z-text z-plain">michael@morbo:mffi% </span></code></pre> <p>It’s a fairly new extension, so currently it doesn’t have that many features. One thing it can do at the moment is handle custom C structs. You can see this in the code below.</p> <pre data-lang="php" class="language-php z-code"><code class="language-php" data-lang="php"><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-punctuation z-section z-embedded z-begin z-php">&lt;?php</span></span><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-meta z-use z-php"><span class="z-keyword z-other z-use z-php">use</span> </span><span class="z-meta z-use z-php"><span class="z-meta z-path z-php"><span class="z-support z-other z-namespace z-php">MFFI</span><span class="z-punctuation z-separator z-namespace z-php">\</span></span><span class="z-meta z-path z-php"><span class="z-support z-class z-php">Type</span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-meta z-use z-php"><span class="z-keyword z-other z-use z-php">use</span> </span><span class="z-meta z-use z-php"><span class="z-meta z-path z-php"><span class="z-support z-other z-namespace z-php">MFFI</span><span class="z-punctuation z-separator z-namespace z-php">\</span></span><span class="z-meta z-path z-php"><span class="z-support z-class z-php">Struct</span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-meta z-use z-php"><span class="z-keyword z-other z-use z-php">use</span> </span><span class="z-meta z-use z-php"><span class="z-meta z-path z-php"><span class="z-support z-other z-namespace z-php">MFFI</span><span class="z-punctuation z-separator z-namespace z-php">\</span></span><span class="z-meta z-path z-php"><span class="z-support z-class z-php">Library</span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-meta z-class z-php"><span class="z-storage z-type z-class z-php">class</span> <span class="z-entity z-name z-class z-php">TimeStruct</span> <span class="z-storage z-modifier z-extends z-php">extends</span> <span class="z-meta z-path z-php"><span class="z-entity z-other z-inherited-class z-php">Struct</span></span> <span class="z-meta z-block z-php"><span class="z-punctuation z-section z-block z-php">{</span></span></span><span class="z-meta z-class z-php"><span class="z-meta z-block z-php"> </span></span></span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-meta z-class z-php"><span class="z-meta z-block z-php"> <span class="z-storage z-modifier z-php">static</span> <span class="z-meta z-function z-php"><span class="z-storage z-type z-function z-php">function</span> <span class="z-entity z-name z-function z-php">definition</span></span><span class="z-meta z-function z-parameters z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function z-parameters z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span><span class="z-meta z-function z-php"> <span class="z-meta z-block z-php"><span class="z-punctuation z-section z-block z-begin z-php">{</span></span></span><span class="z-meta z-function z-php"><span class="z-meta z-block z-php"> </span></span></span></span></span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-meta z-class z-php"><span class="z-meta z-block z-php"><span class="z-meta z-function z-php"><span class="z-meta z-block z-php"> <span class="z-keyword z-control z-php">return</span> <span class="z-meta z-array z-php"><span class="z-punctuation z-section z-array z-begin z-php">[</span> </span></span></span></span></span></span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-meta z-class z-php"><span class="z-meta z-block z-php"><span class="z-meta z-function z-php"><span class="z-meta z-block z-php"><span class="z-meta z-array z-php"> <span class="z-string z-quoted z-single z-php"><span class="z-punctuation z-definition z-string z-begin z-php">&#39;</span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php"></span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php">tm_sec</span><span class="z-punctuation z-definition z-string z-end z-php">&#39;</span></span> <span class="z-keyword z-operator z-key z-php">=&gt;</span> <span class="z-meta z-path z-php"><span class="z-support z-class z-php">Type</span></span><span class="z-punctuation z-accessor z-double-colon z-php">::</span><span class="z-constant z-other z-class z-php">TYPE_INT</span>, <span class="z-comment z-block z-php"><span class="z-punctuation z-definition z-comment z-begin z-php">/*</span> seconds (0 - 60) <span class="z-punctuation z-definition z-comment z-end z-php">*/</span></span> </span></span></span></span></span></span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-meta z-class z-php"><span class="z-meta z-block z-php"><span class="z-meta z-function z-php"><span class="z-meta z-block z-php"><span class="z-meta z-array z-php"> <span class="z-string z-quoted z-single z-php"><span class="z-punctuation z-definition z-string z-begin z-php">&#39;</span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php"></span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php">tm_min</span><span class="z-punctuation z-definition z-string z-end z-php">&#39;</span></span> <span class="z-keyword z-operator z-key z-php">=&gt;</span> <span class="z-meta z-path z-php"><span class="z-support z-class z-php">Type</span></span><span class="z-punctuation z-accessor z-double-colon z-php">::</span><span class="z-constant z-other z-class z-php">TYPE_INT</span>, <span class="z-comment z-block z-php"><span class="z-punctuation z-definition z-comment z-begin z-php">/*</span> minutes (0 - 59) <span class="z-punctuation z-definition z-comment z-end z-php">*/</span></span> </span></span></span></span></span></span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-meta z-class z-php"><span class="z-meta z-block z-php"><span class="z-meta z-function z-php"><span class="z-meta z-block z-php"><span class="z-meta z-array z-php"> <span class="z-string z-quoted z-single z-php"><span class="z-punctuation z-definition z-string z-begin z-php">&#39;</span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php"></span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php">tm_hour</span><span class="z-punctuation z-definition z-string z-end z-php">&#39;</span></span> <span class="z-keyword z-operator z-key z-php">=&gt;</span> <span class="z-meta z-path z-php"><span class="z-support z-class z-php">Type</span></span><span class="z-punctuation z-accessor z-double-colon z-php">::</span><span class="z-constant z-other z-class z-php">TYPE_INT</span>, <span class="z-comment z-block z-php"><span class="z-punctuation z-definition z-comment z-begin z-php">/*</span> hours (0 - 23) <span class="z-punctuation z-definition z-comment z-end z-php">*/</span></span> </span></span></span></span></span></span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-meta z-class z-php"><span class="z-meta z-block z-php"><span class="z-meta z-function z-php"><span class="z-meta z-block z-php"><span class="z-meta z-array z-php"> <span class="z-string z-quoted z-single z-php"><span class="z-punctuation z-definition z-string z-begin z-php">&#39;</span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php"></span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php">tm_mday</span><span class="z-punctuation z-definition z-string z-end z-php">&#39;</span></span> <span class="z-keyword z-operator z-key z-php">=&gt;</span> <span class="z-meta z-path z-php"><span class="z-support z-class z-php">Type</span></span><span class="z-punctuation z-accessor z-double-colon z-php">::</span><span class="z-constant z-other z-class z-php">TYPE_INT</span>, <span class="z-comment z-block z-php"><span class="z-punctuation z-definition z-comment z-begin z-php">/*</span> day of month (1 - 31) <span class="z-punctuation z-definition z-comment z-end z-php">*/</span></span> </span></span></span></span></span></span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-meta z-class z-php"><span class="z-meta z-block z-php"><span class="z-meta z-function z-php"><span class="z-meta z-block z-php"><span class="z-meta z-array z-php"> <span class="z-string z-quoted z-single z-php"><span class="z-punctuation z-definition z-string z-begin z-php">&#39;</span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php"></span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php">tm_mon</span><span class="z-punctuation z-definition z-string z-end z-php">&#39;</span></span> <span class="z-keyword z-operator z-key z-php">=&gt;</span> <span class="z-meta z-path z-php"><span class="z-support z-class z-php">Type</span></span><span class="z-punctuation z-accessor z-double-colon z-php">::</span><span class="z-constant z-other z-class z-php">TYPE_INT</span>, <span class="z-comment z-block z-php"><span class="z-punctuation z-definition z-comment z-begin z-php">/*</span> month of year (0 - 11) <span class="z-punctuation z-definition z-comment z-end z-php">*/</span></span> </span></span></span></span></span></span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-meta z-class z-php"><span class="z-meta z-block z-php"><span class="z-meta z-function z-php"><span class="z-meta z-block z-php"><span class="z-meta z-array z-php"> <span class="z-string z-quoted z-single z-php"><span class="z-punctuation z-definition z-string z-begin z-php">&#39;</span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php"></span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php">tm_year</span><span class="z-punctuation z-definition z-string z-end z-php">&#39;</span></span> <span class="z-keyword z-operator z-key z-php">=&gt;</span> <span class="z-meta z-path z-php"><span class="z-support z-class z-php">Type</span></span><span class="z-punctuation z-accessor z-double-colon z-php">::</span><span class="z-constant z-other z-class z-php">TYPE_INT</span>, <span class="z-comment z-block z-php"><span class="z-punctuation z-definition z-comment z-begin z-php">/*</span> year - 1900 <span class="z-punctuation z-definition z-comment z-end z-php">*/</span></span> </span></span></span></span></span></span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-meta z-class z-php"><span class="z-meta z-block z-php"><span class="z-meta z-function z-php"><span class="z-meta z-block z-php"><span class="z-meta z-array z-php"> <span class="z-string z-quoted z-single z-php"><span class="z-punctuation z-definition z-string z-begin z-php">&#39;</span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php"></span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php">tm_wday</span><span class="z-punctuation z-definition z-string z-end z-php">&#39;</span></span> <span class="z-keyword z-operator z-key z-php">=&gt;</span> <span class="z-meta z-path z-php"><span class="z-support z-class z-php">Type</span></span><span class="z-punctuation z-accessor z-double-colon z-php">::</span><span class="z-constant z-other z-class z-php">TYPE_INT</span>, <span class="z-comment z-block z-php"><span class="z-punctuation z-definition z-comment z-begin z-php">/*</span> day of week (Sunday = 0) <span class="z-punctuation z-definition z-comment z-end z-php">*/</span></span> </span></span></span></span></span></span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-meta z-class z-php"><span class="z-meta z-block z-php"><span class="z-meta z-function z-php"><span class="z-meta z-block z-php"><span class="z-meta z-array z-php"> <span class="z-string z-quoted z-single z-php"><span class="z-punctuation z-definition z-string z-begin z-php">&#39;</span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php"></span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php">tm_yday</span><span class="z-punctuation z-definition z-string z-end z-php">&#39;</span></span> <span class="z-keyword z-operator z-key z-php">=&gt;</span> <span class="z-meta z-path z-php"><span class="z-support z-class z-php">Type</span></span><span class="z-punctuation z-accessor z-double-colon z-php">::</span><span class="z-constant z-other z-class z-php">TYPE_INT</span>, <span class="z-comment z-block z-php"><span class="z-punctuation z-definition z-comment z-begin z-php">/*</span> day of year (0 - 365) <span class="z-punctuation z-definition z-comment z-end z-php">*/</span></span> </span></span></span></span></span></span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-meta z-class z-php"><span class="z-meta z-block z-php"><span class="z-meta z-function z-php"><span class="z-meta z-block z-php"><span class="z-meta z-array z-php"> <span class="z-string z-quoted z-single z-php"><span class="z-punctuation z-definition z-string z-begin z-php">&#39;</span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php"></span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php">tm_isdst</span><span class="z-punctuation z-definition z-string z-end z-php">&#39;</span></span> <span class="z-keyword z-operator z-key z-php">=&gt;</span> <span class="z-meta z-path z-php"><span class="z-support z-class z-php">Type</span></span><span class="z-punctuation z-accessor z-double-colon z-php">::</span><span class="z-constant z-other z-class z-php">TYPE_INT</span>, <span class="z-comment z-block z-php"><span class="z-punctuation z-definition z-comment z-begin z-php">/*</span> is summer time in effect? <span class="z-punctuation z-definition z-comment z-end z-php">*/</span></span> </span></span></span></span></span></span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-meta z-class z-php"><span class="z-meta z-block z-php"><span class="z-meta z-function z-php"><span class="z-meta z-block z-php"><span class="z-meta z-array z-php"> <span class="z-string z-quoted z-single z-php"><span class="z-punctuation z-definition z-string z-begin z-php">&#39;</span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php"></span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php">tm_zone</span><span class="z-punctuation z-definition z-string z-end z-php">&#39;</span></span> <span class="z-keyword z-operator z-key z-php">=&gt;</span> <span class="z-meta z-path z-php"><span class="z-support z-class z-php">Type</span></span><span class="z-punctuation z-accessor z-double-colon z-php">::</span><span class="z-constant z-other z-class z-php">TYPE_STRING</span>, <span class="z-comment z-block z-php"><span class="z-punctuation z-definition z-comment z-begin z-php">/*</span> abbreviation of timezone name <span class="z-punctuation z-definition z-comment z-end z-php">*/</span></span> </span></span></span></span></span></span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-meta z-class z-php"><span class="z-meta z-block z-php"><span class="z-meta z-function z-php"><span class="z-meta z-block z-php"><span class="z-meta z-array z-php"> <span class="z-punctuation z-section z-array z-end z-php">]</span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span></span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-meta z-class z-php"><span class="z-meta z-block z-php"><span class="z-meta z-function z-php"><span class="z-meta z-block z-php"> </span></span><span class="z-meta z-function z-php"><span class="z-meta z-block z-php"><span class="z-punctuation z-section z-block z-end z-php">}</span></span></span> </span></span></span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-meta z-class z-php"><span class="z-meta z-block z-php"></span></span><span class="z-meta z-class z-php"><span class="z-meta z-block z-php"><span class="z-punctuation z-section z-block z-php">}</span></span></span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>tm</span> <span class="z-keyword z-operator z-assignment z-php">=</span> <span class="z-keyword z-other z-new z-php">new</span> <span class="z-meta z-path z-php"><span class="z-support z-class z-php">TimeStruct</span></span><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span><span class="z-punctuation z-section z-group z-end z-php">)</span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>tm</span><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-other z-member z-php">tm_sec</span> <span class="z-keyword z-operator z-assignment z-php">=</span> <span class="z-constant z-numeric z-integer z-decimal z-php">0</span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>tm</span><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-other z-member z-php">tm_min</span> <span class="z-keyword z-operator z-assignment z-php">=</span> <span class="z-constant z-numeric z-integer z-decimal z-php">30</span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>tm</span><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-other z-member z-php">tm_hour</span> <span class="z-keyword z-operator z-assignment z-php">=</span> <span class="z-constant z-numeric z-integer z-decimal z-php">15</span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>tm</span><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-other z-member z-php">tm_mday</span> <span class="z-keyword z-operator z-assignment z-php">=</span> <span class="z-constant z-numeric z-integer z-decimal z-php">5</span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>tm</span><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-other z-member z-php">tm_mon</span> <span class="z-keyword z-operator z-assignment z-php">=</span> <span class="z-constant z-numeric z-integer z-decimal z-php">3</span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>tm</span><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-other z-member z-php">tm_year</span> <span class="z-keyword z-operator z-assignment z-php">=</span> <span class="z-constant z-numeric z-integer z-decimal z-php">115</span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>tm</span><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-other z-member z-php">tm_zone</span> <span class="z-keyword z-operator z-assignment z-php">=</span> <span class="z-string z-quoted z-double z-php"><span class="z-punctuation z-definition z-string z-begin z-php">&quot;</span></span><span class="z-string z-quoted z-double z-php"><span class="z-meta z-string-contents z-quoted z-double z-php"></span></span><span class="z-string z-quoted z-double z-php"><span class="z-meta z-string-contents z-quoted z-double z-php">BST</span><span class="z-punctuation z-definition z-string z-end z-php">&quot;</span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>lib</span> <span class="z-keyword z-operator z-assignment z-php">=</span> <span class="z-keyword z-other z-new z-php">new</span> <span class="z-meta z-path z-php"><span class="z-support z-class z-php">Library</span></span><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span><span class="z-punctuation z-section z-group z-end z-php">)</span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>asctime</span> <span class="z-keyword z-operator z-assignment z-php">=</span> <span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>lib</span><span class="z-meta z-function-call z-method z-php"><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-function z-php">bind</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-string z-quoted z-single z-php"><span class="z-punctuation z-definition z-string z-begin z-php">&#39;</span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php"></span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php">asctime</span><span class="z-punctuation z-definition z-string z-end z-php">&#39;</span></span><span class="z-punctuation z-separator z-php">,</span> <span class="z-meta z-array z-php"><span class="z-punctuation z-section z-array z-begin z-php">[</span> <span class="z-meta z-path z-php"><span class="z-support z-class z-php">TimeStruct</span></span><span class="z-punctuation z-accessor z-double-colon z-php">::</span><span class="z-constant z-class z-php">class</span> <span class="z-punctuation z-section z-array z-end z-php">]</span></span><span class="z-punctuation z-separator z-php">,</span> <span class="z-meta z-path z-php"><span class="z-support z-class z-php">Type</span></span><span class="z-punctuation z-accessor z-double-colon z-php">::</span><span class="z-constant z-other z-class z-php">TYPE_STRING</span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-meta z-function-call z-php"><span class="z-support z-function z-var z-php">var_dump</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-meta z-function-call z-invoke z-php"><span class="z-punctuation z-definition z-variable z-php">$</span><span class="z-variable z-other z-php">asctime</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>tm</span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span></code></pre> <p>We create a class called <code>TimeStruct</code> that extends <code>MFFI\Struct</code>. This has no methods other than a static one called <code>definition()</code>. This is called the first time one of these classes is instantiated, and returns an array containing the member names and their types. We then create one of these, and set the various parameters to a date. Finally, we bind to <code>libc</code>’s <code>asctime</code> function, which converts the time struct into an ASCII string. This looks like:</p> <pre class="z-code"><code><span class="z-text z-plain">michael@morbo:mffi% php7 test2.php </span><span class="z-text z-plain">string(25) &quot;Sun Apr 5 15:30:00 2015 </span><span class="z-text z-plain">&quot; </span><span class="z-text z-plain">michael@morbo:mffi% </span></code></pre> <p>It gets a spare trailing newline, but it looks pretty good on the whole.</p> <h2 id="binding-to-your-own-libraries">Binding to your own libraries<a class="zola-anchor" style="display: none;" href="#binding-to-your-own-libraries" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>You can also write your own libraries and use those. Here’s a bit of C code that provides a function that reverses strings. It’s trivial, and PHP already provides this, but it’s just an example. The <code>strrev</code> function in the code accepts a string (as a <code>char *</code>), and returns another string (again a <code>char *</code>).</p> <pre data-lang="c" class="language-c z-code"><code class="language-c" data-lang="c"><span class="z-source z-c"><span class="z-comment z-block z-c"><span class="z-punctuation z-definition z-comment z-c">/*</span> Taken from http://www8.cs.umu.se/~isak/snippets/strrev.c <span class="z-punctuation z-definition z-comment z-c">*/</span></span> </span><span class="z-source z-c"><span class="z-meta z-preprocessor z-include z-c"><span class="z-keyword z-control z-import z-include z-c">#include</span> <span class="z-string z-quoted z-other z-lt-gt z-include z-c"><span class="z-punctuation z-definition z-string z-begin z-c">&lt;</span>string.h<span class="z-punctuation z-definition z-string z-end z-c">&gt;</span></span> </span></span><span class="z-source z-c"> </span><span class="z-source z-c"><span class="z-storage z-type z-c">char</span> <span class="z-keyword z-operator z-c">*</span><span class="z-meta z-function z-c"><span class="z-entity z-name z-function z-c">strrev</span></span><span class="z-meta z-function z-parameters z-c"><span class="z-meta z-group z-c"><span class="z-punctuation z-section z-group z-begin z-c">(</span></span></span><span class="z-meta z-function z-parameters z-c"><span class="z-meta z-group z-c"><span class="z-storage z-type z-c">char</span> <span class="z-keyword z-operator z-c">*</span><span class="z-variable z-parameter z-c">str</span><span class="z-punctuation z-section z-group z-end z-c">)</span></span></span><span class="z-meta z-function z-c"> </span></span><span class="z-source z-c"><span class="z-meta z-function z-c"></span><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"><span class="z-punctuation z-section z-block z-begin z-c">{</span></span></span><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"> </span></span></span><span class="z-source z-c"><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"> <span class="z-storage z-type z-c">char</span> <span class="z-keyword z-operator z-c">*</span>p1<span class="z-punctuation z-separator z-c">,</span> <span class="z-keyword z-operator z-c">*</span>p2<span class="z-punctuation z-terminator z-c">;</span> </span></span></span><span class="z-source z-c"><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"> </span></span></span><span class="z-source z-c"><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"> <span class="z-keyword z-control z-c">if</span> <span class="z-meta z-group z-c"><span class="z-punctuation z-section z-group z-begin z-c">(</span><span class="z-keyword z-operator z-arithmetic z-c">!</span> str <span class="z-keyword z-operator z-arithmetic z-c">||</span> <span class="z-keyword z-operator z-arithmetic z-c">!</span> <span class="z-keyword z-operator z-c">*</span>str<span class="z-punctuation z-section z-group z-end z-c">)</span></span> </span></span></span><span class="z-source z-c"><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"> <span class="z-keyword z-control z-flow z-return z-c">return</span> str<span class="z-punctuation z-terminator z-c">;</span> </span></span></span><span class="z-source z-c"><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"> <span class="z-keyword z-control z-c">for</span> <span class="z-meta z-group z-c"><span class="z-punctuation z-section z-group z-begin z-c">(</span>p1 <span class="z-keyword z-operator z-assignment z-c">=</span> str<span class="z-punctuation z-separator z-c">,</span> p2 <span class="z-keyword z-operator z-assignment z-c">=</span> str <span class="z-keyword z-operator z-arithmetic z-c">+</span> <span class="z-meta z-function-call z-c"><span class="z-support z-function z-C99 z-c">strlen</span><span class="z-meta z-group z-c"><span class="z-punctuation z-section z-group z-begin z-c">(</span></span></span><span class="z-meta z-function-call z-c"><span class="z-meta z-group z-c">str</span></span><span class="z-meta z-function-call z-c"><span class="z-meta z-group z-c"><span class="z-punctuation z-section z-group z-end z-c">)</span></span></span> <span class="z-keyword z-operator z-arithmetic z-c">-</span> <span class="z-constant z-numeric z-integer z-decimal z-c">1</span><span class="z-punctuation z-terminator z-c">;</span> p2 <span class="z-keyword z-operator z-comparison z-c">&gt;</span> p1<span class="z-punctuation z-terminator z-c">;</span> <span class="z-keyword z-operator z-arithmetic z-c">++</span>p1<span class="z-punctuation z-separator z-c">,</span> <span class="z-keyword z-operator z-arithmetic z-c">--</span>p2<span class="z-punctuation z-section z-group z-end z-c">)</span></span> </span></span></span><span class="z-source z-c"><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"> <span class="z-meta z-block z-c"><span class="z-punctuation z-section z-block z-begin z-c">{</span> </span></span></span></span><span class="z-source z-c"><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"><span class="z-meta z-block z-c"> <span class="z-keyword z-operator z-c">*</span>p1 <span class="z-keyword z-operator z-assignment z-augmented z-c">^=</span> <span class="z-keyword z-operator z-c">*</span>p2<span class="z-punctuation z-terminator z-c">;</span> </span></span></span></span><span class="z-source z-c"><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"><span class="z-meta z-block z-c"> <span class="z-keyword z-operator z-c">*</span>p2 <span class="z-keyword z-operator z-assignment z-augmented z-c">^=</span> <span class="z-keyword z-operator z-c">*</span>p1<span class="z-punctuation z-terminator z-c">;</span> </span></span></span></span><span class="z-source z-c"><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"><span class="z-meta z-block z-c"> <span class="z-keyword z-operator z-c">*</span>p1 <span class="z-keyword z-operator z-assignment z-augmented z-c">^=</span> <span class="z-keyword z-operator z-c">*</span>p2<span class="z-punctuation z-terminator z-c">;</span> </span></span></span></span><span class="z-source z-c"><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"><span class="z-meta z-block z-c"> <span class="z-punctuation z-section z-block z-end z-c">}</span></span> </span></span></span><span class="z-source z-c"><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"> <span class="z-keyword z-control z-flow z-return z-c">return</span> str<span class="z-punctuation z-terminator z-c">;</span> </span></span></span><span class="z-source z-c"><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"></span></span><span class="z-meta z-function z-c"><span class="z-meta z-block z-c"><span class="z-punctuation z-section z-block z-end z-c">}</span></span></span> </span></code></pre> <p>Save this as <code>rev.c</code>, and compile using a command like <code>gcc -o rev.so -g -fPIC -shared rev.c</code>. You can then use the following PHP code to access the function:</p> <pre data-lang="php" class="language-php z-code"><code class="language-php" data-lang="php"><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-punctuation z-section z-embedded z-begin z-php">&lt;?php</span></span><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>library</span> <span class="z-keyword z-operator z-assignment z-php">=</span> <span class="z-keyword z-other z-new z-php">new</span> <span class="z-meta z-path z-php"><span class="z-support z-other z-namespace z-php">MFFI</span><span class="z-punctuation z-separator z-namespace z-php">\</span><span class="z-support z-class z-php">Library</span></span><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span><span class="z-string z-quoted z-single z-php"><span class="z-punctuation z-definition z-string z-begin z-php">&#39;</span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php"></span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php">rev.so</span><span class="z-punctuation z-definition z-string z-end z-php">&#39;</span></span><span class="z-punctuation z-section z-group z-end z-php">)</span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>strrev</span> <span class="z-keyword z-operator z-assignment z-php">=</span> <span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>library</span><span class="z-meta z-function-call z-method z-php"><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-function z-php">bind</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-string z-quoted z-single z-php"><span class="z-punctuation z-definition z-string z-begin z-php">&#39;</span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php"></span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php">strrev</span><span class="z-punctuation z-definition z-string z-end z-php">&#39;</span></span><span class="z-punctuation z-separator z-php">,</span> <span class="z-meta z-array z-php"><span class="z-punctuation z-section z-array z-begin z-php">[</span> <span class="z-meta z-path z-php"><span class="z-support z-other z-namespace z-php">MFFI</span><span class="z-punctuation z-separator z-namespace z-php">\</span><span class="z-support z-class z-php">Type</span></span><span class="z-punctuation z-accessor z-double-colon z-php">::</span><span class="z-constant z-other z-class z-php">TYPE_STRING</span> <span class="z-punctuation z-section z-array z-end z-php">]</span></span><span class="z-punctuation z-separator z-php">,</span> <span class="z-meta z-path z-php"><span class="z-support z-other z-namespace z-php">MFFI</span><span class="z-punctuation z-separator z-namespace z-php">\</span><span class="z-support z-class z-php">Type</span></span><span class="z-punctuation z-accessor z-double-colon z-php">::</span><span class="z-constant z-other z-class z-php">TYPE_STRING</span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-support z-function z-construct z-php">echo</span> <span class="z-meta z-function-call z-invoke z-php"><span class="z-punctuation z-definition z-variable z-php">$</span><span class="z-variable z-other z-php">strrev</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-string z-quoted z-single z-php"><span class="z-punctuation z-definition z-string z-begin z-php">&#39;</span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php"></span></span><span class="z-string z-quoted z-single z-php"><span class="z-meta z-string-contents z-quoted z-single z-php">Hello world</span><span class="z-punctuation z-definition z-string z-end z-php">&#39;</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span></code></pre> <p>The result seems to make sense:</p> <pre class="z-code"><code><span class="z-text z-plain">michael@morbo:mffi% php7 test3.php </span><span class="z-text z-plain">dlrow olleH </span><span class="z-text z-plain">michael@morbo:mffi% </span></code></pre> <h2 id="a-word-of-warning">A word of warning<a class="zola-anchor" style="display: none;" href="#a-word-of-warning" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>This is probably a very, very bad idea. The potential for security problems is monumental. If you get a definition even slightly wrong, you’ll either crash PHP straight away, which is probably the best result, or corrupt some memory somewhere and have unseen and weird effects. I would never advise anyone to use it unless it was a one-shot script or something running in a very controlled, non-public facing environment. If you use this, and it breaks, you get to keep all the bits!</p> <h2 id="plans">Plans<a class="zola-anchor" style="display: none;" href="#plans" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>This extension is only for PHP 7. It doesn’t work on PHP 5, and I don’t intend to change that. There’s a fairly big TODO list of features I want to add. I want to make it possible to define structs that contain other structs, which is a work in progress, and then handle large arrays of scalar types. This would make it possible to solve the Mandelbrot problem from the previous post fairly efficiently using a small C library that did the maths, without knowing anything about how to write PHP extensions.</p> <p>I have not done any benchmarking yet. The translation through libffi will have a cost, but I have not yet worked out what that will be.</p> 2015-02-23T00:00:00+00:00 2015-02-23T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/php-at-the-speed-of-c/ <p>For common cases, most dynamic languages are “fast enough”. For most PHP applications, the most overhead is probably hitting network services like the database, and this accounts for far more of the run time than the speed of the language’s function calls. Also, PHP is a fairly thin glue layer on top of a lot of C libraries, which mean that for any significant algorithmic work, you’re probably already calling out to a C library underneath. If this wasn’t the case, basic operations like string searches and regular expressions would be extremely slow.</p> <p>However, every now and again you do want to do something with PHP that does involve complex algorithms, for which you might not have a C library to hand. Even if you do have a C library or the ability to create one, writing extensions for PHP to expose this functionality has a bit of a learning curve. Wouldn’t it be nice if we could write code in PHP, and have it accelerated somehow to C-like speeds?</p> <h2 id="the-mandelbrot-set">The Mandelbrot set<a class="zola-anchor" style="display: none;" href="#the-mandelbrot-set" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p><a href="http://en.wikipedia.org/wiki/Mandelbrot_set">The Mandelbrot set</a> is a set of complex numbers generated using an algorithm that involves a large number of iterations. When plotted with an appropriate colouring scheme, they make fractals which can look very nice. Plotting them is relatively simple, though it’s quite CPU intensive. I have written some fairly naïve code to generate these images, which you can find <a href="https://gist.github.com/mgdm/39e62755c586dc0c8907#file-mandelbrot-php">in this gist</a>. The output looks like this:</p> <p><img src="/assets/img/posts/mandel.png" alt="Mandelbrot set" /></p> <p>There’s a fair bit of code there, but the function that takes most of the time is the <code>mandelbrot()</code> one. It’s where all the iteration happens. On the VM I’m testing this in^{<a name="fn1" href="#vm">1</a>} it takes around 80 seconds to run, under a release build of PHP 5.6.5:</p> <pre class="z-code"><code><span class="z-text z-plain">vagrant@vagrant-ubuntu-trusty-64:~/src$ php colourmandel.php </span><span class="z-text z-plain">Data took 78.744249 </span><span class="z-text z-plain">Render took 0.631219 </span></code></pre> <p>That’s pretty slow, by any metric. It’s iterating up to 4096 times for every pixel in an 800x512 image, so unlike most PHP apps it’s going to be CPU-bound rather than dependent on the network or other services. I wonder if there’s a way to speed it up, without having to drop down to pure C?</p> <h2 id="hhvm">HHVM<a class="zola-anchor" style="display: none;" href="#hhvm" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>Well, one way is to install HHVM:</p> <pre class="z-code"><code><span class="z-text z-plain">vagrant@vagrant-ubuntu-trusty-64:~/src$ hhvm mandelbrot.php </span><span class="z-text z-plain">Data took 5.852351 </span><span class="z-text z-plain">Render took 0.236238 </span></code></pre> <p>HHVM, in its default setup, features a Just-In-Time (JIT) compiler, that traces every code path that is executed. If it finds a path that is “hot” – executed a large number of times – it will jump in and compile that code branch down to straight machine code and execute that instead. As this code is essentially arithmetic taking place in a tight loop, it’s a perfect case for HHVM’s JIT compiler to jump in and optimize.</p> <p>If you’ve got complete control over your infrastructure and don’t mind changing PHP implementations, HHVM could well be the way to go. It also has a number of other extras which may be of interest. But what if you don’t?</p> <h2 id="recki-compiler-toolkit">Recki Compiler Toolkit<a class="zola-anchor" style="display: none;" href="#recki-compiler-toolkit" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>Recently, Anthony Ferrara (known throughout the Internet and beyond as <a href="https://twitter.com/ircmaxell">@ircmaxell</a>) and Joe Watkins (similarly well-known as <a href="https://twitter.com/krakjoe">@krakjoe</a>) have been working on a new set of toys for solving this problem while staying on the “standard” PHP runtime. <a href="https://github.com/google/recki-ct">Recki-CT</a> is a set of tools that implement a PHP compiler, in PHP. While this might you think of things like <a href="http://pypy.org">PyPy</a>, which implements a Python virtual machine in Python, this is not Recki’s goal – it doesn’t provide a VM, so it can’t run PHP by itself. However, it can parse PHP code and generate other code from it.</p> <p>Recki uses the well-known <a href="https://github.com/nikic/PHP-Parser">PHP-Parser library</a> by <a href="http://nikic.github.io/">Nikita Popov</a> to generate a graph-based representation of the code, and convert it to an <em>intermediate representation</em>. To get here involves a few steps, which are described in <a href="https://github.com/google/recki-ct/blob/master/doc/2_basic_operation.md">Recki’s documentation</a>, but essentially:</p> <ul> <li>It generates a tree-based representation of the code called an <a href="http://en.wikipedia.org/wiki/Abstract_syntax_tree">abstract syntax tree</a></li> <li>From the AST, it generates a <a href="http://en.wikipedia.org/wiki/Control_flow_graph">control flow graph</a></li> <li>It then converts this graph into <a href="http://en.wikipedia.org/wiki/Static_single_assignment_form">static single assignment form</a>, where every variable is only assigned once, and all are defined before use. This makes it simpler to optimize.</li> <li>Next, it repeatedly runs optimizations on this graph</li> </ul> <p>This intermediate form is pretty low-level, and it is comparatively simple to generate code from it for a variety of targets. One of the targets Recki can use is a second component, JitFu, which is a PHP extension allowing us to generate machine code at run time.</p> <h3 id="first-steps-with-recki-and-jitfu">First steps with Recki and JitFu<a class="zola-anchor" style="display: none;" href="#first-steps-with-recki-and-jitfu" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <p>So, I’ve decided I’m not yet ready to throw out all my existing infrastructure, and that (perhaps) I’m tied to some extensions that currently only work on main-line PHP. Where do I start?</p> <p>First off, I need to follow the instructions on Recki’s README, which include installing <a href="https://www.gnu.org/software/libjit/">libjit</a> and the <a href="https://github.com/krakjoe/jitfu">JitFu extension</a>. Then, we need to install Recki using Composer, and then add Composer’s autoloader to the top of the PHP file as normal. Once we have these in place, we can start to look at how to use JitFu to speed our code up.</p> <pre class="z-code"><code><span class="z-text z-plain">composer require recki-ct/recki-ct </span></code></pre> <p>If you have a look at the <code>mandelbrot()</code> function in the file, we can see it does all the work itself. For each pixel in the image, it iterates over a mathematical function until either the absolute value of the complex number “escapes” (tends towards infinity), or it hits the maximum number of iterations that we’ve previously defined to be 4096.</p> <p>For each row of pixels in the image it will create an array containing the number of iterations for each pixel. Then, it creates another array representing columns, which contains the list of rows. Finally, the <code>render()</code> function is used to convert the number of iterations into a colour, but we can ignore that for now as it’s fairly quick by comparison.</p> <p>We’ll change the <code>main()</code> function so that it will attempt to precompile the <code>mandelbrot()</code> function, and then execute that. Also, we’ll need to tell Recki what the expected input and output types are for our function so that it knows what it’s dealing with. We do this using a docblock, in the same way that we would if we were using phpDocumentor or similar. You can see this version of the code <a href="https://gist.github.com/mgdm/39e62755c586dc0c8907#file-mandelbrotwithrecki-php">here</a>.</p> <pre data-lang="php" class="language-php z-code"><code class="language-php" data-lang="php"><span class="z-embedding z-php"><span class="z-text z-html z-basic">/** </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> * @param double $xMin </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> * @param double $xMax </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> * @param double $yMin </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> * @param double $yMax </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> * @param int $xPx </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> * @param int $yPx </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> * @return array </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> */ </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> function mandelbrot($xMin, $xMax, $yMin, $yMax, $xPx, $yPx) </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> // ... </span></span></code></pre> <p>And now the changes to the main function – these replace lines 100 to 104:</p> <pre data-lang="php" class="language-php z-code"><code class="language-php" data-lang="php"><span class="z-embedding z-php"><span class="z-text z-html z-basic"> $start = microtime(true); </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> $function = ReckiCT\Jit::jitfu(&#39;mandelbrot&#39;); </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> printf(&quot;Compiling took %F\n&quot;, $mandelTime); </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> $start = microtime(true); </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> $data = $function($xMin, $xMax, $yMin, $yMax, $xPx, $yPx); </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> $mandelTime = microtime(true) - $start; </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> printf(&quot;Data took %F\n&quot;, $mandelTime); </span></span></code></pre> <p>Then we try and run it:</p> <pre class="z-code"><code><span class="z-text z-plain">vagrant@vagrant-ubuntu-trusty-64:~/src/recki-ct$ php mandelbrotWithRecki.php </span><span class="z-text z-plain"> </span><span class="z-text z-plain">Fatal error: Uncaught exception &#39;LogicException&#39; with message &#39;Found node without parser rule: Expr_Array&#39; in /home/vagrant/src/recki-ct/lib/ReckiCT/Parser/Parser.php:112 </span><span class="z-text z-plain">Stack trace: </span><span class="z-text z-plain">#0 /home/vagrant/src/recki-ct/lib/ReckiCT/Parser/Rules/Assign.php(41): ReckiCT\Parser\Parser-&gt;parseNode(Object(PhpParser\Node\Expr\Array_), Object(ReckiCT\Parser\State)) </span><span class="z-text z-plain">#1 /home/vagrant/src/recki-ct/lib/ReckiCT/Parser/Parser.php(109): ReckiCT\Parser\Rules\Assign-&gt;parse(Object(PhpParser\Node\Expr\Assign), Object(ReckiCT\Parser\State)) </span><span class="z-text z-plain">#2 /home/vagrant/src/recki-ct/lib/ReckiCT/Parser/Parser.php(96): ReckiCT\Parser\Parser-&gt;parseNode(Object(PhpParser\Node\Expr\Assign), Object(ReckiCT\Parser\State)) </span><span class="z-text z-plain">#3 /home/vagrant/src/recki-ct/lib/ReckiCT/Parser/Parser.php(86): ReckiCT\Parser\Parser-&gt;parseStmtList(Array, Object(ReckiCT\Parser\State)) </span><span class="z-text z-plain">#4 /home/vagrant/src/recki-ct/lib/ReckiCT/Jit.php(222): ReckiCT\Parser\Parser-&gt;parseFunction(Object(PhpParser\Node\Stmt\Function_)) </span><span class="z-text z-plain">#5 /home/vagrant/src/recki-ct/lib/ReckiCT/Jit.php(240): ReckiCT\Jit-&gt;getFunctionG in /home/vagrant/src/recki-ct/lib/ReckiCT/Parser/Parser.php on line 112 </span></code></pre> <p>Oh. The problem here is that Recki is still pretty young (it’s only around 6 months since its first release), and it currently doesn’t handle arrays. That’s OK, we can refactor the code a little, and break out the part of the iteration that returns a number into a separate function, which looks like the one below. Note that Recki doesn’t currently do constant lookups either, so we pass the maximum number of iterations as a parameter.</p> <pre data-lang="php" class="language-php z-code"><code class="language-php" data-lang="php"><span class="z-embedding z-php"><span class="z-text z-html z-basic">/** </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> * @param double $cReal </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> * @param double $cImag </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> * @param int $maxIterations </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> * @return int </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> */ </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic">function mandelbrotIteration($cReal, $cImag, $maxIterations) </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic">{ </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> $originalReal = $cReal; </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> $originalImag = $cImag; </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> $ni = 0; </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> while ($ni &lt; $maxIterations &amp;&amp; ($cReal * $cReal * $cImag * $cImag &lt; 4.0)) { </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> $tmp = $cReal * $cReal - $cImag * $cImag + $originalReal; </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> $cImag = $cImag * $cReal * 2 + $originalImag; </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> $cReal = $tmp; </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> $ni++; </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> } </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> if ($ni == $maxIterations) { </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> $ni = 0; </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> } </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> return $ni; </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic">} </span></span></code></pre> <p>The new version of the file is <a href="https://gist.github.com/mgdm/39e62755c586dc0c8907#file-mandelbrotafterrefactor-php">here</a>. Now, let’s try it:</p> <pre class="z-code"><code><span class="z-text z-plain">vagrant@vagrant-ubuntu-trusty-64:~/src/recki-ct$ php mandelbrotAfterRefactor.php </span><span class="z-text z-plain">Compiling took 1.537641 </span><span class="z-text z-plain">Data took 3.724597 </span><span class="z-text z-plain">Render took 0.595848 </span></code></pre> <p>Not bad! It took about 1.5 seconds to compile, and then ran in 3.7 - somewhere around a 2100% improvement! We can do a little better too, as we can grab the intermediate representation that JitFu has made, and cache it.</p> <pre data-lang="php" class="language-php z-code"><code class="language-php" data-lang="php"><span class="z-embedding z-php"><span class="z-text z-html z-basic"> if (file_exists(&#39;mandelbrotIteration.ir&#39;)) { </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> $start = microtime(true); </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> $ir = file_get_contents(&#39;mandelbrotIteration.ir&#39;); </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> $iterationFunc = ReckiCT\Jit::getInstance()-&gt;compileIrJitFu($ir, &#39;fibo&#39;); </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> $loadTime = microtime(true) - $start; </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> printf(&quot;Loading took %F\n&quot;, $loadTime); </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> } else { </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> $start = microtime(true); </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> $iterationFunc = ReckiCT\Jit::jitfu(&#39;mandelbrotIteration&#39;); </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> $compileTime = microtime(true) - $start; </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> printf(&quot;Compiling took %F\n&quot;, $compileTime); </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> file_put_contents(&#39;mandelbrotIteration.ir&#39;, </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> ReckiCT\Jit::getInstance()-&gt;getFunctionIr(&#39;mandelbrotIteration&#39;)); </span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"> } </span></span></code></pre> <p>Follow along with the code <a href="https://gist.github.com/mgdm/39e62755c586dc0c8907#file-mandelbrotwithcachedir-php">here</a>. Now let’s see how that looks:</p> <pre class="z-code"><code><span class="z-text z-plain">vagrant@vagrant-ubuntu-trusty-64:~/src/recki-ct$ php mandelbrotWithCachedIR.php </span><span class="z-text z-plain">Loading took 0.177504 </span><span class="z-text z-plain">Data took 3.636709 </span><span class="z-text z-plain">Render took 0.653733 </span></code></pre> <p>Nice.</p> <h3 id="the-next-step-with-recki">The next step with Recki<a class="zola-anchor" style="display: none;" href="#the-next-step-with-recki" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <p>Recki has another trick up its sleeve that we’ve not seen yet. Earlier, I hinted that it can generate code for multiple targets. One of these targets happens to be C code, wrapped in a PHP extension. This might be an interesting option for people who are a little wary of running code like JitFu on their production servers. Also, it means we can throw all of the knowledge and tricks contained within our C compiler at the code as well, to get as much speed as possible.</p> <p>To demonstrate this, let’s lift the <code>mandelbrotIteration()</code> function right out of the code, and put it in its own file. I’ll call it <code>iteration.php</code>, and you can see it <a href="https://gist.github.com/mgdm/39e62755c586dc0c8907#file-iteration-php">here</a>. I can then run this command, assuming I’m in the Recki-CT root directory:</p> <pre class="z-code"><code><span class="z-text z-plain">php bin/compile_pecl compile -O mandelbrot iteration.php </span></code></pre> <p>I’m passing the <code>-O</code> option, which passes <code>-O3</code> to the C compiler to dial up its own optimizations. You should see some familiar text scrolling past as Recki generates, configures, and compiles a PECL extension for us. Next, we can return to our Mandelbrot code rip out all the additions we put in for our JitFu experiments, and indeed remove the <code>mandelbrotIteration()</code> function entirely. I’ve chosen to wrap it inside a <code>function_exists()</code> check instead, so that I can test with or without the extension loaded to prove I get the same output. You can get that version <a href="https://gist.github.com/mgdm/39e62755c586dc0c8907#file-mandelbrotwithextension-php">here</a>.</p> <pre class="z-code"><code><span class="z-text z-plain">vagrant@vagrant-ubuntu-trusty-64:~/src/recki-ct$ php -dextension=./mandelbrot.so mandelbrotWithExtension.php </span><span class="z-text z-plain">Data took 2.411557 </span><span class="z-text z-plain">Render took 0.637504 </span></code></pre> <p>Now we’re down to 2.4 seconds. Not bad at all. Remember, we’re only optimizing the innermost loop here – once Recki is able to optimize functions using arrays, we could potentially speed it up even further.</p> <h2 id="summary">Summary<a class="zola-anchor" style="display: none;" href="#summary" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>Recki-CT is a very young project, so this is not the kind of thing I’d be looking to use in production quite yet, but it’s certainly something to keep an eye on. Particularly where we have certain functions within our applications that are very CPU-heavy, we might well be able to take a considerable amount of load off by optimizing just those functions and leaving the rest of our app stack as standard. As mentioned before, PHP has been talked about in the past as being a glue layer between C libraries and application code. If we can write PHP and generate optimized C code from that, then the barrier to entry for taking advantage of this is considerably lower.</p> <h3 id="update">Update<a class="zola-anchor" style="display: none;" href="#update" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <p>For comparison, I translated the code to straight C reasonably directly, which you can see <a href="https://gist.github.com/mgdm/31c8ceb27802d2ef6606">here</a>. It’s a standalone C program, not a PHP extension. It runs in around 2.1 seconds on the same VM:</p> <pre class="z-code"><code><span class="z-text z-plain">vagrant@vagrant-ubuntu-trusty-64:~/src/Mandelbrot-browser$ ./basic-mandelbrot </span><span class="z-text z-plain">Data took 2.107163 </span><span class="z-text z-plain">Render took 0.000840 </span></code></pre> <hr /> <p><a name="vm" href="#fn1">1.</a> I realise that VMs don't give a terribly accurate benchmark, but these tests deal with orders of magnitude, so they'll reflect the speed difference enough to be getting on with.</p> 2014-04-02T00:00:00+00:00 2014-04-02T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/vagrantfile-inheritance/ <p>I’ve recently been using a lot of Vagrant machines at work, and needed to come up with a way to permit people to customise settings such as the developer’s own DNS zone, the amount of RAM and number of vCPUs assigned to each machine, etc.</p> <p>In the <a href="http://docs.vagrantup.com/v2/vagrantfile/index.html">Vagrantfile reference</a> I noticed that Vagrant will merge several Vagrantfiles together on launch.</p> <ul> <li>First, the Vagrantfile packaged in the box</li> <li>The Vagrantfile in <code>~/.vagrant.d/Vagrantfile</code></li> <li>The Vagrantfile in the project directory</li> </ul> <p>I knew that each Vagrantfile is just a Ruby script, so I decided to try adding a global variable in <code>~/.vagrant.d/Vagrantfile</code> to see if it was inherited.</p> <pre data-lang="ruby" class="language-ruby z-code"><code class="language-ruby" data-lang="ruby"><span class="z-source z-ruby"><span class="z-variable z-other z-readwrite z-global z-ruby"><span class="z-punctuation z-definition z-variable z-ruby">$</span>global_vm_memory</span> <span class="z-keyword z-operator z-assignment z-ruby">=</span> <span class="z-constant z-numeric z-integer z-decimal z-ruby">1024</span> </span><span class="z-source z-ruby"><span class="z-variable z-other z-readwrite z-global z-ruby"><span class="z-punctuation z-definition z-variable z-ruby">$</span>global_vm_cpus</span> <span class="z-keyword z-operator z-assignment z-ruby">=</span> <span class="z-constant z-numeric z-integer z-decimal z-ruby">4</span> </span><span class="z-source z-ruby"><span class="z-variable z-other z-readwrite z-global z-ruby"><span class="z-punctuation z-definition z-variable z-ruby">$</span>global_vm_tld</span> <span class="z-keyword z-operator z-assignment z-ruby">=</span> <span class="z-meta z-string z-ruby"><span class="z-string z-quoted z-single z-ruby"><span class="z-punctuation z-definition z-string z-begin z-ruby">&#39;</span>.mgdm<span class="z-punctuation z-definition z-string z-end z-ruby">&#39;</span></span></span> </span></code></pre> <p>Then, in the project’s Vagrantfile I added:</p> <pre data-lang="ruby" class="language-ruby z-code"><code class="language-ruby" data-lang="ruby"><span class="z-source z-ruby">config<span class="z-punctuation z-accessor z-dot z-ruby">.</span>vm<span class="z-punctuation z-accessor z-dot z-ruby">.</span>provider <span class="z-meta z-string z-ruby"><span class="z-string z-quoted z-double z-ruby"><span class="z-punctuation z-definition z-string z-begin z-ruby">&quot;</span>virtualbox<span class="z-punctuation z-definition z-string z-end z-ruby">&quot;</span></span></span> <span class="z-keyword z-control z-start-block z-ruby">do</span> <span class="z-meta z-block z-parameters z-ruby"><span class="z-punctuation z-definition z-parameters z-begin z-ruby">|</span></span><span class="z-meta z-block z-parameters z-ruby"><span class="z-variable z-parameter z-ruby">v</span><span class="z-meta z-block z-parameters z-ruby"><span class="z-punctuation z-definition z-parameters z-end z-ruby">|</span></span></span> </span><span class="z-source z-ruby"> v<span class="z-punctuation z-accessor z-dot z-ruby">.</span>memory <span class="z-keyword z-operator z-assignment z-ruby">=</span> <span class="z-variable z-other z-readwrite z-global z-ruby"><span class="z-punctuation z-definition z-variable z-ruby">$</span>global_vm_memory</span> <span class="z-keyword z-operator z-assignment z-augmented z-ruby">||=</span> <span class="z-constant z-numeric z-integer z-decimal z-ruby">512</span> </span><span class="z-source z-ruby"> v<span class="z-punctuation z-accessor z-dot z-ruby">.</span>cpus <span class="z-keyword z-operator z-assignment z-ruby">=</span> <span class="z-variable z-other z-readwrite z-global z-ruby"><span class="z-punctuation z-definition z-variable z-ruby">$</span>global_vm_cpus</span> <span class="z-keyword z-operator z-assignment z-augmented z-ruby">||=</span> <span class="z-constant z-numeric z-integer z-decimal z-ruby">2</span> </span><span class="z-source z-ruby"><span class="z-keyword z-control z-ruby">end</span> </span></code></pre> <p>That worked fine. I’m pretty sure it’s not the cleanest way, but it certainly is simple.</p> 2013-05-19T00:00:00+00:00 2013-05-19T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/zsh-antigen/ <p>I’ve been using <a href="http://www.zsh.org">zsh</a> for a while, with <a href="https://github.com/robbyrussell/oh-my-zsh">oh-my-zsh</a> to add a pile of new features to it. Oh-my-zsh is great, and has a lot of useful things in it, but I found it a bit of a pain to keep my fork of it in sync with the main repository and another fork that I’d stolen things from. The fact that it was one giant repository with everything in there didn’t seem quite right to me. Then, I came across <a href="https://github.com/zsh-users/antigen">zsh-antigen</a>, which promises to be to zsh what <a href="https://github.com/tpope/vim-pathogen">Pathogen</a> is to Vim. That sounded good to me, so I gave it a try.</p> <p>The main difference is that instead of having one repository with all the plugins, it lets you load plugins from pretty much any Git repository. It includes a special case for the oh-my-zsh repository, so you can load plugins and themes from there by default. I’d recommend reading the <a href="https://github.com/zsh-users/antigen">README in the Antigen repository</a> to get the full story, but I’ll give a quick overview of how I’ve got mine set up here.</p> <p>First off, I created my own Git repository, and added the Antigen repository as a submodule. I’ve called mine <code>.zsh-antigen</code>, and it lives in my home directory. It’s on <a href="http://github.com/mgdm/zsh-antigen">Github</a> if you want to clone or fork it.</p> <script src="https://gist.github.com/mgdm/cf14dc0aad25e468411f.js"> </script> <p>Next I created my <code>zshrc</code> file, at <code>~/.zsh-antigen/zshrc</code>. I symlink it into my home directory to <code>~/.zshrc</code>.</p> <script src="https://gist.github.com/mgdm/5608243.js"> </script> <p>On lines 1 and 2, I set up a variable to refer to the antigen directory, and then load the antigen script itself.</p> <p>On line 4, we tell antigen to use oh-my-zsh by default. That way, we can load plugins from there with the <code>antigen-bundle</code> command. I use a few, the vagrant and git plugins on all my machines, and I detect Mac OS X and load the osx plugin if I’m on a Mac.</p> <p>I use a couple of bundles that don’t belong to oh-my-zsh, too – these are the <code>zsh-syntax-highlighting</code> bundle from zsh-users, which adds nice syntax highlighting to the command line, and the <code>zsh-history-substring-search</code> bundle that provides history search when you type part of a command and press the up cursor key. As you can see, if they’re hosted on Github, it’s as easy as specifying the repos in a “username/repo-name” format.</p> <p>On line 14, I load a bundle which is local to my config. (It just provides some path detection to tell me what app I’m in when editing files on my work computer. These appear in the right prompt.) You can add as many bundles and as you like, without them having to be part of oh-my-zsh.</p> <p>Next, on line 16 I apply a theme. This is a theme in the same format as oh-my-zsh, and just specifies what appears in the left prompt (<code>PS1</code>, as in Bash and similar shells) and <code>RPS1</code> (which is right-aligned and doesn’t normally exist in Bash).</p> <p>We then call <code>antigen-apply</code> last of all, which applies all the previous configuration. When <code>antigen-apply</code> is called, it will ensure that it has all the bundles and themes that have been requested. If not, it will go off and download them and cache them locally. This means that the first time you load antigen in a new shell, it’ll wait a few seconds while it goes and grabs the oh-my-zsh code along with any other bundles you’ve requested.</p> <p>Underneath this, I place all my useful environment variables, aliases, and install things like rbenv, the same way as I used to do in Bash. I find this way of working to be quite a bit cleaner than oh-my-zsh - the <code>.zshrc</code> I ended up with in there had quite a lot going on, along with various ‘magic’ things that I wasn’t keen on. The automatic updating really got on my nerves, even though it’s supposed to be convenient. Feel free to grab <a href="https://github.com/mgdm/zsh-antigen">my repo</a> and use it as a starting point if it’s helpful.</p> 2012-11-19T00:00:00+00:00 2012-11-19T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/fosdem-2013/ <p>I’ve been intending to go to FOSDEM for a few years now, but so far I’ve never quite got around to making it. This year, though, <a href="http://lornajane.net">Lorna</a> asked me if I was up for helping to run the PHP devroom. I said yes, and so this year I will be there!</p> <p>There are calls for papers open for the main track and lightning talks, which you can find on the <a href="http://fosdem.org">FOSDEM site</a>. We’d also like to see submissions for talks in the PHP room–head on over to the <a href="http://lrnja.net/W1A2Nk">submission form</a> to post your talks. This year, we’re going to try an anonymized process to choose the talks. All we’ll have to go on will be the talk title, abstract and length; no personally identifiable information will be there. As the form says, we don’t care who you are, we just want great talks.</p> <p>FOSDEM is a huge, community-run event that takes place in Brussels every year. Attendance and participation is completely free to anyone. This does mean that we can’t cover costs for accommodation or getting to and from Brussels, but we hope that doesn’t put too many people off. As well as the PHP room, there will be a lot more going on in other tracks and rooms around the event, so there’ll be plenty of things to see and do elsewhere. Hopefully I’ll see some of you there.</p> 2012-09-14T00:00:00+00:00 2012-09-14T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/why-one-line-installers-are-bad/ <p>There has been a trend in the last while for various bits of useful software to have a one-line shell command recommended as the installation method. The usual form of this is to pipe something like <code>curl</code> or <code>wget</code> to some interpreter, be it <code>bash</code>, <code>php</code>, <code>ruby</code>, or some such. They look like this (taken from <a href="https://rvm.io/rvm/install/">RVM’s installation page</a>):</p> <pre class="z-code"><code><span class="z-text z-plain">curl -L https://get.rvm.io | bash -s stable --ruby </span></code></pre> <p>RVM is by no means the first bit of software to do this–I remember at various stages PEAR, Homebrew, and others suggesting the same thing. This command takes the output of <code>curl</code> and pipes it straight to <code>bash</code>. I have several issues with this.</p> <h2 id="it-gives-you-no-opportunity-to-inspect-the-code-that-you-are-running">It gives you no opportunity to inspect the code that you are running<a class="zola-anchor" style="display: none;" href="#it-gives-you-no-opportunity-to-inspect-the-code-that-you-are-running" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>In this example, you are blindly assuming that the code you get from http://get.rvm.io is actually the RVM installer. There is no verification of any form - if, for example, the host has been compromised, the installer might have been replaced with something as simple as an <code>rm -rf .</code>, which would blow away anything in the current directory. Or, indeed, it could any other command that you have permission to run as the current user. It might not even be intentionally malicious–there could just be a bug which has some <a href="http://pcgamingwiki.com/wiki/Half-Life#Severe_System_Crippling_and_Data_Destroying_Uninstall_Bug">unexpected side-effect</a>.</p> <h2 id="there-is-no-way-to-verify-the-source-of-the-code">There is no way to verify the source of the code<a class="zola-anchor" style="display: none;" href="#there-is-no-way-to-verify-the-source-of-the-code" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>It could be argued that because the code is delivered over HTTPS in this case, there is some verification of the identity of the source. This is not enough, as can be demonstrated by the case above, where the source is compromised. With packaging systems such as <code>dpkg</code> or <code>rpm</code>, there are cryptographic signatures attached to each package. These sign each release of the package in question, and hence there is a trust layer you can use to help decide if you wish to use the package source. Also, it can then flag up when the source may have been compromised.</p> <p>You can, of course, override the signature verification step in a package manager along the lines of <code>dpkg</code> or <code>rpm</code> or many others that I have not used; however, at that point, you are making the decision to do so yourself and therefore take responsibility for the results.</p> <p>In the case of PHP’s Composer, which I’ll come back to, <a href="http://twitter.com/padraicb">Pádraic Brady</a> <a href="https://twitter.com/padraicb/status/246305589964181504">pointed out</a> that the Composer people are aware of the issue of running arbitrary unsigned code, and a <a href="https://github.com/composer/composer/pull/1092">pull request has been filed</a> to fix part of it. Although I agree with many of the changes made, I don’t believe it does anything to protect the installer itself. (I may be wrong here–if I have misread, please let me know.)</p> <h2 id="it-teaches-bad-habits">It teaches bad habits<a class="zola-anchor" style="display: none;" href="#it-teaches-bad-habits" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>The example that prompted this blog post is that of the PHP utility, <a href="http://getcomposer.org">Composer</a>. This is a wonderful piece of software for managing library dependencies within PHP projects. However, on their <a href="http://getcomposer.org/download/">download page</a>, they have two variants of the one-line installer, the first being along the pattern already discussed:</p> <pre class="z-code"><code><span class="z-text z-plain">curl -s https://getcomposer.org/installer | php </span></code></pre> <p>And the second, for “if you don’t have <code>curl</code>”:</p> <pre class="z-code"><code><span class="z-text z-plain">php -r &quot;eval(&#39;?&gt;&#39;. </span><span class="z-text z-plain">file_get_contents(&#39;https://getcomposer.org/installer&#39;));&quot; </span></code></pre> <p>In the former case, you are teaching your users that piping code downloaded from web sites directly to an interpreter is OK; in the latter example, you are teaching people that doing the same with <code>eval</code> in PHP is also just fine. I, personally, think this is the worst part of all–while running <code>curl</code> to get the code is bad, it might also end up suggesting to people that <code>eval</code> is just fine for this purpose, when even the most rudimentary material on security will say just the opposite. Software written in PHP has a bad enough reputation for careless use of its capability to open, and if you’re not careful, execute ‘files’ across the Web, and advocating this as part of an installation process is really not a good thing.</p> <h2 id="in-summary">In summary<a class="zola-anchor" style="display: none;" href="#in-summary" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h2> <p>Although installers like this are undoubtedly convenient for the user, there is a huge risk as a result. Convenience and security undoubtedly do not go hand in hand; in this case, it would be far better to encourage people to download the installer file, and then perhaps verify it using a checksum such as SHA256–although, of course, this is also open to compromise, it would be a start. The package could also be signed with GPG or a similar system. This would take the user a few minutes, which undoubtedly gets in the way of using the software in question. However, I believe in this case the loss of convenience is far outweighed by mitigating the security risks involved.</p> 2012-06-25T00:00:00+00:00 2012-06-25T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/new-toys/ <p>For someone who hardly ever actually writes anything, I do spend quite a long time messing about with the software that powers my site. I think over the years I’ve variously used <a href="http://www.wordpress.org">WordPress</a>, <a href="http://ez.no">eZ Publish</a>, <a href="http://habariproject.org">Habari</a>, and probably other things that I’ve forgotten to run a weblog but never actually use it. Welcome to the latest incarnation, which this time is powered by <a href="https://github.com/mojombo/jekyll">Jekyll</a>. I decided that running a database and a server-side language all the time for a site that doesn’t get much traffic is pointless, and hence Jekyll suits my purposes by generating flat HTML files that require no further processing. It does also mean that I can store the site itself on Github, and write Markdown files in Vim to create posts, which appeals to my geek side.</p> <p>This time I built the theme from scratch myself, too, after spending a little too long reading books with names like <em>The Elements of Typographic Style</em> and <em>Thinking With Type</em>, as well as innumerable articles on <a href="http://alistapart.com">A List Apart</a> and places like that. I like to think that I have the typography of the site working to a reasonable level, and I’ll probably continue to ham-fistedly play with the rest of it. Just writing this post resulted in me messing with the CSS of the navigation as I wasn’t quite happy with how it looked.</p> <p>In the time since my last post, I have actually done useful things to a couple of the software projects I’ve been working on. I’ll aim to try and get some of those written up soon.</p> 2011-12-28T00:00:00+00:00 2011-12-28T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/peclcairo-030-released/ <p>As I’m sure you’ve all been longing to hear, we’ve finally got around to releasing a new version of <a href="http://pecl.php.net/package/cairo">PECL/Cairo</a>! This version fixes some bugs, and adds support for some new features available since Cairo 1.10 — notably, support for subsurfaces, which are surfaces that draw onto a part of a larger surface, and for recording surfaces, which record all drawing operations and can then be used as the source for other drawing operations. Also, <a href="http://www.theskillers.co.uk">Mark Skilbeck</a> has added support for Win32 fonts. Also, the FreeType font handling has been improved, and should now give more informative messages when errors occur.</p> <p>Any questions, comments, bug reports, or contributed code are welcome. Drop us a line on the PECL mailing list, or try #php.pecl on EFNet.</p> 2011-08-03T00:00:00+00:00 2011-08-03T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/froscon-2011/ <p>I’m very pleased to be able to announce that I’ll be giving two talks at this year’s FrOSCon conference in St Augustin near Bonn in Germany, on the 20th and 21st of August. The first is on the <a href="http://froscon.phpugdo.de">PHP track</a>, and is an overview of the <a href="http://pecl.php.net/package/cairo">PECL/Cairo</a> extension for PHP.</p> <p>The second talk is on the main track, and is about <a href="http://programm.froscon.org/2011/events/663.html">Making Software See</a>, where I’ll give an introduction to the <a href="http://www.opencv.org">OpenCV</a> image processing library, including some examples in various popular programming languages.</p> <p>I’m extremely happy to have been selected both by the main FrOSCon organisers and the PHP track, organised by the <a href="http://phpugdo.de/.html">PHP Usergroup Dortmund</a>. I hope to see some of you there!</p> 2011-06-09T00:00:00+00:00 2011-06-09T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/pango-for-php-pecl-channel/ <p>It occurs to me that though I wrote an <a href="http://mgdm.net/weblog/using-pango-for-php-a-taster">introduction post about Pango for PHP</a>, I forgot to mention that it’s actually installable using the PECL installer.</p> <pre> pecl channel-discover pecl.mgdm.net pecl install channel://pecl.mgdm.net/Pango-0.1.0 </pre> <p>Assuming you have the required development packages installed, it should go off and install it. You may still need to add the</p> <pre> extension=pango.so </pre> <p>line to your php.ini. Hope this helps someone give it a go! As ever, any feedback is welcome.</p> 2011-03-21T00:00:00+00:00 2011-03-21T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/android-linearlayouts-webviews-and-black-screens/ <p>Just making a quick note in case anyone else is searching for the same problem.</p> <p>I have an app with an activity which previously was just a WebView, but I wanted to add some other widgets above the WebView as well, so I made an XML layout and wrapped the WebView in a LinearLayout. Upon using this as the content view, my WebView went black. Completely. This resulted in a bit of frantic Googling, whereupon I discovered that it wasn’t an unusual problem. Most of the solutions revolved around making sure the LinearLayout’s width and height were both set to fill_parent — this didn’t work for me. A little bit of investigation then lead me to discover that I’d forgotten to set the orientation on the LinearLayout. Once I’d set that, everything started working fine. A silly problem, but quite often it’s the silly ones that take the most time to figure out.</p> 2011-02-21T00:00:00+00:00 2011-02-21T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/using-pango-for-php-a-taster/ <p>The <a href="http://pecl.php.net/package/cairo">PECL/Cairo</a> library is pretty good at drawing vector graphics (in our opinion, as the developers, at least!), but one thing it’s not able to do by itself is draw text with mildly advanced layout. It has the CairoContext::showText() function, but that doesn’t really let you do anything GD can’t. That’s because the developers of the Cairo library decided to let another more specialised library handle the job of text layout. Much of the time, the library that gets used for this is <a href="http://pango.org">Pango</a>. It has several bindings already for most popular languages on Unix-like platforms. It’s quite capable, and able to lay out text while taking care of things such as paragraph alignment, line breaking, bold/italic text, justification, and various other features. In this post, I intend to give a little tour of some of its features.</p> <p>Quite a while ago now, I wrote an extension to wrap Pango for PHP. It’s available on its <a href="https://github.com/mgdm/php-pango">Github repository</a>. It’s not available on PECL right now, but you can check it out from there and install it using the <a href="http://php.net/manual/en/install.pecl.phpize.php">normal method</a>. (Sorry Windows users, I’ve not had time to test it out on there, and likely won’t any time soon. Patches are welcome, though…). It requires that the Pango headers are installed, and that PECL/Cairo is already installed into PHP.</p> <p>Once installed, you can use it with Cairo. The best way to show it off is probably some example code:</p> <pre data-lang="php" class="language-php z-code"><code class="language-php" data-lang="php"><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-punctuation z-section z-embedded z-begin z-php">&lt;?php</span></span><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-meta z-function-call z-php"><span class="z-support z-function z-network z-php">header</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-string z-quoted z-double z-php"><span class="z-punctuation z-definition z-string z-begin z-php">&quot;</span></span><span class="z-string z-quoted z-double z-php"><span class="z-meta z-string-contents z-quoted z-double z-php"></span></span><span class="z-string z-quoted z-double z-php"><span class="z-meta z-string-contents z-quoted z-double z-php">Content-Type: image/png</span><span class="z-punctuation z-definition z-string z-end z-php">&quot;</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-comment z-block z-php"><span class="z-punctuation z-definition z-comment z-begin z-php">/*</span> Make a 300x300px image surface <span class="z-punctuation z-definition z-comment z-end z-php">*/</span></span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>s</span> <span class="z-keyword z-operator z-assignment z-php">=</span> <span class="z-keyword z-other z-new z-php">new</span> <span class="z-meta z-path z-php"><span class="z-support z-class z-php">CairoImageSurface</span></span><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span><span class="z-meta z-path z-php"><span class="z-support z-class z-php">CairoFormat</span></span><span class="z-punctuation z-accessor z-double-colon z-php">::</span><span class="z-constant z-other z-class z-php">ARGB32</span>, <span class="z-constant z-numeric z-integer z-decimal z-php">300</span>, <span class="z-constant z-numeric z-integer z-decimal z-php">300</span><span class="z-punctuation z-section z-group z-end z-php">)</span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>c</span> <span class="z-keyword z-operator z-assignment z-php">=</span> <span class="z-keyword z-other z-new z-php">new</span> <span class="z-meta z-path z-php"><span class="z-support z-class z-php">CairoContext</span></span><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>s</span><span class="z-punctuation z-section z-group z-end z-php">)</span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-comment z-block z-php"><span class="z-punctuation z-definition z-comment z-begin z-php">/*</span> Set the background to white <span class="z-punctuation z-definition z-comment z-end z-php">*/</span></span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>c</span><span class="z-meta z-function-call z-method z-php"><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-function z-php">setSourceRGB</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-constant z-numeric z-integer z-decimal z-php">1</span><span class="z-punctuation z-separator z-php">,</span> <span class="z-constant z-numeric z-integer z-decimal z-php">1</span><span class="z-punctuation z-separator z-php">,</span> <span class="z-constant z-numeric z-integer z-decimal z-php">1</span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>c</span><span class="z-meta z-function-call z-method z-php"><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-function z-php">paint</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-comment z-block z-php"><span class="z-punctuation z-definition z-comment z-begin z-php">/*</span> Let&#39;s draw using black &#39;ink&#39; <span class="z-punctuation z-definition z-comment z-end z-php">*/</span></span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>c</span><span class="z-meta z-function-call z-method z-php"><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-function z-php">setSourceRGB</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-constant z-numeric z-integer z-decimal z-php">0</span><span class="z-punctuation z-separator z-php">,</span> <span class="z-constant z-numeric z-integer z-decimal z-php">0</span><span class="z-punctuation z-separator z-php">,</span> <span class="z-constant z-numeric z-integer z-decimal z-php">0</span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-comment z-block z-php"><span class="z-punctuation z-definition z-comment z-begin z-php">/*</span> Make a Pango layout, set the font, then set the layout size <span class="z-punctuation z-definition z-comment z-end z-php">*/</span></span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>l</span> <span class="z-keyword z-operator z-assignment z-php">=</span> <span class="z-keyword z-other z-new z-php">new</span> <span class="z-meta z-path z-php"><span class="z-support z-class z-php">PangoLayout</span></span><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>c</span><span class="z-punctuation z-section z-group z-end z-php">)</span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>desc</span> <span class="z-keyword z-operator z-assignment z-php">=</span> <span class="z-keyword z-other z-new z-php">new</span> <span class="z-meta z-path z-php"><span class="z-support z-class z-php">PangoFontDescription</span></span><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span><span class="z-string z-quoted z-double z-php"><span class="z-punctuation z-definition z-string z-begin z-php">&quot;</span></span><span class="z-string z-quoted z-double z-php"><span class="z-meta z-string-contents z-quoted z-double z-php"></span></span><span class="z-string z-quoted z-double z-php"><span class="z-meta z-string-contents z-quoted z-double z-php">Bitstream Charter 28</span><span class="z-punctuation z-definition z-string z-end z-php">&quot;</span></span><span class="z-punctuation z-section z-group z-end z-php">)</span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>l</span><span class="z-meta z-function-call z-method z-php"><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-function z-php">setFontDescription</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>desc</span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>l</span><span class="z-meta z-function-call z-method z-php"><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-function z-php">setWidth</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-constant z-numeric z-integer z-decimal z-php">250</span> <span class="z-keyword z-operator z-arithmetic z-php">*</span> <span class="z-constant z-other z-php">PANGO_SCALE</span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-comment z-block z-php"><span class="z-punctuation z-definition z-comment z-begin z-php">/*</span> Here, we use Pango markup to make part of the text bold <span class="z-punctuation z-definition z-comment z-end z-php">*/</span></span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>l</span><span class="z-meta z-function-call z-method z-php"><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-function z-php">setMarkup</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-string z-quoted z-double z-php"><span class="z-punctuation z-definition z-string z-begin z-php">&quot;</span></span><span class="z-string z-quoted z-double z-php"><span class="z-meta z-string-contents z-quoted z-double z-php"></span></span><span class="z-string z-quoted z-double z-php"><span class="z-meta z-string-contents z-quoted z-double z-php">Hello &lt;b&gt;world!&lt;/b&gt; Here is a rather long paragraph which should get wrapped</span><span class="z-punctuation z-definition z-string z-end z-php">&quot;</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-comment z-block z-php"><span class="z-punctuation z-definition z-comment z-begin z-php">/*</span> Draw the layout on the surface <span class="z-punctuation z-definition z-comment z-end z-php">*/</span></span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>l</span><span class="z-meta z-function-call z-method z-php"><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-function z-php">showLayout</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>c</span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-comment z-block z-php"><span class="z-punctuation z-definition z-comment z-begin z-php">/*</span> Output the PNG to the browser <span class="z-punctuation z-definition z-comment z-end z-php">*/</span></span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>s</span><span class="z-meta z-function-call z-method z-php"><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-function z-php">writeToPng</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-string z-quoted z-double z-php"><span class="z-punctuation z-definition z-string z-begin z-php">&quot;</span></span><span class="z-string z-quoted z-double z-php"><span class="z-meta z-string-contents z-quoted z-double z-php"></span></span><span class="z-string z-quoted z-double z-php"><span class="z-meta z-string-contents z-quoted z-double z-php">php://output</span><span class="z-punctuation z-definition z-string z-end z-php">&quot;</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span></code></pre> <p>If all goes to plan, you should see a PNG in your browser with the above text.</p> <p>Going through each step, firstly we set up the Cairo surface to draw on, and the context we use to draw with. Once we have done that, we can create a PangoLayout object which lets us draw the text we require. The PangoLayout is passed a context, so it can invoke the drawing methods itself to draw the text. We pass the PangoLayout a PangoFontDescription object, which lets us choose from the fonts already installed on the machine. This means that we don’t need to concern ourselves with the paths to actual TrueType font files, or similar - the fonts are resolved by Fontconfig or whatever system is available on your machine.</p> <p>Next, we set the width of the layout. We can also set a height, but I haven’t bothered on this occasion. This lets Pango know where to wrap the text - if we don’t set this, it won’t bother, which may result in text falling off the edge of the image. You may note that the width is multiplied by the PANGO_SCALE - this is because Pango deals in units which are a tiny fraction (1/1024, in fact) of a pixel, in order to handle antialiasing properly. Because we’re using a CairoImageSurface, this means that they layout will be 250 pixels wide.</p> <p>Next up, we set the text to be drawn. There are two methods available to do this; PangoLayout::setText() is used when we just want to render text with no formatting instructions. In this case, I’ve opted for the PangoLayout::setMarkup() method, which lets me use <a href="http://library.gnome.org/devel/pango/stable/PangoMarkupFormat.html">Pango’s markup language</a> to make part of the text bold. Many other attributes can be changed using this markup, which has shortcuts that somewhat resemble HTML.</p> <p>Finally, the call to showLayout() renders the layout onto the surface. You can also render just a path, using layoutPath(), which sets the path on the surface so you can then use more advanced effects with Cairo. The layout is drawn using whatever the current source is on the Cairo context, so you can render using flat colours, gradients, other source images, or whatever takes your fancy. Additionally, this means that any transformations you have set on the Cairo context also take effect, allowing you to rotate, scale, shear and otherwise distort the text.</p> <p>It probably looks a little complex just to write some text on an image, but it is rather flexible. This flexibility is handy when you consider that Cairo can render more than just PNG images; PDFs, PostScript and SVG are also easy to create. I hope that this post may inspire someone else to give it a try. Feedback and reports of issues are always welcome.</p> 2011-02-02T00:00:00+00:00 2011-02-02T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/libsureelec-a-driver-for-sure-electronics-lcd-displays/ <p><img src="/assets/img/posts/IMAG0227.jpg" alt="" /></p> <p>A few months ago I got a pair of 4x20 LCD displays from Sure Electronics, via eBay. Though they worked quite happily through LCDproc, I didn’t really want to use that (as I intend to use one of them on a <a href="http://bifferos.bizhat.com/">Bifferboard</a>). To that end, I’ve attempted to write a driver for them. Until recently I didn’t have a lot of success, which made me think that the documentation provided with the device was incorrect, until <a href="http://www.gjcp.net">Gordon</a> suggested trying the command set through gtkterm. I did, which made it work, and discovered that it requires the entire line to be sent to the display even if you’re only updating part of it. Adjusting my code to handle this, and add some sleep time after sending each command, made it work.</p> <p>I’ve put the code on <a href="http://github.com/mgdm/libsureelec">Github</a> in case anyone else is interested. It’s able to drive my DE-LD023 device quite happily, but I suspect it should work for other 4x20 devices too. Once I add the code to detect the device features it should also work for different sized displays. Any comments or patches are of course welcome.</p> 2011-01-19T00:00:00+00:00 2011-01-19T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/cmses-an-update/ <p>My previous post was typed in a bit of a hurry (was it that obvious?) and could probably have done with a few more sarcasm tags, though I think I still have a valid, if small, point. It surprised me that these well-maintained projects seem to be content to run with what I would consider to be errors. Though they do run, otherwise nobody would be using them. For what it’s worth, SilverStripe and I have made up and it’s all going swimmingly at present. Thanks to everyone who commented with recommendations; I’ll try and follow some of them up when I get time.</p> <p>One comment did ask me to define what I meant when I refer to a “lightweight” CMS – basically, I was looking for something that will allow me to create a hierarchy of pages that consist roughly of a title and content. Nothing fancy. I don’t need multiple content types, complex user management, discussion forums, built-in wikis, etc. It’s the sort of thing I could have coded up in a couple of days using any number of frameworks, but I figured there’d be something off the shelf that’d do what I was after.</p> <p>Another commenter questioned why I cared about coding standards warnings seemingly more than avoiding SQL injection and XSS and CSRF protection – I don’t, those are absolute hard requirements for any application. I’d put avoiding the coding standards warnings somewhere above “nice to have”, but those warnings are indications that something is being done in a way which is either less than ideal (such as using non-static methods in a static context, a la Concrete5), or deprecated (such as using the ereg functions).</p> <p>My next post shall try and be a little more constructive, as I release some code.</p> 2011-01-17T00:00:00+00:00 2011-01-17T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/where-are-all-the-decent-php-cmses/ <p>I’ve been recently asked to check out some CMSes for someone, and try to find a recommendation. What I’ve been finding hasn’t really been encouraging. Out of the several CMSes I’ve tried, they’ve all failed for various reasons. I admit that I might be looking at this from a slightly different perspective than most - that of admining it and coding against it, rather than as a user - but it’s still rather disappointing.</p> <ul> <li>Silverstripe is what I used first - but the forms (including the login to the admin) kept failing to render. In the end, I got sick of debugging it and started my hunt.</li> <li>Next up was Concrete5, which I heard mentioned recently. There were <strong>one hundred and fifty-five</strong> strict warnings in the front page of the installer. Bye bye.</li> <li>Next, CMS Made Simple. Its installer wanted <code>E_\STRICT</code> and <code>E\_DEPRECATED</code> removed from error reporting. At this point some folk might have continued; but that was enough for me not to consider it. I didn’t have enough confidence in the code that I wanted to continue. It lost more points for the tarball not extracting into a subdirectory, which resulted in my document root being filled with nonsense.</li> <li>Joomla!… well, my last experience of it wasn’t pleasant - I couldn’t work out how the code was structured. It’s also overkill for the requirements of this site.</li> <li>I would love to go for Habari, but the site isn’t a blog, so it doesn’t really fit the use case. It’s still the top of the list just now, though.</li> <li>I could do what everyone else does and run Wordpress - but, again, the site isn’t a blog, and I’d rather try something different.</li> </ul> <p>I didn’t consider Drupal or eZ Publish, which again I believed to be massive overkill for what is required of the site. What’s going on? This is what PHP is supposed to be good at. Where are the simple, lightweight CMSes with modern code? Notices, strict and deprecation warnings should be considered errors when developing - this doesn’t seem to be the case with many of the above projects.</p> <p>So, that’s what I’ve found. I now don my asbestos suit.</p> 2011-01-16T00:00:00+00:00 2011-01-16T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/habari-plugins-published/ <p>I’ve just committed two new plugins to the habari-extras Subversion. Firstly, there’s the ‘share’ plugin, which adds the Facebook OpenGraph metadata to pages so that the Like button works, and also adds Facebook and Twitter widgets to the bottom of posts. Secondly, there’s a bitly plugin, that pings bit.ly’s API when every post is published and generates a short URL. This URL is then stored in the post info, and it provides a template to add it to the head of each page using the <a href="http://microformats.org/wiki/rel-shortlink">shortlink</a> markup. I’d be interested in feedback from anyone more familar with Habari in case there’s a better way to do what I’m doing.</p> <ul> <li><a href="http://svn.habariproject.org/habari-extras/plugins/share/">Share plugin</a></li> <li><a href="http://svn.habariproject.org/habari-extras/plugins/bitly/">Bit.ly plugin</a></li> </ul> 2011-01-15T00:00:00+00:00 2011-01-15T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/recent-activity-habari-plugins/ <p>I think I’m just about keeping to this one-post-a-week thing, almost. I’ve recently been playing with the internals of Habari, writing a couple of plugins. I’ve found it quite well put together, though I don’t seem to ever get the Utils::debug() output to display properly - though this might have been due to the code I was writing running before a redirect. I believe though that I now have got 2 plugins which I’m going to investigate publishing - one which generates the Facebook and Twitter widgets at the bottom of the posts (and the relevant OpenGraph metadata for Facebook in the header), and another which pings bit.ly and generates a short URL for each post as it’s published. I’m writing this post in part as a test to see that it works for real :-)</p> <p>If I get a bit of time over the next few days, I’ll finish tidying them up and make them available. If anyone’s interested before I do, drop me an email or catch me on Twitter.</p> 2011-01-02T00:00:00+00:00 2011-01-02T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/2011-now-with-less-tumbleweed/ <p>It’s my intention this year to at write at least one post on here per week, ideally with some substance to it, rather than just recording what I had for breakfast (that being Twitter’s job, after all). I’m not holding my breath about whether I will manage it or not, but I intend to give it a go. I have a draft in the works, which I will hope to get out there some time soon…</p> <p>In actual news, I have been accepted to talk at <a href="http://confoo.ca">ConFoo</a> in Canada, on the subject “Making PHP See” - which is about image recognition using OpenCV and PHP. I’m really looking forward to the conference, and I’m fortunate to be up alongside some excellent speakers, so it should be great fun.</p> 2010-06-12T00:00:00+00:00 2010-06-12T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/cairo-feedback-from-dpc10/ <p>I noted a couple of comments after my talk on Cairo at <a href="http://phpconference.nl">DPC 10</a> yesterday, so I thought I’d respond to them here to clear up what’s going on. It’s always good to get some feedback about something I’ve been working on so I’d like to reply and keep the conversation going if I can.</p> <p>First off, someone mentioned that the wrapper seems a bit beta and incomplete - and yes, that is true. The 0.2.0 release, which is the latest as I wrote this, is still marked as beta. The aim of me coming and talking about it is to try and get some more people interested in using it, so we can get somemore information from real users about what works and what doesn’t. I probably didn’t help by mentioning that there are some features that we intend to implement that aren’t there yet, which I’ll list:</p> <ul> <li><strong>Support for reading and writing images that aren’t PNGs</strong> The developers of the core Cairo API are not really interested in handling all the various graphics formats that exist, so they decided to implement one and one only which is useful for getting data in and out. This is the functionality that we use at present. We (the developers of the wrapper) would like to fix that and handle JPEG, GIF, TIFF, and whatever new format is cool this week, but we’re not quite sure how to go about that yet. We could recycle the code used by GD or use ImageMagick or GraphicsMagick, but those would introduce dependencies that people might not have available. Once we work out a decent solution here, it will be put in.</li> <li><strong>Support for hyperlinks in PDFs and SVGs</strong> Again, this is something we’d quite like but it’s not supported in the underlying library <strong>yet</strong>. There have been rumours about adding an API to do this, but there hasn’t been a concensus about what’s a good way to go about it. It’s something I’d quite like to help fix, so I’ve been learning how Cairo itself is put together with the intention to add the support.</li> </ul> <p>The other comments I saw suggested that the API is a bit fiddly, which I can’t deny. It’s not quite as simple to set up and get running as some of the other libraries in PHP already. This is nearly intentional - we’re wrapping the underlying Cairo API as closely as we can, because we don’t want to diverge too far from how other languages with Cairo wrappers work, so that developers who may be familiar with those already know roughly what is going on. It’s for this reason that, for example, the API uses floating-point values from 0 to 1 to define the intensity of colours, rather than an integer value from 0-255 which most people are probably used to. This is because Cairo is designed not to make any assumptions about the surface you’re going to be drawing to - some day we might get 16-bits-per-channel surfaces, and using float values means that we don’t need to change the API or the values we’re passing in to allow us to write to them. It’ll just carry on working as it did before, just with better colour resolution.</p> <p>However, if anyone has any ideas for how we could make things simpler, then we’d love to hear about it &amp;emdash; drop us a line on the PECL developers mailing list (which you can find on the <a href="http://pecl.php.net/support.php">PECL mailing lists</a> page), or find us on IRC on #php.pecl on EFnet.</p> <p>There are a couple of things that I have worked on which sit on top of the Cairo extension, which can do some interesting things. If you’re still at DPC I’ll be demoing a couple of them on the uncon track at 2pm. It’s only a 15 minute talk, so it shouldn’t be too boring…!</p> 2010-06-01T00:00:00+00:00 2010-06-01T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/ubuntu-accessibility-team/ <p>Recently I’ve become involved in a move to resurrect the Ubuntu Accessibility team, which seemed to have stagnated until quite recently when <a href="http://pendulumtech.wordpress.com">Penelope Stowe</a> got involved and suggested I joined too. She arranged a session at the Ubuntu Developer Summit for the next Ubuntu release, Maverick Meerkat, which seems to have <a href="http://pendulumtech.wordpress.com/2010/05/31/ubuntu-accessibility-team-goals-for-maverick-meerkat/">gone quite well</a>. The main focus for the team right now is getting organised, and to this end we’re aiming to create a set of personas, which are representations of hypothetical users with various accessibility requirements which developers and testers can keep in mind when working on the next Ubuntu release. These personas are going to be developed by a survey which will be dispatched to a host of groups who may have accessibility requirements to get an idea of what the main factors that affect their experience with Ubuntu are, so we can get an idea of the main areas to improve. Hopefully then the work which will take place as a result of these can be sent upstream to the original projects.</p> <p>My own role in the team isn’t quite clear yet, but I intend to help on the development side, if I can, once more work starts in this area. If you would like to help, the team hangs out on <a href="irc://chat.freenode.net/#ubuntu-accessibility">#ubuntu-accessibility</a> on Freenode IRC, and there is a <a href="http://lists.ubuntu.com/mailman/listinfo/ubuntu-accessibility">mailing list</a>. You can also check out the <a href="https://wiki.ubuntu.com/Accessibility">wiki page</a> which we’re working on improving.</p> 2010-04-10T00:00:00+00:00 2010-04-10T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/eurowinphp-contest/ <p>This year I’ve decided to enter the <a href="http://www.eurowinphp.com">EuroWinPHP</a> contest, as I figured it might be interesting to learn a bit more about how PHP works on Windows. I originally started using it there, back in around 2002, running early versions of Apache 1.3 and PHP 4.1 on Windows 98 and 2000. These days, though, I’m running IIS7 and PHP 5.3.2, and have a bit more knowledge of how the bits fit together, though since I’ve been running Linux for the last 5 years I’m a little out of touch.</p> <p>The project I’ve opted to do is something I’ve been thinking about for a little while, but not got around to developing until now. It’s named GarnetCMS, and the idea is to create a CMS that uses the features of PHP 5.3, such as closures, and based around a signal/slot mechanism. The idea is to initially develop it to use NoSQL datastores, and I intend to create back ends for <a href="http://www.microsoft.com/windowsazure/">Microsoft Azure</a> and <a href="http://www.mongodb.org">MongoDB</a> at first. If I do things correctly, though, adding a traditional RDBMS back end later on should be relatively simple.</p> <p>Development-wise, I’m trying out Mercurial for version control, and using Bitbucket to store the code - the project will be on there at <a href="http://bitbucket.org/mgdm/garnet-cms/">http://bitbucket.org/mgdm/garnet-cms/</a> once it gets going. I’m intending to try out Komodo Edit, because while I do love GVim on Windows it doesn’t quite fit with the “complete change” I’m going for with most other things! <a href="http://tortoisehg.bitbucket.org/">TortoiseHg</a> seems quitenice so far, and I’ve managed to get IIS up and running, so now all I need to do is get coding. I’m quite looking forward to seeing how it turns out.</p> 2010-03-01T00:00:00+00:00 2010-03-01T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/handling-fonts-in-peclcairo/ <p>(This is just a quick note to get some information out there for reference, I am adding it to the PHP manual as well!)</p> <p>Currently, in PECL/Cairo the only way to draw text is the referred to as the “toy” text API, which is a very basic way of handling text compared to the facilities available in the Cairo library itself. However, it’s sufficient for most purposes that I’ve come across so far. In version 0.1.0 of PECL/Cairo, there was only one way to choose what font you wished to use, which was the <a href="http://www.php.net/manual/en/cairocontext.selectfontface.php">CairoContext::selectFontFace()</a> method. You pass a string to this method with the name of the font you want, along with the optional slant and weight parameters. This then invokes your system’s font handling to find the font you’re after, or an alternative if it’s not available, so you need to have the font you want installed into your system’s font library. This is occasionally not handy.</p> <p>In version 0.2.0 FreeType support was added. It allows you to choose any font file you’d like, as long as PHP’s streams API can find it. Yes, this means <code>http://</code> streams, but I wouldn’t recommend it.</p> <pre data-lang="php" class="language-php z-code"><code class="language-php" data-lang="php"><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-punctuation z-section z-embedded z-begin z-php">&lt;?php</span></span><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-comment z-block z-php"><span class="z-punctuation z-definition z-comment z-begin z-php">/*</span> Set up the surface, and make the background white <span class="z-punctuation z-definition z-comment z-end z-php">*/</span></span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>s</span> <span class="z-keyword z-operator z-assignment z-php">=</span> <span class="z-keyword z-other z-new z-php">new</span> <span class="z-meta z-path z-php"><span class="z-support z-class z-php">CairoImageSurface</span></span><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span><span class="z-meta z-path z-php"><span class="z-support z-class z-php">CairoFormat</span></span><span class="z-punctuation z-accessor z-double-colon z-php">::</span><span class="z-constant z-other z-class z-php">ARGB32</span>, <span class="z-constant z-numeric z-integer z-decimal z-php">300</span>, <span class="z-constant z-numeric z-integer z-decimal z-php">100</span><span class="z-punctuation z-section z-group z-end z-php">)</span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>c</span> <span class="z-keyword z-operator z-assignment z-php">=</span> <span class="z-keyword z-other z-new z-php">new</span> <span class="z-meta z-path z-php"><span class="z-support z-class z-php">CairoContext</span></span><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>s</span><span class="z-punctuation z-section z-group z-end z-php">)</span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>c</span><span class="z-meta z-function-call z-method z-php"><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-function z-php">setSourceRgb</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-constant z-numeric z-integer z-decimal z-php">1</span><span class="z-punctuation z-separator z-php">,</span> <span class="z-constant z-numeric z-integer z-decimal z-php">1</span><span class="z-punctuation z-separator z-php">,</span> <span class="z-constant z-numeric z-integer z-decimal z-php">1</span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>c</span><span class="z-meta z-function-call z-method z-php"><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-function z-php">paint</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-comment z-block z-php"><span class="z-punctuation z-definition z-comment z-begin z-php">/*</span> Draw the text using the Vollkorn font, from </span></span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-comment z-block z-php"> http://friedrichalthausen.de/2006/01/01/vollkorn/ <span class="z-punctuation z-definition z-comment z-end z-php">*/</span></span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>c</span><span class="z-meta z-function-call z-method z-php"><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-function z-php">setSourceRGB</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-constant z-numeric z-integer z-decimal z-php">0</span><span class="z-punctuation z-separator z-php">,</span> <span class="z-constant z-numeric z-integer z-decimal z-php">0</span><span class="z-punctuation z-separator z-php">,</span> <span class="z-constant z-numeric z-integer z-decimal z-php">0</span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>c</span><span class="z-meta z-function-call z-method z-php"><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-function z-php">moveTo</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-constant z-numeric z-integer z-decimal z-php">10</span><span class="z-punctuation z-separator z-php">,</span> <span class="z-constant z-numeric z-integer z-decimal z-php">60</span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>f</span> <span class="z-keyword z-operator z-assignment z-php">=</span> <span class="z-keyword z-other z-new z-php">new</span> <span class="z-meta z-path z-php"><span class="z-support z-class z-php">CairoFtFontFace</span></span><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span><span class="z-meta z-function-call z-php"><span class="z-support z-function z-file z-php">dirname</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-constant z-language z-php">__FILE__</span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span></span></span> <span class="z-keyword z-operator z-string z-php">.</span> <span class="z-string z-quoted z-double z-php"><span class="z-punctuation z-definition z-string z-begin z-php">&quot;</span></span><span class="z-string z-quoted z-double z-php"><span class="z-meta z-string-contents z-quoted z-double z-php"></span></span><span class="z-string z-quoted z-double z-php"><span class="z-meta z-string-contents z-quoted z-double z-php">/vollkorn.otf</span><span class="z-punctuation z-definition z-string z-end z-php">&quot;</span></span><span class="z-punctuation z-section z-group z-end z-php">)</span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>c</span><span class="z-meta z-function-call z-method z-php"><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-function z-php">setFontFace</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>f</span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>c</span><span class="z-meta z-function-call z-method z-php"><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-function z-php">setFontSize</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-constant z-numeric z-integer z-decimal z-php">50</span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>c</span><span class="z-meta z-function-call z-method z-php"><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-function z-php">showText</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-string z-quoted z-double z-php"><span class="z-punctuation z-definition z-string z-begin z-php">&quot;</span></span><span class="z-string z-quoted z-double z-php"><span class="z-meta z-string-contents z-quoted z-double z-php"></span></span><span class="z-string z-quoted z-double z-php"><span class="z-meta z-string-contents z-quoted z-double z-php">Hello world</span><span class="z-punctuation z-definition z-string z-end z-php">&quot;</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-comment z-block z-php"><span class="z-punctuation z-definition z-comment z-begin z-php">/*</span> Send the image to the browser <span class="z-punctuation z-definition z-comment z-end z-php">*/</span></span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-meta z-function-call z-php"><span class="z-support z-function z-network z-php">header</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-string z-quoted z-double z-php"><span class="z-punctuation z-definition z-string z-begin z-php">&quot;</span></span><span class="z-string z-quoted z-double z-php"><span class="z-meta z-string-contents z-quoted z-double z-php"></span></span><span class="z-string z-quoted z-double z-php"><span class="z-meta z-string-contents z-quoted z-double z-php">Content-type: image/png</span><span class="z-punctuation z-definition z-string z-end z-php">&quot;</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"><span class="z-variable z-other z-php"><span class="z-punctuation z-definition z-variable z-php">$</span>s</span><span class="z-meta z-function-call z-method z-php"><span class="z-punctuation z-accessor z-arrow z-php">-&gt;</span><span class="z-variable z-function z-php">writeToPng</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-begin z-php">(</span></span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-string z-quoted z-double z-php"><span class="z-punctuation z-definition z-string z-begin z-php">&quot;</span></span><span class="z-string z-quoted z-double z-php"><span class="z-meta z-string-contents z-quoted z-double z-php"></span></span><span class="z-string z-quoted z-double z-php"><span class="z-meta z-string-contents z-quoted z-double z-php">php://output</span><span class="z-punctuation z-definition z-string z-end z-php">&quot;</span></span><span class="z-meta z-function-call z-php"><span class="z-meta z-group z-php"><span class="z-punctuation z-section z-group z-end z-php">)</span></span></span></span></span><span class="z-punctuation z-terminator z-expression z-php">;</span> </span></span></span></span><span class="z-embedding z-php"><span class="z-text z-html z-basic"><span class="z-meta z-embedded z-block z-php"><span class="z-source z-php"></span><span class="z-punctuation z-section z-embedded z-end z-php">?&gt;</span><span class="z-meta z-html-newline-after-php z-php"> </span></span></span></span></code></pre> <p>The output image should hopefully look like this:</p> <img alt="Script output" src="http://mgdm.net/weblog/user/files/cairo-freetype.png"> <p>(For those who are familiar with the Cairo library, this function maps to the <code>cairo_ft_font_face_create_for_ft_face</code> function in its API.)</p> <p>There is still some work to be done in this area, notably to support the Windows and Mac OS X font systems, but they’ll be coming in a future release, we hope. If anyone would like to help us with that, or any other aspect of it (including documentation!), you can get in touch on the <a href="http://news.php.net/php.pecl.dev">PECL dev mailing list</a>, or if you’re on IRC, drop in to #php.pecl on EFnet. All contributions are welcome!</p> 2010-02-26T00:00:00+00:00 2010-02-26T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/speaking-at-dpc10/ <p>I’m happy to say that I’ll be speaking at the <a href="http://phpconference.nl/">Dutch PHP Conference</a> in June in Amsterdam, on the subject of the <a href="http://pecl.php.net/package/cairo">PECL/Cairo extension</a> I’ve been helping out by working on for the past few months. This will be my first appearance as a speaker at a technical conference so I’m a little nervous, but I’ve no doubt I’ll be practicing a bit before it happens. Apologies in advance to anyone I inflict the talk on before the event.</p> 2010-02-03T00:00:00+00:00 2010-02-03T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/peclcairo-020-released/ <p>The first beta version of the <a href="http://pecl.php.net/package/cairo">PECL/Cairo extension</a> has just been released. This version includes support for loading arbitrary fonts via Freetype, cloning matrices, and has a rather large set of bug fixes. If you’ve been using 0.1.0 for anything at all I’d really recommend an upgrade. Windows builds will appear soon over at <a href="http://perisama.net/cairo/">perisama.net</a> for all the major PHP variants courtesy of <a href="http://www.elizabethmariesmith.com/">Elizabeth M. Smith</a>. Many thanks to Mark Skilbeck for helping get this release working on Windows!</p> 2010-01-21T00:00:00+00:00 2010-01-21T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/cairo-article-published/ <p>I’ve got 3 posts or so queued up in my mind to write out, but some of them rather depend on doing other things first, so this will just be a quick note to say I have an article in <a href="http://www.phparch.com/magazine/index/111">the most recent php|architect magazine</a> on the subject of “Vector Graphics with Cairo”. This is a basic introduction to the <a href="http://pecl.php.net/pecl/cairo">PECL/Cairo</a> extension which I’ve been helping out on for a while now. Thanks to <a href="http://caseysoftware.com/">Keith Casey</a> for his input, to the folks at php|architect for letting me do it, and the long-suffering <a href="http://www.elizabethmariesmith.com/">Elizabeth Marie Smith</a> for putting up with all my stupid questions while working on this and other extensions!</p> <p>Now, back to my regularly scheduled coding…</p> 2009-11-09T00:00:00+00:00 2009-11-09T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/a-brief-update/ <p>Last weekend I spent a couple of days on the yacht <a href="http://www.flickr.com/photos/mgdm/3087512670/">Hebridean</a> as part of an <a href="http://www.rya.org.uk/coursestraining/courses/sailcruising/Pages/Dayskipper.aspx">RYA sail cruising day skipper course</a>. This was the second half, as we were aboard before last month but had to cancel due to the weather. This time, we combined the course with delivering the yacht to Oban Marina on Kerrera where it will spend the winter. I’m happy to say I passed, so theoretically at least I could be able to charter a yacht for a trip at some stage. I might investigate that for one summer, but I’ll want a bit more mileage and experience first. I also noticed that the local college were running a Coastal Skipper theory course, so I’m on that now too.</p> <p>I’m considering going to <a href="http://fosdem.org">FOSDEM</a> in Belgium in February. I missed <a href="http://lugradio.org/live/">LugRadio Live</a> this year so I’m thinking of it as a sort of substitute. Also considering PHP London, but we’ll see.</p> 2009-07-16T00:00:00+00:00 2009-07-16T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/debug-of-the-year/ <p>I saw <a href="http://news.bbc.co.uk/1/hi/world/americas/8152278.stm">this article</a> on the BBC, where a man who bought a packet of cigarettes from a petrol station in the US ended up being charged $23,148,855,308,184,500 instead. Wow. I was a little curious as to how this could have happened, though the number didn’t look familar - I know roughly what 2^32, 2^32 / 2, 2^64, 2^64 / 2 look like, and this didn’t appear to be any of those. A swift Google though led me to Stack Overflow, which is a fantastic site. Therein lies <a href="http://stackoverflow.com/questions/1133581/is-23-148-855-308-184-500-a-magic-number-or-sheer-chance/1133612#1133612">a quite plausible explanation</a>:</p> <blockquote> <p>Add the cents to the number and you get 2314885530818450000, which in hexadecimal is 2020 2020 2020 1250. Do you see the pattern? The first six bytes has been overwritten by spaces (hex 20, dec 32).</p> </blockquote> <p>Ingenious! I can imagine spending quite a while trying to come up with that…</p> 2009-07-06T00:00:00+00:00 2009-07-06T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/peclcairo-010-released/ <p>The first alpha of the <a href="http://pecl.php.net/package/cairo">PECL Cairo</a> extension I’ve been helping out with has been released. Please download it, play with it, try and break it, and file bugs if you do! There isn’t an official manual yet (for licencing-related reasons which I’m hoping to sort out soon), but I’ll update this post when there is one. For now, there is a lot of stuff in the <a href="http://cvs.php.net/viewvc.cgi/pecl/cairo/examples/">examples/</a> directory in the source which can be used as inspiration.</p> 2009-05-15T00:00:00+00:00 2009-05-15T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/what-ive-been-up-to/ <p>I finally decided to get around to doing something about my site, so I’ve made a front page for it which is better than a straight redirect onto the blog. Hopefully it’s reasonable.</p> <p>I’ve been learning a bit more recently about the internals of PHP, and how to write extensions for it. I’ve been practicing by doing a bit of work on a <a href="http://trac.macvicar.net/php-cairo">Cairo wrapper</a> that was started in the Google Summer of Code last year and until now has been mainly looked after by <a href="http://elizabethmariesmith.com/">Elizabeth Marie Smith</a>. She’s put up with many, many newbie questions from me and so I’ve learned a lot while doing it. It’s now at the stage where it’ll run the eZ Components Graph component without too much complaining. It has a dual procedural and object-oriented API, so it should manage to run most things that use the existing cairo_wrapper extension. Thanks to all the regulars on #php.pecl and #php.doc on EFNet, and #php-gtk on Freenode for putting up with me while doing this. I’ve also written a quick wrapper for <a href="http://tokyocabinet.sourceforge.net/">Tokyo Cabinet</a> for the DBA extension in PHP which with a bit of luck will get committed to PHP6 some time soon.</p> <p>In other news, I’ve been occasionally helping in the maintenance of Jubilee and An Sulaire, the two Sgoth Niseach boats I go sailing on occasionally. Hopefully they’ll be going back in the water soon and we can get some sailing done. I’m looking forward to it.</p> 2009-05-08T00:00:00+00:00 2009-05-08T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/php-530rc2/ <p><a href="http://www.php.net/archive/2009.php#id2009-05-07-1">PHP 5.3.0RC2 has been released</a>, which means that the next version of PHP is just around the corner. It comes right in the middle of the annual <a href="http://wiki.php.net/qa/testfest">PHP Testfest</a>. I rambled on a bit on the <a href="http://podcast.freenode.net/episodes/free-as-in-node-season-1-episode-3-soapbox">Freenode podcast</a> a couple of weeks ago about this - it’s an annual project where PHP usergroups and individuals around the world get together to improve the unit testing of PHP. This has benefits for everyone involved - PHP is improved, and more likely to maintain backward compatibility, and the people involved in writing the tests get to contribute in a meaningful way to the project, and hopefully learn something on the way. Several usergroups have events scheduled, and a couple have already taken place to great success. If you’re a serious PHP user, and interested in getting involved, it’s well worth checking out.</p> <p>Even just grabbing the latest PHP release, compiling it on your platform, and running “make test” helps the project. It doesn’t take much to do, and you can run it in the background while doing something else. Go for it!</p> 2008-10-19T00:00:00+00:00 2008-10-19T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/my-favourite-ever-dialogue-box/ <pre class="z-code"><code><span class="z-text z-plain">&lt;p&gt;I saw this on &lt;a href=&quot;http://www.thedailywtf.com&quot;&gt;TheDailyWTF&lt;/a&gt; recently, and decided I had to try it for myself, and it works!&lt;/p&gt; </span></code></pre> <img alt="the-best-dialogue-box-ever.png" src="/weblog/user/files/the-best-dialogue-box-ever.png"> <p>You get it if you go to "Clear History" in the "Go" menu in Nautilus.</p> 2008-09-29T00:00:00+00:00 2008-09-29T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/custom-functions-on-the-400d/ <p>This is a camera-geek post, so if you’re not one of those, feel free to skip. I’ve just been having a bit more of a play with the custom functions available to change how the 400D operates. There are a couple of things that I’ve found that I can’t believe I didn’t know about - they look like they’ll make things a lot easier. I’m making a note here so that if I lose them, I can find them again.</p> <p>Now I just need to get out and do some more shooting. Not done very much of late that hasn’t been with my N95, which is disappointing.</p> <table><thead><tr><th>Setting name</th><th>Value</th><th>Effect</th></tr></thead><tbody> <tr><td>CF01 - Set button/cross keys function</td><td>4: Cross keys - AF Frame select</td><td>Makes the arrow keys choose the autofocus point when shooting, rather than selecting the picture “style”, which is something I’ve never used beyond setting it on “neutral”</td></tr> <tr><td>CF04 - Shutter/AE lock button</td><td>1: AE Lock/AF</td><td>This makes the “*” button cause the camera to autofocus, and have pressing the shutter button half way set the exposure (in non-manual modes). This might well drive me nuts, but I like the idea, in theory.</td></tr> <tr><td>CF05 - AF-assist beam</td><td>2: Only external flash emits</td><td>This should stop it flashing like mad when trying to focus in low light, when I’m not using my 430EX. I don’t use the on-camera flash that much anyway, so it won’t make a lot of difference, but it is an irritating feature when I am using it.</td></tr> <tr><td>CF09 - Shutter curtain sync</td><td>1: Second curtain sync</td><td>This makes the flash fire just before the shutter closes, rather than when the shutter opens. This might not seem to make a lot of sense at first glance, but it means that if I take a photo with a relatively long exposure with the intention of getting some motion blur, the shutter will fire at the end of the exposure, so that the object is “frozen” with the trail behind it, rather than seeming to go ahead of it.</td></tr> <tr><td>CF11 - LCD display when power ON</td><td>1: Retain power OFF status</td><td>This stops the display coming on when I switch the camera on. I don’t use the display much so I have it turned off most of the time to save battery power.</td></tr> </tbody></table> 2008-07-28T00:00:00+00:00 2008-07-28T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/making-timelapses-with-linux/ <p>In other news, I’ve been spending far too much time recently with my camera. One thing I quite like doing is timelapse photography, which my camera (a Canon 400D) can’t do by itself, however I did acquire a cheap intervalometer from eBay, which together with my new tripod (a Velbon Sherpa 250, highly recommended) is quite for this kind of work.</p> <p>I can’t, however, ever remember the command to create the timelapses. So, here’s a note to myself and anyone else who’s interested. If you have a folder full of .JPG files, with a sequential numbering scheme like most cameras have by default, you can run this command to create a video file using each JPEG image as a frame:</p> <pre class="z-code"><code><span class="z-text z-plain">mencoder &quot;mf://*.JPG&quot; -mf fps=25 -vf scale -zoom -xy 1024 -ovc lavc -lavcopts vcodec=mpeg4 -o output.avi </span></code></pre> <p>This will output an MPEG4 file in the file output.avi. It will scale the images so that the video ends up being 1024 pixels wide, and set the height to keep the aspect ratio. It will also run at 25 frames per second. These values can be tweaked, depending on the effect you want.</p> <p>I’ll update this post once I’ve figured out a few more things - I’d like to be able to do panning and zooming as well, and I know it’s possible, though perhaps a bit fiddly.</p> 2008-06-04T00:00:00+00:00 2008-06-04T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/lugradio-live-2008/ <p><a href="http://lugradio.org/live/UK2008/">LugRadio Live 2008</a> is happening on the 19th and 20th of July at The Lighthouse Media Center, Chubb Buildings, Fryer Street, Wolverhampton. This’ll be my first time there, and so I’ve been very sensible and volunteered for the crew. Should be fun!</p> 2008-04-23T00:00:00+00:00 2008-04-23T00:00:00+00:00 Michael Maclean https://mgdm.net/weblog/the-linksys-nslu2-part-one/ <p>I’ve spent some time recently playing with a couple of Linksys NSLU2s. These are small devices which have 2 USB ports and an Ethernet port, and are intended to be used to share USB drives across a network via CIFS (Windows file sharing). The best feature of them though is that like the early WRT54G routers, it runs Linux by default, and uses Samba to share the files. This means that with a bit of hacking, it’s fairly easy to extend it to make it more functional. There are a few projects doing this, the most famous being the Unslung firmware.</p> <p>Another project that has been working on support for the NSLU2 is Debian. As of version 4.0 (Etch), the default Debian installer has had support for them out of the box. More information on this can be found at <a href="http://cyrius.com/debian/nslu2/">http://cyrius.com/debian/nslu2/</a>. I’ve installed Etch on both my NSLU2s, though I found it quicker and easier to use the manual install method, which just means unpacking a tarball of the base system. This is described at <a href="http://cyrius.com/debian/nslu2/unpack.html">http://cyrius.com/debian/nslu2/unpack.html</a>. The instructions there are far better than I could describe, so I won’t bother trying to replicate them.</p> <p>So, what can you do with these things once they’re running Debian? Lots of things. Just now I’m going to describe one of the most basic things I use it for, which is for serving DHCP and DNS on my LAN.</p> <p>You can, if you like, run the full ISC BIND and dhcpd servers on your Slug, but there isn’t really that much point. They’re big, sometimes tricky to configure, and overkill for a small LAN. A decent, light alternative is dnsmasq. It’s a small daemon that uses the existing /etc/hosts and other related files to handle serving the network. I run it for a couple of reasons - firstly, my ISP’s DNS servers are somewhat less than reliable, so I use dnsmasq to proxy to OpenDNS. Secondly, it allows me to use internal DNS names for my machines. So, for example, rather than having to remember that my router is 192.168.1.1, if I want to go and change something in its configuration, I can just type “portal” into my browser. Similarly, my NSLU2s can be accessed by just typing “ssh kaylee” or “ssh inara”. I’ve picked a domain name that doesn’t exist in the wider Internet to ensure I don’t collide with anything real.</p> <h3 id="setting-it-up">Setting it up<a class="zola-anchor" style="display: none;" href="#setting-it-up" aria-hidden="true"><img height="16" width="16" src="/icons/link.svg" alt=""></a></h3> <p>dnsmasq doesn’t come installed by default, but it’s only an apt-get away.</p> <pre class="z-code"><code><span class="z-text z-plain">apt-get install dnsmasq </span></code></pre> <p>After that, there is a configuration file at /etc/dnsmasq.conf which you may want to have a look at. The defaults are mostly sensible. In my setup, I have changed the following options:</p> <pre class="z-code"><code><span class="z-text z-plain"># Set the domain name for hosts on this network </span><span class="z-text z-plain">domain=internal.lan </span><span class="z-text z-plain"># Set the start and end of the DHCP pool, and set the </span><span class="z-text z-plain"># default lease time to 24 hours </span><span class="z-text z-plain">dhcp-range=192.168.1.100,192.168.1.150,24h </span><span class="z-text z-plain"> </span><span class="z-text z-plain"># Set DHCP option 3 (which supplies the default route) </span><span class="z-text z-plain"># to the IP address of your router - otherwise dnsmasq </span><span class="z-text z-plain"># will assume that the gateway is the machine it is running on </span><span class="z-text z-plain">dhcp-option 3,192.168.1.1 </span><span class="z-text z-plain"> </span><span class="z-text z-plain"># Answer DNS queries based on the interface the request was sent </span><span class="z-text z-plain"># to - the effect of this means that you will never get 127.0.0.1 </span><span class="z-text z-plain"># returned when looking up the name of the dnsmasq server. </span><span class="z-text z-plain">localise-queries </span></code></pre> <p>You can also hard-code the MAC addresses of your machines into the dnsmasq.conf file, using the dhcp-host statement, so that each machine will always get the same IP address. This is a neat trick, but personally I don’t think that dnsmasq.conf is the right place to do that when there is already an <code>/etc/ethers</code> file that stores this information. So, for now, uncomment the “read-ethers” line in dnsmasq.conf. I’ll get back to setting up that file in a moment.</p> <p>Next, you want to set up your resolv.conf file. This will contain the IP addresses of the nameservers you want to use, and which dnsmasq will proxy for. In my case, I’m using OpenDNS, so I’ve grabbed the DNS servers from <a href="https://www.opendns.com/start">https://www.opendns.com/start</a>. My resolv.conf now looks like:</p> <pre class="z-code"><code><span class="z-text z-plain">domain internal.lan </span><span class="z-text z-plain">nameserver 208.67.222.222 </span><span class="z-text z-plain">nameserver 208.67.220.220 </span></code></pre> <p>To set up internal hosts on your LAN, you just need to enter them in <code>/etc/hosts</code>, and dnsmasq will read them from there. Beyond the normal entries for localhost and some IPv6 ones, I have things that look like:</p> <pre class="z-code"><code><span class="z-text z-plain">192.168.1.1 portal.internal.lan portal </span><span class="z-text z-plain">192.168.1.5 zoidberg.internal.lan zoidberg </span><span class="z-text z-plain">192.168.1.10 inara.internal.lan inara </span><span class="z-text z-plain">192.168.1.11 kaylee.internal.lan kaylee </span></code></pre> <p>You can add as many of these as you like, making sure that the domain name part of them matches what is in /etc/dnsmasq.conf, otherwise they may not work properly.</p> <p>Finally, back to <code>/etc/ethers</code>. This is a simple mapping between MAC addresses and either IP addresses or names, as long as the names can be resolved by either DNS or (in this case) the /etc/hosts file.</p> <pre class="z-code"><code><span class="z-text z-plain">00:12:34:56:78:9A zoidberg.planetexpress.lan </span><span class="z-text z-plain">00:BC:DE:F0:12:34 animal.planetexpress.lan </span></code></pre> <p>All the hosts in this file will be issued the same IP address every time they do a DHCP request, provided they also exist in <code>/etc/hosts</code>. Bear in mind that MAC addresses are per interface, not per machine, so if you have a laptop with a wired and a wireless interface I’d advise having two entries, one for each. Last of all, ensure that your router or other device isn’t running a DHCP server any more, and run <code>/etc/init.d/dnsmasq restart</code>. The new settings should take effect, and all being well, everything should work.</p> <p>(I have updated this based on feedback from <a href="http://www.dropshock.com/blog/">Jason Liquorish</a>, adding a bit about DHCP option 3 and the localise-queries option, which I forgot earlier. Thanks Jason!)</p>