MarcoPolo – Partially Functional - foundationsMarcoZola2021-03-07T00:00:00+00:00https://marcopolo.io/tags/foundations/atom.xmlBackups made simple2021-03-07T00:00:00+00:002021-03-07T00:00:00+00:00https://marcopolo.io/code/backups-made-simple/<p>I've made a backup system I can be proud of, and I'd like to share it with you
today. It follows a philosophy I've been fleshing out called <em>The
Functional Infra</em>. Concretely it aims to:</p>
<ul>
<li>Be pure. An output should only be a function of its inputs.</li>
<li>Be declarative and reproducible. A by product of being pure.</li>
<li>Support rollbacks. Also a by product of being pure.</li>
<li>Surface actionable errors. The corollary being it should be easy to understand
and observe what is happening.</li>
</ul>
<p>At a high level, the backup system works like so:</p>
<ol>
<li>ZFS creates automatic snapshots every so often.</li>
<li>Those snapshots are replicated to an EBS-backed EC2 instance that is only
alive while backup replication is happening. Taking advantage of ZFS'
incremental snapshot to make replication generally quite fast.</li>
<li>The EBS drive itself stays around after the instance is terminated. This
drive is a Cold HDD (sc1) which costs about $0.015 gb/month.</li>
</ol>
<h2 id="zfs">ZFS</h2>
<p>To be honest I haven't used ZFS all that much, but that's kind of my point. I,
as a non-expert in ZFS, have been able to get a lot out of it just by
following the straightforward documentation. It seems like the API is well
thought out and the semantics are reasonable. For example, a consistent snapshot
is as easy as doing <code>zfs snapshot tank/home/marco@friday</code>.</p>
<h3 id="automatic-snapshots">Automatic snapshots</h3>
<p>On NixOS setting up automatic snapshots is a breeze, just add the following to
your NixOS Configuration:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
services.zfs.autoSnapshot.enable = true;
}
</code></pre>
<p>and setting the <code>com.sun:auto-snapshot</code> option on the filesystem. E.g.: <code>zfs set com.sun:auto-snapshot=true <pool>/<fs></code>. Note that this can also be done on
creation of the filesystem: <code>zfs create -o mountpoint=legacy -o com.sun:auto-snapshot=true tank/home</code>.</p>
<p>With that enabled, ZFS will keep a snapshot for the latest 4 15-minute, 24
hourly, 7 daily, 4 weekly and 12 monthly snapshots.</p>
<h3 id="on-demand-ec2-instance-for-backups">On Demand EC2 Instance for Backups</h3>
<p>Now that we've demonstrated how to setup snapshotting, we need to tackle the
problem of replicating those snapshots somewhere so we can have real backups.
For that I use one of my favorite little tools:
<a href="https://github.com/stephank/lazyssh">lazyssh</a>. Its humble description betrays
little information at its true usefulness. The description is simply:
<em>A jump-host SSH server that starts machines on-demand</em>. What it enables is
pretty magical. It essentially lets you run arbitrary code when something SSHs
through the jump-host.</p>
<p>Let's take the classic ZFS replication example from the
<a href="https://docs.oracle.com/cd/E18752_01/html/819-5461/gbchx.html">docs</a>:
<code>host1# zfs send tank/dana@snap1 | ssh host2 zfs recv newtank/dana</code>. This
command copies a snapshot from a machine named <code>host1</code> to another machine named
<code>host2</code> over SSH. Simple and secure backups. But it relies on <code>host2</code> being
available. With <code>lazyssh</code> we can make <code>host2</code> only exist when needed.
<code>host2</code> would start when the ssh command is invoked and terminated when the ssh
command finishes. The command with <code>lazyssh</code> would look something like this
(assuming you have a <code>lazyssh</code> target in your <code>.ssh/config</code> as explained in the
<a href="https://github.com/stephank/lazyssh">docs</a>):</p>
<pre><code>host1# zfs send tank/dana@snap1 | ssh -J lazyssh host2 zfs recv newtank/dana
</code></pre>
<p>Note the only difference is the <code>-J lazyssh</code>.</p>
<p>So how do we actually setup <code>lazyssh</code> to do this? Here is my configuration:</p>
<div >
<script src="https://gist.github.com/MarcoPolo/13462e986711f62bfc6b7b8e494c5cc8.js"></script>
</div>
<p>Note there are a couple of setup steps:</p>
<ol>
<li>Create the initial sc1 EBS Drive. I did this in the AWS Console, but you
could do this in Terraform or the AWS CLI.</li>
<li>Create the ZFS pool on the drive. I launched my lazy archiver without the ZFS
filesystem option and ran: <code>zpool create -o ashift=12 -O mountpoint=none POOL_NAME /dev/DRIVE_LOCATION</code>. Then I created the
<code>POOL_NAME/backup</code> dataset with <code>zfs create -o acltype=posixacl -o xattr=sa -o mountpoint=legacy POOL_NAME/backup</code>.</li>
</ol>
<p>As a quality of life and security improvement I setup
<a href="https://github.com/nix-community/home-manager">homemanager</a> to manage my SSH
config and known_hosts file so these are automatically correct and properly
setup. I generate the lines for known_hosts when I generate the host keys
that go in the <code>user_data</code> field in the <code>lazsyssh-config.hcl</code> above. Here's the
relevant section from my homemanager config:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
programs.ssh = {
enable = true;
# I keep this file tracked in Git alongside my NixOS configs.
userKnownHostsFile = "/path/to/known_hosts";
matchBlocks = {
"archiver" = {
user = "root";
hostname = "archiver";
proxyJump = "lazyssh";
identityFile = "PATH_TO_AWS_KEYPAIR";
};
"lazyssh" = {
# This assume you are running lazyssh locally, but it can also
# reference another machine.
hostname = "localhost";
port = 7922;
user = "jump";
identityFile = "PATH_TO_LAZYSSH_CLIENT_KEY";
identitiesOnly = true;
extraOptions = {
"PreferredAuthentications" = "publickey";
};
};
};
};
}
</code></pre>
<p>Finally, I use the provided NixOS Module for <code>lazyssh</code> to manage starting it and
keeping it up. Here's the relevant parts from my <code>flake.nix</code>:</p>
<pre><code>{
# My fork that supports placements and terminating instances after failing to
# attach volume.
inputs.lazyssh.url = "github:marcopolo/lazyssh/attach-volumes";
inputs.lazyssh.inputs.nixpkgs.follows = "nixpkgs";
outputs =
{ self
, nixpkgs
, lazyssh
}: {
nixosConfigurations = {
nixMachineHostName = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
{
imports = [lazyssh.nixosModule]
services.lazyssh.configFile =
"/path/to/lazyssh-config.hcl";
# You'll need to add the correct AWS credentials to `/home/lazyssh/.aws`
# This could probably be a symlink with home-manager to a
# managed file somewhere else, but I haven't go down that path
# yet
users.users.lazyssh = {
isNormalUser = true;
createHome = true;
};
}
];
};
};
}
}
</code></pre>
<p>With all that setup, I can ssh into the archiver by simple running <code>ssh archiver</code>. Under the hood, <code>lazyssh</code> starts the EC2 instance and attaches the
EBS drive to it. And since <code>ssh archiver</code> works, so does the original example
of: <code>zfs send tank/dana@snap1 | ssh archiver zfs recv newtank/dana</code>.</p>
<h2 id="automatic-replication">Automatic Replication</h2>
<p>The next part of the puzzle is to have backups happen automatically. There are
various tools you can use for this. Even a simple cron that runs the <code>send/recv</code>
on a schedule. I opted to go for what NixOS supports out of the box, which is
<a href="https://github.com/alunduil/zfs-replicate">https://github.com/alunduil/zfs-replicate</a>.
Unfortunately, I ran into a couple issues that led me to make a fork. Namely:</p>
<ol>
<li>Using <code>/usr/bin/env - ssh</code> fails to use the ssh config file. My fork supports
specifying a custom ssh binary to use.</li>
<li>Support for <code>ExecStartPre</code>. This is to "warm up" the archiver instance. I run
<code>nixos-rebuild switch</code> which is basically a no-op if there is no changes to
apply from the configuration file, or blocks until the changes have been
applied. In my case these are usually the changes inside the UserData field.</li>
<li>Support for <code>ExecStopPost</code>. This is to add observability to this process.</li>
<li>I wanted to raise the systemd timeout limit. In case the <code>ExecStartPre</code> takes
a while to warm-up the instance.</li>
</ol>
<p>Thankfully with flakes, using my own fork was painless. Here's the relevant
section from my <code>flake.nix</code> file:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix"> # inputs.zfs-replicate.url = "github:marcopolo/zfs-replicate/flake";
# ...
# Inside nixosSystem modules...
({ pkgs, ... }:
{
imports = [ zfs-replicate.nixosModule ];
# Disable the existing module
disabledModules = [ "services/backup/zfs-replication.nix" ];
services.zfs.autoReplication =
let
host = "archiver";
sshPath = "${pkgs.openssh}/bin/ssh";
# Make sure the machine is up-to-date
execStartPre = "${sshPath} ${host} nixos-rebuild switch";
honeycombAPIKey = (import ./secrets.nix).honeycomb_api_key;
honeycombCommand = pkgs.writeScriptBin "reportResult" ''
#!/usr/bin/env ${pkgs.bash}/bin/bash
${pkgs.curl}/bin/curl https://api.honeycomb.io/1/events/zfs-replication -X POST \
-H "X-Honeycomb-Team: ${honeycombAPIKey}" \
-H "X-Honeycomb-Event-Time: $(${pkgs.coreutils}/bin/date -u +"%Y-%m-%dT%H:%M:%SZ")" \
-d "{\"serviceResult\":\"$SERVICE_RESULT\", \"exitCode\": \"$EXIT_CODE\", \"exitStatus\": \"$EXIT_STATUS\"}"
'';
execStopPost = "${honeycombCommand}/bin/reportResult";
in
{
inherit execStartPre execStopPost host sshPath;
enable = true;
timeout = 90000;
username = "root";
localFilesystem = "rpool/safe";
remoteFilesystem = "rpool/backup";
identityFilePath = "PATH_TO_AWS_KEY_PAIR";
};
})
</code></pre>
<p>That sets up a systemd service that runs after every snapshot. It also
reports the result of the replication to
<a href="https://www.honeycomb.io/">Honeycomb</a>, which brings us to our next
section...</p>
<h2 id="observability">Observability</h2>
<p>The crux of any automated process is it failing silently. This is especially bad
in the context of backups, since you don't need them until you do. I solved this
by reporting the result of the replication to Honeycomb after every run. It
reports the <code>$SERVICE_RESULT</code>, <code>$EXIT_CODE</code> and <code>$EXIT_STATUS</code> as returned by
systemd. I then create an alert that fires if there are no successful runs in
the past hour.</p>
<h2 id="future-work">Future Work</h2>
<p>While I like this system for being simple, I think there is a bit more work in
making it pure. For one, there should be no more than 1 manual step for setup,
and 1 manual step for tear down. There should also be a similar simplicity in
upgrading/downgrading storage space.</p>
<p>For reliability, the archiver instance should scrub its drive on a schedule.
This isn't setup yet.</p>
<p>At $0.015 gb/month this is relatively cheap, but not the cheapest. According to
<a href="https://filstats.com/">filstats</a> I could use
<a href="https://www.filecoin.com/">Filecoin</a> to store data for much less. There's no
Block Device interface to this yet, so it wouldn't be as simple as ZFS
<code>send/recv</code>. You'd lose the benefits of incremental snapshots. But it may be
possible to build a block device interface on top. Maybe with an <a href="https://en.wikipedia.org/wiki/Network_block_device">nbd-server</a>?</p>
<h2 id="extra">Extra</h2>
<p>Bits and pieces that may be helpful if you try setting something similar up.</p>
<h3 id="setting-host-key-and-nix-configuration-with-userdata">Setting host key and Nix Configuration with UserData</h3>
<p>NixOS on AWS has this undocumented nifty feature of setting the ssh host
key and a new <code>configuration.nix</code> file straight from the <a href="https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_UserData.html">UserData
field</a>.
This lets you one, be sure that your SSH connection isn't being
<a href="https://en.wikipedia.org/wiki/Man-in-the-middle_attack">MITM</a>, and two, configure
the machine in a simple way. I use this feature to set the SSH host key and set
the machine up with ZFS and the the <code>lz4</code> compression package.</p>
<h3 id="questions-comments">Questions? Comments?</h3>
<p>Email me if you set this system up. This is purposely not a tutorial, so you may
hit snags. If you think something could be clearer feel free to make an
<a href="https://github.com/marcopolo/marcopolo.github.io">edit</a>.</p>
Goodbye, bit rot2021-02-01T00:00:00+00:002021-02-01T00:00:00+00:00https://marcopolo.io/code/goodbye-bit-rot/<p>Take a look at this picture:</p>
<p><img src="https://marcopolo.io/code/goodbye-bit-rot/smalltalk-76.png" alt="Smalltalk 76" /></p>
<p>That's a photo of Smalltalk 76 running the prototypical desktop UI. It's
taken for granted that this photo will be viewable for the indefinite future
(or as long as we keep a PNG viewer around). But when we think about code,
maybe the very same Smalltalk code we took this photo of, it's assumed that
eventually that code will stop running. It'll stop working because of a
mysterious force known as <a href="https://en.wikipedia.org/wiki/Software_rot">bit
rot</a>. Why? It's this truly
inevitable? Or can we do better?</p>
<h2 id="we-can-do-better">We can do better</h2>
<p>Bit rot often manifests in the case where some software <em>A</em> relies on a certain
configured environment. Imagine <em>A</em> relies on a shared library <em>B</em>. As time
progresses, the shared library <em>B</em> can (and probably will) be updated
independently of <em>A</em>. Thus breaking <em>A</em>. But what if <em>A</em> could say it
explicitly depends on version <em>X.Y.Z</em> of <em>B</em>, or even better yet, the version
of the library that hashes to the value <code>0xBADCOFFEE</code>. Then you break the
implicit dependency of a correctly configured environment. <em>A</em> stops
depending on the world being in a certain state. Instead, <em>A</em>
<em>explicitly defines</em> what the world it needs should look like.</p>
<h2 id="enter-nix">Enter Nix</h2>
<p>This is what <a href="https://nixos.org/">Nix</a> gives you. A way to explicitly define
what a piece of software needs to build and run. Here's an example of the
definition on how to build the <a href="https://www.gnu.org/software/hello/">GNU
Hello</a> program:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">with (import <nixpkgs> {});
derivation {
name = "hello";
builder = "${bash}/bin/bash";
args = [ ./builder.sh ];
buildInputs = [ gnutar gzip gnumake gcc binutils-unwrapped coreutils gawk gnused gnugrep ];
src = ./hello-2.10.tar.gz;
system = builtins.currentSystem;
}
</code></pre>
<p>It's not necessary to explain this <a href="https://nixos.org/guides/nix-pills/generic-builders.html#idm140737320275008">code in
detail</a>.
It's enough to point out that <code>buildInputs</code> defines what the environment should
contain (i.e. it should contain <code>gnutar</code>, <code>gzip</code>, <code>gnumake</code>, etc.). And the
versions of these dependencies are defined by the current version of
<code><nixpkgs></code>. These dependencies can be further pinned (or <em>locked</em> in the
terminology of some languages like Javascript and Rust) to ensure that this
program will always be built with the same exact versions of its dependencies.
This extends to the runtime as well. This means you can run two different
programs that each rely on a different <code>glibc</code>. Or to bring it back to our
initial example, software <em>A</em> will always run because it will always use the
same exact shared library <em>B</em>.</p>
<h2 id="a-concrete-example-this-will-never-bit-rot">A concrete example. This will never bit rot.</h2>
<p>To continue our Smalltalk theme, here's a "Hello World" program that, barring a
fundamental change in how Nix Flakes works, will work forever<sup class="footnote-reference"><a href="#1">1</a></sup> on an x86_64
linux machine.</p>
<p>The definition of our program, <code>flake.nix</code></p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-20.09";
outputs =
{ self, nixpkgs }:
let
pkgs = nixpkgs.legacyPackages.x86_64-linux;
in
{
defaultPackage.x86_64-linux = pkgs.writeScriptBin "hello-smalltalk" ''
${pkgs.gnu-smalltalk}/bin/gst <<< "Transcript show: 'Hello World!'."
'';
};
}
</code></pre>
<p>The pinned version of all our dependencies, <code>flake.lock</code></p>
<pre data-lang="json" class="language-json "><code class="language-json" data-lang="json">{
"nodes": {
"nixpkgs": {
"locked": {
"lastModified": 1606669556,
"narHash": "sha256-9rlqZ5JwnA6nK04vKhV0s5ndepnWL5hpkaTV1b4ASvk=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "ae47c79479a086e96e2977c61e538881913c0c08",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixos-20.09",
"repo": "nixpkgs",
"type": "github"
}
},
"root": {
"inputs": {
"nixpkgs": "nixpkgs"
}
}
},
"root": "root",
"version": 7
}
</code></pre>
<p>copy those files into a directory and run it:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">❯ nix run
Hello World!
</code></pre>
<h2 id="solid-foundations">Solid Foundations</h2>
<p>With Nix, we can make steady forward progress. Without fear that our foundations
will collapse under us like sand castles. Once we've built something in Nix we
can be pretty sure it will work for our colleague or ourselves in 10 years. Nix
is building a solid foundation that I can no longer live without.</p>
<p>If you haven't used Nix before, here's your call to action:</p>
<ul>
<li>Nix's homepage: <a href="https://nixos.org/">https://nixos.org/</a></li>
<li>Nix's Learning page: <a href="https://nixos.org/learn">https://nixos.org/learn</a></li>
<li>Learn Nix in little bite-sized pills: <a href="https://nixos.org/guides/nix-pills/">https://nixos.org/guides/nix-pills/</a></li>
</ul>
<hr />
<h2 id="disclaimer">Disclaimer</h2>
<p>There are various factors that lead to bit rot. Some are easier to solve than
others. For the purpose of this post I'm only considering programs that are
roughly self contained. For example, if a program relies on hitting a specific
Google endpoint, the only way to use this program would be to emulate the whole
Google stack or rely on that <a href="https://gcemetery.co/">endpoint existing</a>.
Sometimes it's doable to emulate the external API, and sometimes it isn't. This
post is specifically about cases where it is feasible to emulate the external API.</p>
<h3 id="footnotes">Footnotes</h3>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>Okay forever is a really long time. And this will likely not run forever. But why? The easy reasons are: "Github is down", "A source tarball you need can't be fetched from the internet", "x86_64 processors can't be found or emulated". But what's a weird reason that this may fail in the future? It'll probably be hard to predict, but maybe something like: SHA256 has been broken and criminals and/or pranksters have published malicious packages that match a certain SHA256. So build tools that rely on a deterministic and hard to break hash algorithm like SHA256 (like what Nix does) will no longer be reliable. That would be a funny future. Send me your weird reasons: <code>"marco+forever" ++ "@marcopolo.io"</code></p>
</div>