MarcoPolo – Partially FunctionalMarcoZola2021-06-13T00:00:00+00:00https://marcopolo.io/atom.xmlNew Keyboard2021-06-13T00:00:00+00:002021-06-13T00:00:00+00:00https://marcopolo.io/life/keyboard/<p>I got around to building this new keyboard. It's essentially the <a href="https://twitter.com/adereth/status/852975999905964032">lightcycle
version</a> of the <a href="https://github.com/adereth/dactyl-keyboard">Dactyl
keyboard</a>. The few differences are
that the space and port for the board is bigger to accommodate a
<a href="https://nicekeyboards.com/nice-nano/">nice!nano</a> board with pins.</p>
<p><img src="/life/kb1.jpg" alt="My new keyboard" /></p>
<p>The keyboard is wireless thanks to the nice!nano. The board has a USB-C
connector, but I added these <a href="https://www.amazon.com/Magnetic-Charging-CAFELE-Braided-Compatible/dp/B08BLJ62J8">magnetic charging and data
cables</a>
to make charging and connecting easy and lessen the force on the port on the
board.</p>
<p><img src="/life/port.jpg" alt="Charging port" /></p>
<p>I can also sling the keyboard halves with some cord and wear it as a stylish
scarf.</p>
<p><img src="/life/carry.jpg" alt="Slung keyboard" /></p>
Declarative Dev Environments2021-05-10T00:00:00+00:002021-05-10T00:00:00+00:00https://marcopolo.io/code/declarative-dev-environments/<p>I don't install development tools globally. I don't have <code>node</code> added to my
<code>PATH</code> in my <code>~/.zshrc</code> file, and running <code>cargo</code> outside a project folder
returns "command not found." I wipe my computer on every reboot. With the
exception of four folders (<code>/boot</code>, <code>/nix</code>, <code>/home</code>, and <code>/persist</code>), everything
gets <a href="https://grahamc.com/blog/erase-your-darlings">deleted</a>. And it has worked
out great.</p>
<p>Instead of installing development packages globally, I declare them as a
dependency in my project's dev environment. They become available as soon as I
<code>cd</code> into the project folder. If two projects use the same tool then I only keep
one version of that tool on my computer.</p>
<p>I think installing dev tools globally is a bad pattern that leads to nothing but
heartache and woe. If you are running <code>sudo apt-get install</code> or <code>brew install</code>
prior to building a project, you are doing it wrong. By defining your dev tool
dependencies explicitly you allow your projects to easily build on any
machine at any point in time. Whether it's on a friends machine today, or a new
laptop in 10 years. It even makes CI integration a breeze.</p>
<h2 id="what-do-i-mean-by-a-declarative-dev-environment">What do I mean by a declarative dev environment?</h2>
<p>I mean a project that has a special file (or files) that define all the
dependencies required to build and run your project. It doesn't necessarily have
to include the actual binaries you will run in the repo, but it should be
reproducible. If you clone my project you should be running the exact
same tools as me.</p>
<p>Just like you have explicit dependencies on libraries you use in your program, a
declarative dev environment lets you define your tooling dependencies (e.g.
which version of Node, Yarn, or your specific cross compiler toolchain).</p>
<h2 id="how-i-setup-my-declarative-dev-environments">How I setup my declarative dev environments</h2>
<p>To accomplish this I use <a href="https://nixos.org">Nix</a> with <a href="https://www.tweag.io/blog/2020-05-25-flakes/">Nix Flakes</a> and <a href="https://direnv.net/">direnv</a>. There are three
relevant files: <code>flake.nix</code> which defines the build of the project and the tools
I need for development; <code>flake.lock</code> which is similar in spirit to a <code>yarn.lock</code>
or <code>Cargo.lock</code> file, it <em>locks</em> the exact version of any tool used and
generated automatically the first time you introduce dependencies; and finally a
<code>.envrc</code> file which simply tells direnv to ask Nix what the environment should
be, and sets up the environment when you <code>cd</code> into the folder. Here are some
simple examples:
<a href="https://github.com/MarcoPolo/templates/tree/master/trivial">flake.nix</a>,
<a href="https://github.com/MarcoPolo/templates/blob/master/trivial/.envrc">.envrc</a>
(<code>flake.lock</code> omitted since it's automatically generated).</p>
<p>As a shortcut for setting up a <code>flake.nix</code> and <code>.envrc</code>, you can use a template
to provide the boilerplate. When I start a new project I'll run <code>nix flake init -t github:marcopolo/templates</code> which copies the files from this
<a href="https://github.com/MarcoPolo/templates/tree/master/trivial">repo</a> and puts them
in your current working directory. Then running <code>direnv allow</code> will setup your
local environment, installing any missing dependencies through Nix as a side
effect.</p>
<p>This blog itself makes use of <a href="https://github.com/MarcoPolo/marcopolo.github.io/blob/master/flake.nix#L14">declarative dev
environments</a>.
Zola is the static site generator I use. When I <code>cd</code> into my blog my environment
is automatically setup with Zola available for previewing the blog.</p>
<h2 id="how-nix-works-roughly">How Nix works, roughly</h2>
<p>This all works off <a href="https://nixos.org">Nix</a>. Nix is a fantastic package manager and build tool that
provides reproducible versions of packages that don't rely on a specific global
system configuration. Specifically packages installed through Nix don't rely an
a user's <code>/usr/lib</code> or anything outside of <code>/nix/store</code>. You don't even need
glibc installed (as may be the case if you are on <a href="https://www.alpinelinux.org/">Alpine
Linux</a>).</p>
<p>For a deeper dive see <a href="https://nixos.org/guides/how-nix-works.html">How Nix Works</a>.</p>
<h2 id="an-example-how-to-setup-a-yarn-based-js-project">An example, how to setup a Yarn based JS project.</h2>
<p>To be concrete, let me show an example. If I wanted to start a JS project and
use <a href="https://yarnpkg.com/">Yarn</a> as my dependency manager, I would do something
like this: </p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash"># 1. Create the project folder
mkdir my-project
# 2. Add the boilerplate files.
nix flake init -t github:marcopolo/templates
# 3. Edit flake.nix file to add yarn and NodeJS.
# With your text editor apply this diff:
# - buildInputs = [ pkgs.hello ];
# + buildInputs = [ pkgs.yarn pkgs.nodejs-12_x ];
# 4. Allow direnv to run this environment. This will also fetch yarn with Nix
# and add it to your path.
direnv allow
# 5. Yarn is now available, proceed as normal.
yarn init
</code></pre>
<p>You can simplify this further by making a Nix Flake template that already has
Yarn and NodeJS included. </p>
<h2 id="another-example-setting-up-a-rust-project">Another example. Setting up a Rust project.</h2>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash"># 1. Create the project folder
mkdir rust-project
# 2. Add the boilerplate files.
nix flake init -t github:marcopolo/templates#rust
# 3. Cargo and rust is now available, proceed as normal.
cargo init
cargo run
</code></pre>
<p>Here we used a Rust specific template, so no post template init changes were required.</p>
<h2 id="dissecting-the-flake-nix-file">Dissecting the <code>flake.nix</code> file</h2>
<p>Let's break down the <code>flake.nix</code> file so we can understand what it is we are
declaring.</p>
<p>First off, the file is written in <a href="https://nixos.wiki/wiki/Nix_Expression_Language">Nix, the programming
language</a>. At a high level you
can read this as JSON but with functions. Like JSON it can only represent
expressions (you can only have one top level JSON object), unlike JSON you can
have functions and variables. </p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix"># This is our top level set expression. Equivalent to the top level JSON object.
{
# These are comments
# Here we are defining a set. This is equivalent to a JSON object.
# The key is description, and the value is the string.
description = "A very basic flake";
# You can define nested sets by using a `.` between key parts.
# This is equivalent to the JSON object {inputs: {flake-utils: {url: "github:..."}}}
inputs.flake-utils.url = "github:numtide/flake-utils";
# Functions are defined with the syntax of `param: functionBodyExpression`.
# The param can be destructured if it expects a set, like what we are doing here.
# This defines the output of this flake. Our dev environment will make use of
# the devShell attribute, but you can also define the release build of your
# package here.
outputs = { self, nixpkgs, flake-utils }:
# This is a helper to generate these outputs for each system (x86-linux,
# arm-linux, macOS, ...)
flake-utils.lib.eachDefaultSystem (system:
let
# The nixpkgs repo has to know which system we are using.
pkgs = import nixpkgs { system = system; };
in
{
# This is the environment that direnv will use. You can also enter the
# shell with `nix shell`. The packages in `buildInputs` are what become
# available to you in your $PATH. As an example this only has the hello
# package.
devShell = pkgs.mkShell {
buildInputs = [ pkgs.hello ];
};
# You can also define a package that is built by default when you run
# `nix build`. The build command creates a new folder, `result`, that
# is a symlink to the build output.
defaultPackage = pkgs.hello;
});
}
</code></pre>
<h2 id="on-dev-tools-and-a-dev-setup">On Dev Tools and A Dev Setup</h2>
<p>There is a subtle distinction on what constitutes a Dev Tool vs A Dev Setup. I
classify Dev Tools as things that need to be available to build or develop a given
project specifically. Think of <code>gcc</code>, <code>yarn</code>, or <code>cargo</code>. The Dev Setup category
are for things that are useful when developing in general. Vim, Emacs,
<a href="https://geoff.greer.fm/ag/">ag</a> are some examples.</p>
<p>Dev tools are worth defining explicitly in your project's declarative dev environment (in
a <code>flake.nix</code> file). A Dev Setup is highly personal and not worth defining in the
project's declarative dev environment. But that's not to say your dev setup in not
worth defining at all. In fact, if you are (or when you become) familiar with
Nix, you can extend the same ideas of this post to your user account with <a href="https://github.com/nix-community/home-manager">Home
Manager</a>. </p>
<p>With Home Manager You can declaratively define which programs you want available
in your dev setup, what Vim plugins you want installed, what ZSH plugins you
want available and much more. It's the core idea of declarative dev environments
taken to the user account level.</p>
<h2 id="why-not-docker">Why not Docker?</h2>
<p>Many folks use Docker to get something like this, but while it gets close – and
in some cases functionally equivalent – it has some shortcomings:</p>
<p>For one, a Dockerfile is not reproducible out of the box. It is common to use
<code>apt-get install</code> in a Dockerfile to add packages. This part isn't reproducible
and brings you back to the initial problem I outlined. </p>
<p>Docker is less effecient with storage. It uses layers as the base block of
Docker images rather than packages. This means that it's relatively easy to end
up with many similar docker images (for a more thorough analysis check
out <a href="https://grahamc.com/blog/nix-and-layered-docker-images">Optimising Docker Layers for Better Caching with
Nix</a>).</p>
<p>Spinning up a container and doing development inside may not leverage your
existing dev setup. For example you may have Vim setup neatly on your machine,
but resort to <code>vi</code> when developing inside a container. Or worse, you'll
rebuild your dev setup inside the container, which does nothing more than
add dead weight to the container since it's an addition solely for you and not
really part of the project. Of course there are some workarounds to this issue,
you can bind mount a folder and VS Code supports opening a project inside a
container. <a href="https://github.com/zmkfirmware/zmk">ZMK</a> does this and it has
worked great.</p>
<p>If you are on MacOS, developing inside a container is actually slower. Docker
on Mac relies on running a linux VM in the background and running containers in
that VM. By default that VM is underpowered relative to the host MacOS machine.</p>
<p>There are cases where you actually do only want to run the code in an
x86-linux environment and Docker provides a convenient proxy for this. In these
cases I'd suggest using Nix to generate the Docker images. This way you get the
declarative and reproducible properties from Nix and the convenience from Docker.</p>
<p>As a caveat to all of the above, if you already have a reproducible dev environment
with a Docker container that works for you, please don't throw that all out and
redesign your system from scratch. Keep using it until it stops meeting your
needs and come back to this when it happens. Until then, keep building.</p>
<h2 id="on-nix-flakes">On Nix Flakes</h2>
<p>Nix Flakes is still new and in beta, so it's likely that if you install Nix from
their <a href="https://nixos.org/download.html">download page</a> you won't have Nix Flakes
available. If you don't already have Nix installed, you can install a version
with Nix Flakes <a href="https://github.com/numtide/nix-unstable-installer">with the unstable installer</a>,
otherwise read the section on <a href="https://nixos.wiki/wiki/Flakes#Installing_flakes">installing flakes</a>.</p>
<h2 id="closing-thoughts">Closing thoughts</h2>
<p>In modern programming languages we define all our dependencies explicitly and
lock the specific versions used. It's about time we do that for all our tools
too. Let's get rid of the <code>apt-get install</code> and <code>brew install</code> section of READMEs.</p>
Simple Declarative VMs2021-03-24T00:00:00+00:002021-03-24T00:00:00+00:00https://marcopolo.io/code/simple-vms/<p>I've been on a hunt to find a simple and declarative way to define VMs. I wanted
something like <a href="https://nixos.org/manual/nixos/stable/#ch-containers">NixOS
Containers</a>, but with a
stronger security guarantee. I wanted to be able to use a Nix expression to
define what the VM should look like, then reference that on my Server's
expression and have it all work automatically. I didn't want to manually
run any commands. The hunt is over, I finally found it.</p>
<h2 id="my-use-case">My Use Case</h2>
<p>I want a machine that I can permanently hook up to a WireGuard VPN and treat
as if it were in a remote place. At first I did this with a physical machine,
but I didn't want to commit the whole machine's compute for a novelty. What I
really want is a small VM that is permanently hooked up to a WireGuard VPN.
Minimal investment with all the upsides.</p>
<h2 id="nixos-qemu">NixOS QEMU</h2>
<p>Nix OS supports building your system in a QEMU runnable environment right out of
the box. <code>nixos-rebuild build-vm</code> is a wrapper over <code>nix build github:marcopolo/marcopolo.github.io#nixosConfigurations.small-vm.config.system.build.vm</code>. (Side note, with
flakes you can build this exact VM by running that command<sup class="footnote-reference"><a href="#1">1</a></sup>). This means NixOS
already did the hard work of turning a NixOS configuration into a valid VM that
can be launched with QEMU. Not only that, but the VM shares the <code>/nix/store</code>
with the host. This results in a really small VM (disk size is 5MB).</p>
<p>NixOS does the heavy lifting of converting a configuration into a script that
will run a VM, so all I need to do is write a service that manages this process.
Enter <a href="https://github.com/MarcoPolo/simple-vms/">simple-vms</a>, heavily inspired by
<a href="https://github.com/Nekroze/vms.nix">vms.nix</a> and
<a href="https://github.com/Mic92/nixos-shell">nixos-shell</a>. <a href="https://github.com/MarcoPolo/simple-vms/">simple-vms</a> is a NixOS
module that takes in a reference to the
<code>nixosConfigurations.small-vm.config.system.build.vm</code> derivation and the
option of whether you want state to be persisted, and defines a Systemd
service for the vm (There can be multiple VMs). This really is a simple
module, the NixOS service definition is about 10 lines long, and its
<code>ExecStart</code> is simply:</p>
<pre><code>mkdir -p /var/lib/simple-vms/${name}
cd /var/lib/simple-vms/${name}
exec ${cfg.vm.out}/bin/run-nixos-vm;
</code></pre>
<p>With this service we can get and keep our VMs up and running.</p>
<h2 id="stateless-vms">Stateless VMs</h2>
<p>I got a sticker recently that said "You either have one source of truth, of
multiple sources of lies." To that end, I wanted to make my VM completely
stateless. QEMU lets you mount folders into the VM, so I used that to mount host
folders in the VM's <code>/etc/wireguard</code> and <code>/etc/ssh</code> so that the host can
provide the VM with WireGuard keys, and the VM can persist it's SSH host keys.</p>
<p>That's all the VM really needs. Every time my VM shuts down I delete the drive.
And just to be safe, I try deleting any drive on boot too.</p>
<p>If you're running a service on the VM, you'll likely want to persist that
service's state files too in a similar way.</p>
<h2 id="fin">Fin</h2>
<p>That's it. Just a small post for a neat little trick. If you set this up let
me know! I'm interested in hearing your use case.</p>
<h3 id="footnotes">Footnotes</h3>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>User/pass = root/root. Exit qemu with C-a x.</p>
</div>
Backups made simple2021-03-07T00:00:00+00:002021-03-07T00:00:00+00:00https://marcopolo.io/code/backups-made-simple/<p>I've made a backup system I can be proud of, and I'd like to share it with you
today. It follows a philosophy I've been fleshing out called <em>The
Functional Infra</em>. Concretely it aims to:</p>
<ul>
<li>Be pure. An output should only be a function of its inputs.</li>
<li>Be declarative and reproducible. A by product of being pure.</li>
<li>Support rollbacks. Also a by product of being pure.</li>
<li>Surface actionable errors. The corollary being it should be easy to understand
and observe what is happening.</li>
</ul>
<p>At a high level, the backup system works like so:</p>
<ol>
<li>ZFS creates automatic snapshots every so often.</li>
<li>Those snapshots are replicated to an EBS-backed EC2 instance that is only
alive while backup replication is happening. Taking advantage of ZFS'
incremental snapshot to make replication generally quite fast.</li>
<li>The EBS drive itself stays around after the instance is terminated. This
drive is a Cold HDD (sc1) which costs about $0.015 gb/month.</li>
</ol>
<h2 id="zfs">ZFS</h2>
<p>To be honest I haven't used ZFS all that much, but that's kind of my point. I,
as a non-expert in ZFS, have been able to get a lot out of it just by
following the straightforward documentation. It seems like the API is well
thought out and the semantics are reasonable. For example, a consistent snapshot
is as easy as doing <code>zfs snapshot tank/home/marco@friday</code>.</p>
<h3 id="automatic-snapshots">Automatic snapshots</h3>
<p>On NixOS setting up automatic snapshots is a breeze, just add the following to
your NixOS Configuration:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
services.zfs.autoSnapshot.enable = true;
}
</code></pre>
<p>and setting the <code>com.sun:auto-snapshot</code> option on the filesystem. E.g.: <code>zfs set com.sun:auto-snapshot=true <pool>/<fs></code>. Note that this can also be done on
creation of the filesystem: <code>zfs create -o mountpoint=legacy -o com.sun:auto-snapshot=true tank/home</code>.</p>
<p>With that enabled, ZFS will keep a snapshot for the latest 4 15-minute, 24
hourly, 7 daily, 4 weekly and 12 monthly snapshots.</p>
<h3 id="on-demand-ec2-instance-for-backups">On Demand EC2 Instance for Backups</h3>
<p>Now that we've demonstrated how to setup snapshotting, we need to tackle the
problem of replicating those snapshots somewhere so we can have real backups.
For that I use one of my favorite little tools:
<a href="https://github.com/stephank/lazyssh">lazyssh</a>. Its humble description betrays
little information at its true usefulness. The description is simply:
<em>A jump-host SSH server that starts machines on-demand</em>. What it enables is
pretty magical. It essentially lets you run arbitrary code when something SSHs
through the jump-host.</p>
<p>Let's take the classic ZFS replication example from the
<a href="https://docs.oracle.com/cd/E18752_01/html/819-5461/gbchx.html">docs</a>:
<code>host1# zfs send tank/dana@snap1 | ssh host2 zfs recv newtank/dana</code>. This
command copies a snapshot from a machine named <code>host1</code> to another machine named
<code>host2</code> over SSH. Simple and secure backups. But it relies on <code>host2</code> being
available. With <code>lazyssh</code> we can make <code>host2</code> only exist when needed.
<code>host2</code> would start when the ssh command is invoked and terminated when the ssh
command finishes. The command with <code>lazyssh</code> would look something like this
(assuming you have a <code>lazyssh</code> target in your <code>.ssh/config</code> as explained in the
<a href="https://github.com/stephank/lazyssh">docs</a>):</p>
<pre><code>host1# zfs send tank/dana@snap1 | ssh -J lazyssh host2 zfs recv newtank/dana
</code></pre>
<p>Note the only difference is the <code>-J lazyssh</code>.</p>
<p>So how do we actually setup <code>lazyssh</code> to do this? Here is my configuration:</p>
<div >
<script src="https://gist.github.com/MarcoPolo/13462e986711f62bfc6b7b8e494c5cc8.js"></script>
</div>
<p>Note there are a couple of setup steps:</p>
<ol>
<li>Create the initial sc1 EBS Drive. I did this in the AWS Console, but you
could do this in Terraform or the AWS CLI.</li>
<li>Create the ZFS pool on the drive. I launched my lazy archiver without the ZFS
filesystem option and ran: <code>zpool create -o ashift=12 -O mountpoint=none POOL_NAME /dev/DRIVE_LOCATION</code>. Then I created the
<code>POOL_NAME/backup</code> dataset with <code>zfs create -o acltype=posixacl -o xattr=sa -o mountpoint=legacy POOL_NAME/backup</code>.</li>
</ol>
<p>As a quality of life and security improvement I setup
<a href="https://github.com/nix-community/home-manager">homemanager</a> to manage my SSH
config and known_hosts file so these are automatically correct and properly
setup. I generate the lines for known_hosts when I generate the host keys
that go in the <code>user_data</code> field in the <code>lazsyssh-config.hcl</code> above. Here's the
relevant section from my homemanager config:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
programs.ssh = {
enable = true;
# I keep this file tracked in Git alongside my NixOS configs.
userKnownHostsFile = "/path/to/known_hosts";
matchBlocks = {
"archiver" = {
user = "root";
hostname = "archiver";
proxyJump = "lazyssh";
identityFile = "PATH_TO_AWS_KEYPAIR";
};
"lazyssh" = {
# This assume you are running lazyssh locally, but it can also
# reference another machine.
hostname = "localhost";
port = 7922;
user = "jump";
identityFile = "PATH_TO_LAZYSSH_CLIENT_KEY";
identitiesOnly = true;
extraOptions = {
"PreferredAuthentications" = "publickey";
};
};
};
};
}
</code></pre>
<p>Finally, I use the provided NixOS Module for <code>lazyssh</code> to manage starting it and
keeping it up. Here's the relevant parts from my <code>flake.nix</code>:</p>
<pre><code>{
# My fork that supports placements and terminating instances after failing to
# attach volume.
inputs.lazyssh.url = "github:marcopolo/lazyssh/attach-volumes";
inputs.lazyssh.inputs.nixpkgs.follows = "nixpkgs";
outputs =
{ self
, nixpkgs
, lazyssh
}: {
nixosConfigurations = {
nixMachineHostName = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
{
imports = [lazyssh.nixosModule]
services.lazyssh.configFile =
"/path/to/lazyssh-config.hcl";
# You'll need to add the correct AWS credentials to `/home/lazyssh/.aws`
# This could probably be a symlink with home-manager to a
# managed file somewhere else, but I haven't go down that path
# yet
users.users.lazyssh = {
isNormalUser = true;
createHome = true;
};
}
];
};
};
}
}
</code></pre>
<p>With all that setup, I can ssh into the archiver by simple running <code>ssh archiver</code>. Under the hood, <code>lazyssh</code> starts the EC2 instance and attaches the
EBS drive to it. And since <code>ssh archiver</code> works, so does the original example
of: <code>zfs send tank/dana@snap1 | ssh archiver zfs recv newtank/dana</code>.</p>
<h2 id="automatic-replication">Automatic Replication</h2>
<p>The next part of the puzzle is to have backups happen automatically. There are
various tools you can use for this. Even a simple cron that runs the <code>send/recv</code>
on a schedule. I opted to go for what NixOS supports out of the box, which is
<a href="https://github.com/alunduil/zfs-replicate">https://github.com/alunduil/zfs-replicate</a>.
Unfortunately, I ran into a couple issues that led me to make a fork. Namely:</p>
<ol>
<li>Using <code>/usr/bin/env - ssh</code> fails to use the ssh config file. My fork supports
specifying a custom ssh binary to use.</li>
<li>Support for <code>ExecStartPre</code>. This is to "warm up" the archiver instance. I run
<code>nixos-rebuild switch</code> which is basically a no-op if there is no changes to
apply from the configuration file, or blocks until the changes have been
applied. In my case these are usually the changes inside the UserData field.</li>
<li>Support for <code>ExecStopPost</code>. This is to add observability to this process.</li>
<li>I wanted to raise the systemd timeout limit. In case the <code>ExecStartPre</code> takes
a while to warm-up the instance.</li>
</ol>
<p>Thankfully with flakes, using my own fork was painless. Here's the relevant
section from my <code>flake.nix</code> file:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix"> # inputs.zfs-replicate.url = "github:marcopolo/zfs-replicate/flake";
# ...
# Inside nixosSystem modules...
({ pkgs, ... }:
{
imports = [ zfs-replicate.nixosModule ];
# Disable the existing module
disabledModules = [ "services/backup/zfs-replication.nix" ];
services.zfs.autoReplication =
let
host = "archiver";
sshPath = "${pkgs.openssh}/bin/ssh";
# Make sure the machine is up-to-date
execStartPre = "${sshPath} ${host} nixos-rebuild switch";
honeycombAPIKey = (import ./secrets.nix).honeycomb_api_key;
honeycombCommand = pkgs.writeScriptBin "reportResult" ''
#!/usr/bin/env ${pkgs.bash}/bin/bash
${pkgs.curl}/bin/curl https://api.honeycomb.io/1/events/zfs-replication -X POST \
-H "X-Honeycomb-Team: ${honeycombAPIKey}" \
-H "X-Honeycomb-Event-Time: $(${pkgs.coreutils}/bin/date -u +"%Y-%m-%dT%H:%M:%SZ")" \
-d "{\"serviceResult\":\"$SERVICE_RESULT\", \"exitCode\": \"$EXIT_CODE\", \"exitStatus\": \"$EXIT_STATUS\"}"
'';
execStopPost = "${honeycombCommand}/bin/reportResult";
in
{
inherit execStartPre execStopPost host sshPath;
enable = true;
timeout = 90000;
username = "root";
localFilesystem = "rpool/safe";
remoteFilesystem = "rpool/backup";
identityFilePath = "PATH_TO_AWS_KEY_PAIR";
};
})
</code></pre>
<p>That sets up a systemd service that runs after every snapshot. It also
reports the result of the replication to
<a href="https://www.honeycomb.io/">Honeycomb</a>, which brings us to our next
section...</p>
<h2 id="observability">Observability</h2>
<p>The crux of any automated process is it failing silently. This is especially bad
in the context of backups, since you don't need them until you do. I solved this
by reporting the result of the replication to Honeycomb after every run. It
reports the <code>$SERVICE_RESULT</code>, <code>$EXIT_CODE</code> and <code>$EXIT_STATUS</code> as returned by
systemd. I then create an alert that fires if there are no successful runs in
the past hour.</p>
<h2 id="future-work">Future Work</h2>
<p>While I like this system for being simple, I think there is a bit more work in
making it pure. For one, there should be no more than 1 manual step for setup,
and 1 manual step for tear down. There should also be a similar simplicity in
upgrading/downgrading storage space.</p>
<p>For reliability, the archiver instance should scrub its drive on a schedule.
This isn't setup yet.</p>
<p>At $0.015 gb/month this is relatively cheap, but not the cheapest. According to
<a href="https://filstats.com/">filstats</a> I could use
<a href="https://www.filecoin.com/">Filecoin</a> to store data for much less. There's no
Block Device interface to this yet, so it wouldn't be as simple as ZFS
<code>send/recv</code>. You'd lose the benefits of incremental snapshots. But it may be
possible to build a block device interface on top. Maybe with an <a href="https://en.wikipedia.org/wiki/Network_block_device">nbd-server</a>?</p>
<h2 id="extra">Extra</h2>
<p>Bits and pieces that may be helpful if you try setting something similar up.</p>
<h3 id="setting-host-key-and-nix-configuration-with-userdata">Setting host key and Nix Configuration with UserData</h3>
<p>NixOS on AWS has this undocumented nifty feature of setting the ssh host
key and a new <code>configuration.nix</code> file straight from the <a href="https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_UserData.html">UserData
field</a>.
This lets you one, be sure that your SSH connection isn't being
<a href="https://en.wikipedia.org/wiki/Man-in-the-middle_attack">MITM</a>, and two, configure
the machine in a simple way. I use this feature to set the SSH host key and set
the machine up with ZFS and the the <code>lz4</code> compression package.</p>
<h3 id="questions-comments">Questions? Comments?</h3>
<p>Email me if you set this system up. This is purposely not a tutorial, so you may
hit snags. If you think something could be clearer feel free to make an
<a href="https://github.com/marcopolo/marcopolo.github.io">edit</a>.</p>
Goodbye, bit rot2021-02-01T00:00:00+00:002021-02-01T00:00:00+00:00https://marcopolo.io/code/goodbye-bit-rot/<p>Take a look at this picture:</p>
<p><img src="https://marcopolo.io/code/goodbye-bit-rot/smalltalk-76.png" alt="Smalltalk 76" /></p>
<p>That's a photo of Smalltalk 76 running the prototypical desktop UI. It's
taken for granted that this photo will be viewable for the indefinite future
(or as long as we keep a PNG viewer around). But when we think about code,
maybe the very same Smalltalk code we took this photo of, it's assumed that
eventually that code will stop running. It'll stop working because of a
mysterious force known as <a href="https://en.wikipedia.org/wiki/Software_rot">bit
rot</a>. Why? It's this truly
inevitable? Or can we do better?</p>
<h2 id="we-can-do-better">We can do better</h2>
<p>Bit rot often manifests in the case where some software <em>A</em> relies on a certain
configured environment. Imagine <em>A</em> relies on a shared library <em>B</em>. As time
progresses, the shared library <em>B</em> can (and probably will) be updated
independently of <em>A</em>. Thus breaking <em>A</em>. But what if <em>A</em> could say it
explicitly depends on version <em>X.Y.Z</em> of <em>B</em>, or even better yet, the version
of the library that hashes to the value <code>0xBADCOFFEE</code>. Then you break the
implicit dependency of a correctly configured environment. <em>A</em> stops
depending on the world being in a certain state. Instead, <em>A</em>
<em>explicitly defines</em> what the world it needs should look like.</p>
<h2 id="enter-nix">Enter Nix</h2>
<p>This is what <a href="https://nixos.org/">Nix</a> gives you. A way to explicitly define
what a piece of software needs to build and run. Here's an example of the
definition on how to build the <a href="https://www.gnu.org/software/hello/">GNU
Hello</a> program:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">with (import <nixpkgs> {});
derivation {
name = "hello";
builder = "${bash}/bin/bash";
args = [ ./builder.sh ];
buildInputs = [ gnutar gzip gnumake gcc binutils-unwrapped coreutils gawk gnused gnugrep ];
src = ./hello-2.10.tar.gz;
system = builtins.currentSystem;
}
</code></pre>
<p>It's not necessary to explain this <a href="https://nixos.org/guides/nix-pills/generic-builders.html#idm140737320275008">code in
detail</a>.
It's enough to point out that <code>buildInputs</code> defines what the environment should
contain (i.e. it should contain <code>gnutar</code>, <code>gzip</code>, <code>gnumake</code>, etc.). And the
versions of these dependencies are defined by the current version of
<code><nixpkgs></code>. These dependencies can be further pinned (or <em>locked</em> in the
terminology of some languages like Javascript and Rust) to ensure that this
program will always be built with the same exact versions of its dependencies.
This extends to the runtime as well. This means you can run two different
programs that each rely on a different <code>glibc</code>. Or to bring it back to our
initial example, software <em>A</em> will always run because it will always use the
same exact shared library <em>B</em>.</p>
<h2 id="a-concrete-example-this-will-never-bit-rot">A concrete example. This will never bit rot.</h2>
<p>To continue our Smalltalk theme, here's a "Hello World" program that, barring a
fundamental change in how Nix Flakes works, will work forever<sup class="footnote-reference"><a href="#1">1</a></sup> on an x86_64
linux machine.</p>
<p>The definition of our program, <code>flake.nix</code></p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-20.09";
outputs =
{ self, nixpkgs }:
let
pkgs = nixpkgs.legacyPackages.x86_64-linux;
in
{
defaultPackage.x86_64-linux = pkgs.writeScriptBin "hello-smalltalk" ''
${pkgs.gnu-smalltalk}/bin/gst <<< "Transcript show: 'Hello World!'."
'';
};
}
</code></pre>
<p>The pinned version of all our dependencies, <code>flake.lock</code></p>
<pre data-lang="json" class="language-json "><code class="language-json" data-lang="json">{
"nodes": {
"nixpkgs": {
"locked": {
"lastModified": 1606669556,
"narHash": "sha256-9rlqZ5JwnA6nK04vKhV0s5ndepnWL5hpkaTV1b4ASvk=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "ae47c79479a086e96e2977c61e538881913c0c08",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixos-20.09",
"repo": "nixpkgs",
"type": "github"
}
},
"root": {
"inputs": {
"nixpkgs": "nixpkgs"
}
}
},
"root": "root",
"version": 7
}
</code></pre>
<p>copy those files into a directory and run it:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">❯ nix run
Hello World!
</code></pre>
<h2 id="solid-foundations">Solid Foundations</h2>
<p>With Nix, we can make steady forward progress. Without fear that our foundations
will collapse under us like sand castles. Once we've built something in Nix we
can be pretty sure it will work for our colleague or ourselves in 10 years. Nix
is building a solid foundation that I can no longer live without.</p>
<p>If you haven't used Nix before, here's your call to action:</p>
<ul>
<li>Nix's homepage: <a href="https://nixos.org/">https://nixos.org/</a></li>
<li>Nix's Learning page: <a href="https://nixos.org/learn">https://nixos.org/learn</a></li>
<li>Learn Nix in little bite-sized pills: <a href="https://nixos.org/guides/nix-pills/">https://nixos.org/guides/nix-pills/</a></li>
</ul>
<hr />
<h2 id="disclaimer">Disclaimer</h2>
<p>There are various factors that lead to bit rot. Some are easier to solve than
others. For the purpose of this post I'm only considering programs that are
roughly self contained. For example, if a program relies on hitting a specific
Google endpoint, the only way to use this program would be to emulate the whole
Google stack or rely on that <a href="https://gcemetery.co/">endpoint existing</a>.
Sometimes it's doable to emulate the external API, and sometimes it isn't. This
post is specifically about cases where it is feasible to emulate the external API.</p>
<h3 id="footnotes">Footnotes</h3>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>Okay forever is a really long time. And this will likely not run forever. But why? The easy reasons are: "Github is down", "A source tarball you need can't be fetched from the internet", "x86_64 processors can't be found or emulated". But what's a weird reason that this may fail in the future? It'll probably be hard to predict, but maybe something like: SHA256 has been broken and criminals and/or pranksters have published malicious packages that match a certain SHA256. So build tools that rely on a deterministic and hard to break hash algorithm like SHA256 (like what Nix does) will no longer be reliable. That would be a funny future. Send me your weird reasons: <code>"marco+forever" ++ "@marcopolo.io"</code></p>
</div>
Nix and small containers with Docker multi-stage builds2020-05-15T00:00:00+00:002020-05-15T00:00:00+00:00https://marcopolo.io/code/nix-and-small-containers/<p>Multi Stage builds are great for minimizing the size of your container. The
general idea is you have a stage as your builder and another stage as your
product. This allows you to have a full development and build container while
still having a lean production container. The production container only carries
its runtime dependencies.</p>
<pre data-lang="dockerfile" class="language-dockerfile "><code class="language-dockerfile" data-lang="dockerfile">FROM golang:1.7.3
WORKDIR /go/src/github.com/alexellis/href-counter/
RUN go get -d -v golang.org/x/net/html
COPY app.go .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o app .
FROM alpine:latest
RUN apk --no-cache add ca-certificates
WORKDIR /root/
COPY --from=0 /go/src/github.com/alexellis/href-counter/app .
CMD ["./app"]
</code></pre>
<p>(from Docker's <a href="https://docs.docker.com/develop/develop-images/multistage-build/">docs on multi-stage</a>)</p>
<p>Sounds great, right? What's the catch? Well, it's not always easy to know what the
runtime dependencies are. For example you may have installed something in /lib
that was needed in the build process. But it turned out to be a shared library
and now it needs to be included in the production container. Tricky! Is there
some automated way to know all your runtime dependencies?</p>
<h2 id="enter-nix">Enter Nix</h2>
<p><a href="https://nixos.org/">Nix</a> is a functional and immutable package manager. It works great for
reproducible builds. It keeps track of packages and their dependencies via their
content hashes. And, relevant for this exercise, it also keeps track of the
dependencies of a built package. That means we can use Nix to build our project
and then ask Nix what our runtime dependencies are. With that information we can
copy just those files to the product stage of our multi-stage build and end up
with the smallest possible docker container.</p>
<p>Our general strategy will be to use a Nix builder to build our code. Ask the Nix
builder to tell us all the runtime dependencies of our built executable. Then
copy the executable with all it's runtime dependencies to a fresh container. Our
expectation is that this will result in a minimal production container.</p>
<h2 id="example">Example</h2>
<p>As a simple example let's package a "Hello World" program in Rust. The code is
what you'd expect:</p>
<pre data-lang="rust" class="language-rust "><code class="language-rust" data-lang="rust">pub fn main() {
println!("Hello, world!");
}
</code></pre>
<h3 id="nix-build-expression">Nix build expression</h3>
<p>If we were just building this locally, we'd just run <code>cargo build --release</code>.
But we are going to have Nix build this for us so that it can track the runtime
dependencies. Therefore we need a <code>default.nix</code> file to describe the build
process. Our <code>default.nix</code> build file looks like this:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">with (import <nixpkgs> {});
rustPlatform.buildRustPackage {
name = "hello-rust";
buildInputs = [ cargo rustc ];
src = ./.;
# This is a shasum over our crate dependencies
cargoSha256 = "1s4vg081ci6hskb3kk965nxnx384w8xb7n7yc4g93hj55qsk4vw5";
# Use this to figure out the correct Sha256
# cargoSha256 = lib.fakeSha256;
buildPhase = ''
cargo build --release
'';
checkPhase = "";
installPhase = ''
mkdir -p $out/bin
cp target/release/hello $out/bin
'';
}
</code></pre>
<p>Breaking down the Nix expression: we specify what our inputs our to our
build: <code>cargo</code> and <code>rustc</code>; we figure out what the sha256sum is of our crate
dependencies; and we define some commands to build and install the executable.</p>
<p>We can verify this works locally on our machine by running <code>nix-build .</code>
(assuming you have Nix installed locally). You'll end up with a symlink named
result that points the compiled executable residing in /nix/store. Running
<code>./result/bin/hello</code> should print "Hello, world!".</p>
<h3 id="docker-file">Docker file</h3>
<p>Now that we've built our Nix expression that defines how the code is built, we
can add Docker to the mix. The goal is to have a builder stage that runs the
nix-build command, then have a production stage that copies the executable and
its runtime dependencies from builder. The production stage container will
therefore have only the minimal amount of stuff needed to run.</p>
<pre data-lang="Dockerfile" class="language-Dockerfile "><code class="language-Dockerfile" data-lang="Dockerfile"># Use nix as the builder
FROM nixos/nix:latest AS builder
# Update the channel so we can get the latest packages
RUN nix-channel --update nixpkgs
WORKDIR /app
# Run the builder first without our code to fetch build dependencies.
# This will fail, but that's okay. We just want to have the build dependencies
# cached as a layer. This is just a caching optimization that can be removed.
COPY default.nix .
RUN nix-build . || true
COPY . .
# Now that our code is here we actually build it
RUN nix-build .
# Copy all the run time dependencies into /tmp/nix-store-closure
RUN mkdir /tmp/nix-store-closure
RUN echo "Output references (Runtime dependencies):" $(nix-store -qR result/)
RUN cp -R $(nix-store -qR result/) /tmp/nix-store-closure
ENTRYPOINT [ "/bin/sh" ]
# Our production stage
FROM scratch
WORKDIR /app
# Copy the runtime dependencies into /nix/store
# Note we don't actually have nix installed on this container. But that's fine,
# we don't need it, the built code only relies on the given files existing, not
# Nix.
COPY --from=builder /tmp/nix-store-closure /nix/store
COPY --from=builder /app/result /app
CMD ["/app/bin/hello"]
</code></pre>
<p>If we build this <code>Dockerfile</code> with <code>docker build .</code>, we'll end up with an 33MB
container. Compare this to a naive
<a href="https://gist.github.com/MarcoPolo/7953f1ca2691405b5b04659027967336">Dockerfile</a>
where we end up with a 624 MB container! That's an order of magnitude smaller
for a relatively simple change.</p>
<p>Note that our executable has a shared library dependency on libc. Alpine
linux doesn't include libc, but this still works. How? When we build our code we
reference the libc shared library stored inside <code>/nix/store</code>. Then when we copy
the executable nix tells us that the libc shared library is also a dependency so
we copy that too. Our executable uses only the libc inside <code>/nix/store</code> and
doesn't rely on any system provided libraries in <code>/lib</code> or elsewhere.</p>
<h2 id="conclusion">Conclusion</h2>
<p>With a simple Nix build expression and the use of Docker's multi stage builds we
can use Docker's strength of providing a consistent and portable environment
with Nix's fine grained dependency resolution to create a minimal production
container.</p>
<h2 id="a-note-on-statically-linked-executables">A note on statically linked executables</h2>
<p>Yes, you could build the hello world example as a statically linked musl-backed
binary. But that's not the point. Sometimes code relies on a shared library, and
it's just not worth or impossible to convert it. The beauty of this system is
that it doesn't matter if the output executable is fully statically linked or
not. It will work just the same and copy over the minimum amount of code needed
for the production container to work.</p>
<h2 id="a-note-on-nix-s-dockertools">A note on Nix's dockerTools</h2>
<p>Nix proves a set of functions for creating Docker images:
<a href="https://nixos.org/nixpkgs/manual/#sec-pkgs-dockerTools">pkgs.dockerTools</a>. It's
very cool, and I recommend checking it. Unlike docker it produces
deterministic images. Note, for all but the simplest examples, KVM is required.</p>
<h2 id="a-note-on-bazel-s-rules-docker">A note on Bazel's rules_docker</h2>
<p>I don't know much about this, but I'd assume this would be similar to what I've
described. If you know more about this, please let me know!</p>
From C to Rust to C again. Or: Re-exporting C exports in Rust2019-12-12T00:00:00+00:002019-12-12T00:00:00+00:00https://marcopolo.io/code/from-c-to-rust-to-c/<p>The only difference between being a grown up and being a kid, in my experience, is as a grown up, you have much fewer people who are willing to play the game <em>telephone</em> with you. Luckily for me, I have access to a computer, a C compiler, and a Rust compiler. Let me show you how I played telephone with Rust & C.</p>
<p>tl;dr:</p>
<ul>
<li>Rust can't re-export from a linked C library (unless you rename) when compiled as a cdylib.</li>
<li>Look at this <a href="https://github.com/rust-lang/rfcs/issues/2771">issue</a></li>
</ul>
<p>Imagine you have some C code that provides <code>add_two</code>. It looks like this:</p>
<pre data-lang="c" class="language-c "><code class="language-c" data-lang="c">int add_two(int n)
{
return n + 2;
}
</code></pre>
<p>And you can even let Cargo deal with building your C library by making a build.rs with <code>cc</code>. Like so:</p>
<pre data-lang="rust" class="language-rust "><code class="language-rust" data-lang="rust">use cc;
fn main() {
cc::Build::new().file("src/c/foo.c").compile("foo");
}
</code></pre>
<p>Now you want to be able to call <code>add_two</code> from Rust. Easy! You look at the <a href="https://doc.rust-lang.org/nomicon/ffi.html">FFI</a> section in the Nomicon. And follow it like so:</p>
<pre data-lang="rust" class="language-rust "><code class="language-rust" data-lang="rust">#[link(name = "foo", kind = "static")]
#[no_mangle]
extern "C" {
pub fn add_two(x: u32) -> u32;
}
#[no_mangle]
pub extern "C" fn add_one(x: u32) -> u32 {
let a = unsafe { add_two(x) };
a - 1
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn it_works() {
assert_eq!(add_one(2), 3);
assert_eq!(unsafe { add_two(2) }, 4);
}
}
</code></pre>
<p>Now for the last chain in our telephone. We'll make a new C file that will call our Rust defined <code>add_one</code> and our C defined <code>add_two</code>.</p>
<pre data-lang="c" class="language-c "><code class="language-c" data-lang="c">extern int add_one(int n);
extern int add_two(int n);
int main()
{
return add_one(add_two(39));
}
</code></pre>
<p>We use Clang to build this file:</p>
<pre><code>clang call_rust.c -lrust_c_playground -L./target/debug -o call_rust
</code></pre>
<p>Now we have an executable called <code>call_rust</code> which calls a Rust defined function and calls a C defined function that it pulled in from a single Rust Library (called <code>librust_c_playground.dylib</code> on macOS). The flags in the clang command mean: <code>-l</code> link this library; <code>-L</code> look here for the library.</p>
<p>We've built the code, now we can even run it!</p>
<pre><code>./call_rust
echo $? # Print the return code of our program, hopefully 42
</code></pre>
<p>Great! We've called C from a Rust Library from a C program. But there's a catch. This won't work if you are building a <code>cdylib</code>. There isn't an RFC yet on how to re-export C externs. In the mean time you'll either have to: re-export under a different name, or build a <code>dylib</code>. See this issue: <a href="https://github.com/rust-lang/rfcs/issues/2771">Re-exporting C symbols for cdylib</a>.</p>
<p>Hope this helps.</p>
Wasm is the future of serverless. Terrafirma, serverless wasm functions.2019-11-06T00:00:00+00:002019-11-06T00:00:00+00:00https://marcopolo.io/code/terrafirma/<p>When I ran into Fastly's <a href="https://wasm.fastlylabs.com/">Terrarium</a>, the appeal of Webassembly (wasm) finally clicked for me. We could have lightweight sandboxes and bring in my own language and libraries without the overhead of a full OS VM or <a href="https://blog.iron.io/the-overhead-of-docker-run/">Docker</a>. That's great for the serverless provider, but it's also great for the end user. Less overhead means faster startup time and less total cost.</p>
<h2 id="how-much-faster">How much faster?</h2>
<p>On my machine™, a hello world shell script takes 3ms, a docker equivalent takes 700ms, and a wasm equivalent takes 15ms.</p>
<p>Following <a href="https://blog.iron.io/the-overhead-of-docker-run/">this experiment</a> I get these results:</p>
<pre><code>Running: ./hello.sh
avg: 3.516431ms
Running: docker run treeder/hello:sh
avg: 692.306769ms
Running: docker run --rm treeder/hello:sh
avg: 725.912422ms
Running: docker start -a reuse
avg: 655.059021ms
Running: node hello.js
avg: 79.233337ms
Running: wasmer run wasi-hello-world.wasm
avg: 15.155896ms
</code></pre>
<p>When I think about how WASM, Docker, and OS VMs (compute instances) play together, I picture this graph below.</p>
<p><img src="/code/wasm-graph.png" alt="Safety versus overhead – Raw binary is fast unsafe; was is fast and safe; docker is safe." title="Safety vs Overhead" /></p>
<p>The trend is that if you want safety and isolation, you must pay for it with overhead. WASM's exception to that rule is what I think makes it so promising and interesting. Wasm provides the fastest way to run arbitrary user code in a sandboxed environment.</p>
<h2 id="what-is-webassembly">What is Webassembly?</h2>
<p>Webassembly is a spec for a lightweight and sandboxed VM. Webassembly is run by a host, and can't do any side effects, unless it calls a function provided by the host. For example, if your WASM code wanted to make a GET request to a website, it could only do that by asking the host to help. The host exposes these helper function to the WASM guest. In Terrafirma, these are the <code>hostcall_*</code> functions in <a href="https://github.com/MarcoPolo/go-wasm-terrafirma/blob/master/imports.go"><code>imports.go</code></a>. It's called <code>imports.go</code> because it is what your WASM code is importing from the host.</p>
<h2 id="bring-your-own-tools">Bring your own tools</h2>
<p>As long as you can compile everything to a .wasm file, you can use whatever tools and language you want. All I have to do is provide a runtime, and all you have to do is provide a wasm file. However, there is a subtle caveat here. The only way you can run side effects is with the host cooperation. So you (or some library you use) must understand the environment you're running in in order to do anything interesting.</p>
<h2 id="what-about-a-standard-wasm-environment">What about a standard WASM Environment?</h2>
<p>There isn't a mature industry standard for what imports a host should provide to the WASM code running outside the browser. The closest thing we have is <a href="https://wasi.dev/">WASI</a>, which defines a POSIX inspired set of syscalls that a host should implement. It's useful because it allows code would otherwise require a real syscall to work in a WASM environment. For example, In Rust you can build with the <code>--target wasm32-wasi</code> flag and your code will just work in any <a href="https://wasmer.io/">wasi environment</a>.</p>
<h2 id="terrafirma">Terrafirma</h2>
<p>Phew! Finally at TerraFirma. TerraFirma is a WASM runtime environment I wrote to let you run wasm code in the cloud. You upload your wasm file by copying it into a shared <a href="https://keybase.io/docs/kbfs">KBFS folder</a> with the keybase user <a href="https://keybase.io/kbwasm">kbwasm</a>. Then you setup some DNS records to point your domain to TerraFirma's servers. And that's it! You can update the wasm code at any time by overwriting the old .wasm file with the new one.</p>
<h2 id="code-examples">Code Examples</h2>
<ul>
<li><a href="https://github.com/MarcoPolo/terrafirma-hello-world">Hello World</a></li>
<li><a href="https://github.com/MarcoPolo/terrafirma-scraper">Scraper Endpoint</a> – A web scraper that uses Servo – a new browser engine from Mozilla.</li>
</ul>
<h3 id="terrafirma-hello-world-tutorial">Terrafirma – Hello World Tutorial</h3>
<p>This example uses Rust, so if you don't have that setup <a href="https://rustup.rs/">go here first</a>.</p>
<ol>
<li>Point your domain to TerraFirma servers (<code>terrafirma.marcopolo.io</code> or <code>52.53.126.109</code>) with an A record, and set a <code>TXT</code> record to point to your shared folder (e.g. <code>"kbp=/keybase/private/<my_keybase_username>,kbwasm/"</code>)</li>
</ol>
<pre><code>
example.com 300 A terrafirma.marcopolo.io
_keybase_pages.example.com 300 TXT "kbp=/keybase/private/<my_keybase_username>,kbwasm/"
</code></pre>
<ol start="2">
<li>Verify the DNS records are correct</li>
</ol>
<pre><code>
$ dig example.com A
...
;; ANSWER SECTION:
wasm.marcopolo.io. 300 IN A 52.53.126.109
...
</code></pre>
<br/>
<pre><code>
$ dig _keybase_pages.example.com TXT
...
;; ANSWER SECTION:
_keybase_pages.example.com <number> IN TXT "kbp=/keybase/private/<my_keybase_username>,kbpbot/"
...
</code></pre>
<ol start="3">
<li>Clone the Hello World Repo</li>
</ol>
<pre><code>git clone git@github.com:MarcoPolo/terrafirma-hello-world.git
</code></pre>
<ol start="4">
<li>Build it</li>
</ol>
<pre><code>cd terrafirma-hello-world
cargo build --release
</code></pre>
<ol start="5">
<li>Deploy it</li>
</ol>
<pre><code>
cp target/wasm32-unknown-unknown/release/terrafirma_helloworld.wasm /keybase/private/<your_kb_username>,kbwasm/hello.wasm
</code></pre>
<ol start="6">
<li>Test it</li>
</ol>
<pre><code>curl https://example.com/hello.wasm
</code></pre>
Thoughts on "Why is React doing this?"2019-09-06T00:00:00+00:002019-09-06T00:00:00+00:00https://marcopolo.io/code/why-react-response/<h1 id="response-to-why-react">Response to <a href="https://gist.github.com/sebmarkbage/a5ef436427437a98408672108df01919">Why React?</a></h1>
<p>Some quick thoughts I had after reading the <a href="https://gist.github.com/sebmarkbage/a5ef436427437a98408672108df01919">Why React?</a> gist.</p>
<p>Disclaimer: <em>I want to be critical with React. I don't disagree that it has done some amazing things</em></p>
<h2 id="compiled-output-results-in-smaller-apps">"Compiled output results in smaller apps"</h2>
<blockquote>
<p>E.g. Svelte apps start smaller but the compiler output is 3-4x larger per component than the equivalent VDOM approach.</p>
</blockquote>
<p>This may be true currently, but that doesn't mean it will always be true of compiled-to frameworks. A theoretical compiler can produce a component that uses a shared library for all components. If a user doesn't use all the features of a framework, then a compiler could remove the unused features from the output. Which is something that could not happen with a framework that relies on a full runtime.</p>
<p>Note: I'm not advocating for a compiled-to approach, I just think this point was misleading</p>
<h2 id="dom-is-stateful-imperative-so-we-should-embrace-it">"DOM is stateful/imperative, so we should embrace it"</h2>
<p>I agree with OP here. Most use-cases would not benefit from an imperative UI api.</p>
<h2 id="react-leaks-implementation-details-through-usememo">"React leaks implementation details through useMemo"</h2>
<p>A common problem to bite new comers is when they pass a closure to a component, and that closure gets changed every time which causes their component to re-render every time. <code>useMemo</code> can fix this issue, but it offloads a bit of work to the developer.</p>
<p>In the above context, it's an implementation detail. I'm not saying it's the wrong or right trade off, I'm only saying that the reason you have to reach for <code>useMemo</code> when passing around closures is because of how React is implemented. So the quote is accurate.</p>
<p>Is that a bad thing? That's where it gets more subjective. I think it is, because these types of things happen very often and, in a big app, you quickly succumb to death by a thousand cuts (one closure causing a component to re-render isn't a big deal, but when you have hundreds of components with various closures it gets hairy).</p>
<p>The next example OP posts is about setting users in a list.</p>
<pre data-lang="js" class="language-js "><code class="language-js" data-lang="js">setUsers([
...users.filter(user => user.name !== "Sebastian"),
{ name: "Sebastian" }
]);
</code></pre>
<p>If you are happy with that syntax, and the tradeoff of having to use <code>key</code> props whenever you display lists, and relying on React's heuristics to efficiently update the views corresponding to the list, then React is fine. If, however, you are okay with a different syntax you may be interested in another idea I've seen. The basic idea is you keep track of the diffs themselves instead of the old version vs. the new version. Knowing the diffs directly let you know exactly how to update the views directly so you don't have to rely on the <code>key</code> prop, heuristics, and you can efficiently/quickly update the View list. This is similar to how <a href="https://github.com/immerjs/immer">Immer</a> works. <a href="https://docs.rs/futures-signals/0.3.8/futures_signals/tutorial/index.html">Futures Signals</a> also does this to efficiently send updates of a list to consumers (look at <code>SignalVec</code>).</p>
<h2 id="stale-closures-in-hooks-are-confusing">"Stale closures in Hooks are confusing"</h2>
<p>I agree with OP's points here. It's important to know where your data is coming from. In the old hook-less style of React, your data was what you got from your props/state and nothing else. With hooks, it's easier to work with stale data that comes in from outside your props. It's a learning curve, but not necessarily bad.</p>
<p>One thing I find interesting is that the use of hooks moves functional components into becoming more stateful components. I think this is fine, but it loses the pure functional guarantees you had before.</p>
<p>I haven't yet made up my mind about hooks that interact with the context. (i.e. <code>useSelector</code> or <code>useDispatch</code>) since the context is less structured. i.e. This component's selector function for <code>useSelector</code> relies on the state being <code>X</code>, but <code>X</code> isn't passed in, it's set as the store in redux configuration file somewhere else. Now that the component relies on the shape of the store being <code>X</code> it makes it harder to move out. This may not actually matter in practice, and it may be much more useful to be able to pull arbitrary things out of your store. Hence why I'm currently undecided about it.</p>
Bart Facts2019-09-05T00:00:00+00:002019-09-05T00:00:00+00:00https://marcopolo.io/code/bart-facts/<p><a href="/code/bart-facts.html">Click Here</a></p>
Active Projects2019-08-30T00:00:00+00:002019-08-30T00:00:00+00:00https://marcopolo.io/life/active-projects/<h1 id="active-projects-with-updates">Active Projects (with updates)</h1>
<h2 id="react-causal-profiler">React Causal Profiler</h2>
<p>https://www.youtube.com/watch?v=r-TLSBdHe1A&t=11s</p>
<hr />
<h2 id="fastly-labs-terrarium-on-kbfs">fastly labs terrarium on KBFS</h2>
<p>Push wasm code, have it run on distributed servers for you
(done)[https://marcopolo.io/wasm]</p>
<hr />
<h2 id="android-wired-a-rust-ui-framework-for-android">Android Wired - A Rust UI framework for Android</h2>
<p>(repo)[https://github.com/MarcoPolo/android-wired/]</p>
<p>An attempt at building a friend Rust UI library for Android.</p>
<ul>
<li>Coming up: Add flex-box support on android</li>
</ul>
<hr />
<h1 id="interesting-cities-map">Interesting Cities Map</h1>
<p>Yet another map of cities that seem nice to live in. Scored by:</p>
<ul>
<li>Climbing</li>
<li>Bike Score</li>
<li>Walk Score</li>
<li>Population</li>
</ul>
<p>To add:
<a href="https://www.expatistan.com/cost-of-living/index">Cost of living index</a>
Disaster preparedness</p>
<hr />
<h2 id="slack-bridge">Slack Bridge</h2>
<p>Connect kb to slack with the bot library</p>
<ul>
<li>https://github.com/MarcoPolo/slack-keybase-bridge</li>
</ul>
<hr />
<h2 id="move-off-of-betterment">Move off of betterment</h2>
<p>A 3% return is only barely better than a bank.</p>
<hr />
<h2 id="rusty-keybase-bot">Rusty Keybase Bot</h2>
<p>Build a Keybase Bot Library in Rust (With proper typings)</p>
<ul>
<li>This is live! https://crates.io/crates/keybase-bot-api</li>
<li>I rabbit holed into creating an AVDL parser with pest.</li>
</ul>
<hr />
<h2 id="riding-those-new-bart-trains">Riding those new BART trains?</h2>
<ul>
<li>Is it possible to figure out when the new bart trains are arriving so I can optimize my commute to ride those puppies?</li>
</ul>
<hr />
<h2 id="programmatically-find-climbing-crags">Programmatically find Climbing crags</h2>
<ul>
<li>How can I parse USGS and GIS data to find good crags for climbing?</li>
</ul>
My Experience at The Recurse Center2019-08-23T00:00:00+00:002019-08-23T00:00:00+00:00https://marcopolo.io/code/my-recurse-experience/<p><a href="https://www.recurse.com/">The Recurse Center</a> is this magical place where hours fly by, and you can be as distracted or as focused as you choose to be. It's composed of two nice floors of a building in downtown Brooklyn. They provide a space where you can progress on your own programming specific goals at your own self direction.</p>
<p>Some of my best friends are RCers. All of them really enjoyed their time at RC, and would recommend it to anyone who would listen. With glowing praise from people I deeply respect, how could I not give it a try? It also helped that I had a friend and former co-worker (Fred) applying for a week long batch as well.</p>
<p>Initially I was skeptical about it. It also seemed like quite a bit of work. I spent days on their "What do you want to be doing in 2 years question"; I'd have to do two interviews; and I'd have to figure out living logistics. Given those hurdles I was skeptical of what the benefit of the program would be. Despite my skepticism I decided to trust my friends. Well... Truthfully Fred nudged me along, and I'm very glad he did.</p>
<h1 id="the-mini-batch-1-week-long-program">The Mini Batch. (1 Week long program)</h1>
<p>The mini batch seems like a strategy to bring in folks who cannot be in NYC for 6 or 12 weeks or are not ready for that level of commitment (That's me!). It felt much shorter than I was expecting (and I was expecting it to be short). Every day of the week puts you 20% closer to the end of the program! After seeing how fast a week goes by, 6 and 12 weeks don't seem so long.</p>
<h1 id="what-s-special-about-the-recurse-center">What's special about The Recurse Center</h1>
<p>The project I worked on is something I've been toying with in my mind for a while. I even wrote a couple toy implementations that never got fleshed out. I've been toying with the idea for at least 4 months, and hadn't made much progress. After a week at RC, however, starting from scratch, I'd already gotten much farther than ever before. I believe it was a combination of a couple things at RC.</p>
<ol>
<li>You can spend your time programming on your own projects (and people will think that's neat)
<ul>
<li>Contrast this with most places where the external pressures ask:
<ul>
<li>At home: "What are you doing that for? Is it for work? Oh, It isn't? Okay."</li>
<li>On Vacation: "Why are you on your computer when you could be outside exploring?"</li>
<li>Hackathon: "What are you working on? How is going to make us two boat loads of money?"</li>
</ul>
</li>
</ul>
</li>
<li>A social space <em>and</em> quiet space.
<ul>
<li>The Recurse Center has two floors. A quiet floor (imagine a less strict library) and an active/social floor (where people talk freely, express freely, and openly invite each other to pair program something with them.). I think this is really important and, thinking back to previous jobs, I think it would be a great addition to any company with enough programmers. Sometimes I want to talk to people, and be interrupted, and pair with someone; but sometimes I just want to sit down, uninterrupted, and get lost in my problem space and thoughts.</li>
</ul>
</li>
<li>Diversity.
<ol>
<li>The diversity in the people who attend.
<ul>
<li>Being at RC lets me preview a world where our industry is more inclusive and diverse.</li>
<li>People with different backgrounds bring different thoughts and potential solutions.</li>
<li>A monoculture of ideas is a recipe for stagnation. Team work <a href="https://www.nytimes.com/2016/02/28/magazine/what-google-learned-from-its-quest-to-build-the-perfect-team.html?smid=pl-share">objectively</a> leads to better solutions.</li>
</ul>
</li>
<li>The diversity in the knowledge backgrounds.
<ul>
<li>Quick example: Imagine you want to know how VGA character buffers work, you can find someone who spent at least a week studying it in a matter of minutes.</li>
<li>You can get pleasantly side-tracked with someone else's rabbit hole.</li>
<li>You can explore the limits of your knowledge when explaining things you thought you understood to people who are genuinely curious!</li>
</ul>
</li>
</ol>
</li>
<li>Not a startup incubator
<ul>
<li>I used to go to hackathons because I liked the idea of building something fun with friends. I stopped going because it turned into a competition to see which one of these 48 hour projects can become the next unicorn startup. (Incidentally, Brian and I started our own hackathon we call the <a href="https://cryptic.house/">Dumbathon</a> which tries to find the original spirit of the hackathon.)</li>
<li>At RC, no one ever asks "How are you going to make money off this?" If all you end up after your batch is a deeper understanding of problems that interest you, then you've had a successful time there and no one will tell you differently.</li>
</ul>
</li>
<li>Lightweight Social Rules.
<ul>
<li>Interacting with a lot of new people is <em>hard</em>. RC has 4 <a href="https://www.recurse.com/social-rules">lightweight social rules</a> to help people be nice to each other. On top of that, there's an understanding that everyone here is clever and everyone is here because they truly want to be here. That level of commitment isn't common in most interactions. The combination of those two things make it a joy to interact with other RCers in the space and get lost talking about your programming project or hearing about their recent problems.</li>
</ul>
</li>
</ol>
<h1 id="why-i-think-you-should-do-it">Why I think you should do it</h1>
<p>It may seem selfish or purely indulgent to take away time from your life to go to a programmer's retreat, it isn't. Take history's most selfless person, The Buddha. He dedicated his life to reducing the suffering of those around him. One lesser know story about The Buddha is that he would have <a href="https://en.wikipedia.org/wiki/Vassa">3 month retreats</a> yearly where he and his disciples could refocus and reinvest their energies. I tell this anecdote not to liken us to The Buddha, but to point out that the prototypical selfless person found these retreats useful.</p>
<p>I think in the daily grind it's easy to loose sight of the things that matter to you. Often when at a normal job, your goals of growth and learning don't align with the bottom line of the business. That's okay, but it's very nice to be able to step back and re-hone your tool set.</p>
<hr />
<h1 id="my-project">My Project</h1>
<p>In a week, I built an <a href="https://github.com/MarcoPolo/android-wired">Android UI Library for Rust</a>. I am proud of it, and I'll have another post that details it at length.</p>
I moved my blog over to Zola, you should too2019-08-22T00:00:00+00:002019-08-22T00:00:00+00:00https://marcopolo.io/code/migrating-to-zola/<h1 id="blogging">Blogging</h1>
<p>I started this blog like many other folks, on GitHub Pages. It was great at
the time. You can have a source repo that compiles to a blog. Neat! Over time
though I started really feeling the pain points with it. When I wanted to
write a quick post about something I'd often spend hours just trying to get
the right Ruby environment set up so I can see my blog locally. When I got an
email from GitHub saying that my blog repo has a security vulnerability in
one of its Gems, I took the opportunity to switch over to
<a href="https://www.getzola.org">Zola</a>.</p>
<h1 id="zola">Zola</h1>
<p>Zola make more sense to me than Jekyll. I think about my posts in a
hierarchy. I'd like my source code to match my mental representation. If you
look at the <a href="https://marcopolo.io/code/migrating-to-zola/">source</a> of this blog, you'll see I have 3 folders (code, books,
life). In each folder there are relevant posts. I wanted my blog to show the
contents folder as different sections. For the life of me I couldn't figure
out how to do that in Jekyll. I ended up just using a single folder for all
my posts and using the category metadata in the front-matter to create the
different sections. With Zola, this kind of just worked. I had to create an
<code>_index.md</code> file to provide some metadata, but nothing overly verbose.</p>
<h1 id="i-m-not-a-jekyll-pro">I'm not a Jekyll pro...</h1>
<p>Or even really any level past beginner. I image if you've already heavily
invested yourself in the Jekyll ecosystem this probably wouldn't make sense
for you. I'm sure there are all sorts of tricks and features that Jekyll
can do that Zola cannot. I'm Okay with that. I really don't need that much
from my blogging library.</p>
<p>Zola has 3 commands: <code>build</code>, <code>serve</code>, and <code>init</code>. They do what you'd expect
and nothing more. I really admire this philosophy. Whittle down your feature
set and make those features a <em>joy</em> to use.</p>
<h1 id="fast">Fast</h1>
<p>Changes in Zola propagate quickly. Zola rebuilds my (admittedly very small blog) in less than a millisecond. Zola comes with a livereload script that automatically updates your browser when you are in <code>serve</code> mode. It's feasible to write your post and see how it renders almost instantly.</p>
<h1 id="transition">Transition</h1>
<p>The biggest change was converting Jekyll's front-matter (the stuff at the top
of the md files) format into Zola's front-matter format. Which was changing
this:</p>
<pre><code>---
layout: post
title: Interacting with Go from React Native through JSI
categories: javascript react-native jsi go
---
</code></pre>
<p>into this:</p>
<pre><code>+++
title = "Interacting with Go from React Native through JSI"
[taxonomies]
tags = ["javascript", "react-native", "JSI", "Go"]
+++
</code></pre>
<p>There was also a slight rewrite in the template files that was necessary
since Zola uses the <a href="https://tera.netlify.com">Tera Templating Engine</a></p>
<p>The rest was just moving (I'd argue organizing) files around.</p>
<h1 id="prettier-repo">Prettier Repo</h1>
<p>I think at the end the repo became a little prettier to look at. You could
argue it's a small thing, but I think these small things matter. It's already
hard enough to sit down and write a post. I want every bit of the experience
to be beautiful.</p>
<p>But don't take my word for it! judge yourself: <a href="https://github.com/MarcoPolo/marcopolo.github.io/tree/jekyll_archive">Jekyll</a> vs. <a href="https://github.com/MarcoPolo/marcopolo.github.io">Zola</a></p>
Travel Plans2019-08-22T00:00:00+00:002019-08-22T00:00:00+00:00https://marcopolo.io/life/travelling/<ul>
<li>Florida 23 -> 29</li>
<li>Bay Area 29 -> ?</li>
</ul>
Interacting with Go from React Native through JSI2019-06-27T00:00:00+00:002019-06-27T00:00:00+00:00https://marcopolo.io/code/go-rn-jsi/<h1 id="introduction">Introduction</h1>
<p>There are 3 parts that let JS talk to Go:</p>
<ol>
<li>The C++ binding</li>
<li>Installing the binding</li>
<li>Calling Go</li>
</ol>
<p>Not all the code is shown, check out the <a href="https://github.com/MarcoPolo/react-native-hostobject-demo">source code</a> for specifics.</p>
<h3 id="part-1-the-c-binding">Part 1 - The C++ Binding</h3>
<p>The binding is the C++ glue code that will hook up your Go code to the JS runtime. The binding itself is composed of two main parts.</p>
<h4 id="part-1-1-the-c-binding">Part 1.1 - The C++ Binding</h4>
<p>The binding is a c++ class that implements the <code>jsi::HostObject</code> interface. At the very least it's useful for it to have a <code>get</code> method defined. The type of the <code>get</code> method is:</p>
<pre data-lang="c++" class="language-c++ "><code class="language-c++" data-lang="c++">jsi::Value get(jsi::Runtime &runtime, const jsi::PropNameID &name) override;
</code></pre>
<p>It returns a <code>jsi::Value</code> (a value that is safe for JS). It's given the JS runtime and the prop string used by JS when it <code>get</code>s the field. e.g. <code>global.nativeTest.foo</code> will call this method with PropNameID === <code>"foo"</code>.</p>
<h4 id="part-1-2-the-c-binding-s-install">Part 1.2 - The C++ Binding's install</h4>
<p>Now that we've defined our HostObject, we need to install it into the runtime. We use a static member function that we'll call later to set this up. It looks like this:</p>
<pre data-lang="c++" class="language-c++ "><code class="language-c++" data-lang="c++">void TestBinding::install(jsi::Runtime &runtime,
std::shared_ptr<TestBinding> testBinding) {
// What is the name that js will use when it reaches for this?
// i.e. `global.nativeTest` in JS
auto testModuleName = "nativeTest";
// Create a JS object version of our binding
auto object = jsi::Object::createFromHostObject(runtime, testBinding);
// set the "nativeTest" propert
runtime.global().setProperty(runtime, testModuleName, std::move(object));
}
</code></pre>
<h3 id="part-2-installing-the-binding-on-android">Part 2. Installing the binding (on Android)</h3>
<p>Since we have a reference to the runtime in Java land, we'll have to create a JNI method to pass the runtime ptr to the native C++ land. We can add this JNI method to our TestBinding file with a guard.</p>
<pre data-lang="c++" class="language-c++ "><code class="language-c++" data-lang="c++">#if ANDROID
extern "C" {
JNIEXPORT void JNICALL Java_com_testmodule_MainActivity_install(
JNIEnv *env, jobject thiz, jlong runtimePtr) {
auto testBinding = std::make_shared<example::TestBinding>();
jsi::Runtime *runtime = (jsi::Runtime *)runtimePtr;
example::TestBinding::install(*runtime, testBinding);
}
}
#endif
</code></pre>
<p>Then on the Java side (after we compile this into a shared library), we register this native function and call it when we're ready.</p>
<pre data-lang="java" class="language-java "><code class="language-java" data-lang="java">// In MainActivity
public class MainActivity extends ReactActivity implements ReactInstanceManager.ReactInstanceEventListener {
static {
// Load our jni
System.loadLibrary("test_module_jni");
}
//... ellided ...
@Override
public void onReactContextInitialized(ReactContext context) {
// Call our native function with the runtime pointer
install(context.getJavaScriptContextHolder().get());
}
// declare our native function
public native void install(long jsContextNativePointer);
}
</code></pre>
<h3 id="part-3-calling-go">Part 3. Calling Go</h3>
<p>Now that our binding is installed in the runtime, we can make it do something useful.</p>
<pre data-lang="c++" class="language-c++ "><code class="language-c++" data-lang="c++">jsi::Value TestBinding::get(jsi::Runtime &runtime,
const jsi::PropNameID &name) {
auto methodName = name.utf8(runtime);
if (methodName == "runTest") {
return jsi::Function::createFromHostFunction(
runtime, name, 0,
[](jsi::Runtime &runtime, const jsi::Value &thisValue,
const jsi::Value *arguments,
size_t count) -> jsi::Value { return TestNum(); });
}
return jsi::Value::undefined();
}
</code></pre>
<p>Here we return a <code>jsi::Function</code> when JS calls <code>global.nativeTest.runTest</code>. When JS calls it (as in <code>global.nativeTest.runTest()</code>) we execute the code inside the closure, which just returns <code>TestNum()</code>. TestNum is a Go function that's exported through cgo so that it is available to c/c++. Our Go code looks like this:</p>
<pre data-lang="go" class="language-go "><code class="language-go" data-lang="go">package main
import "C"
// TestNum returns a test number to be used in JSI
//export TestNum
func TestNum() int {
return int(9001)
}
func main() {
}
</code></pre>
<p>cgo builds a header and creates a shared library that is used by our binding.</p>
<h3 id="building">Building</h3>
<ul>
<li>Look at the CMakeLists.txt for specifics on building the C++ code.</li>
<li>Look at from-go/build.sh for specifics on building the go code.</li>
</ul>
<h3 id="a-go-shared-library-for-c-java">A Go Shared Library for C + Java</h3>
<p>It's possible to build the Go code as a shared library for both C and Java, but you'll have to define your own JNI methods. It would be nice if gomobile bind also generated C headers for android, but it doesn't seem possible right now. Instead you'll have to run <code>go build -buildmode=c-shared</code> directly and define your jni methods yourself. Take a look at <code>from-go/build.sh</code> and testnum.go for specifics.</p>
<h2 id="further-reading">Further Reading</h2>
<p><a href="https://medium.com/@christian.falch/https-medium-com-christian-falch-react-native-jsi-challenge-1201a69c8fbf">JSI Challenge #1</a></p>
<p><a href="https://medium.com/@christian.falch/react-native-jsi-challenge-2-56fc4dd91613">JSI Challenge #2</a></p>
<p><a href="http://blog.nparashuram.com/2019/01/react-natives-new-architecture-glossary.html">RN Glossary of Terms</a></p>
<p><a href="https://blog.dogan.io/2015/08/15/java-jni-jnr-go/">GO JNI</a></p>
<p><a href="https://rakyll.org/cross-compilation/">GO Cross Compilation</a></p>
A review of _No One At The Wheel_ by Samuel Schwartz2019-02-22T00:00:00+00:002019-02-22T00:00:00+00:00https://marcopolo.io/books/no-one-at-the-wheel/<h1 id="no-one-at-the-wheel-driverless-cars-and-the-road-of-the-future">No One At The Wheel: Driverless Cars and the road of the Future</h1>
<p>I picked this book up after listening to Schwartz talk on <em>Fresh Air</em>. Schwartz
is a traffic engineer, who worked at the chief traffic engineer for NYC. I was
interested in this book because it adds a healthy dose of skepticism to all the
hype I've heard about self-driving cars. I expected him to say "The future will
be perfect, self driving cars will solve all our problems" or "The future is
terrible, self driving cars will run over all the babies". Instead, he was in an
informed center. He acknowledges a lot of benefits AVs (autonomous vehicles) can
bring, refutes some hollow claims from advocates, and points out several
pitfalls to avoid.</p>
<p>The book is an easy read and goes by quickly. I mostly listened to it,
amusingly enough, from a drive from Montana to California (where I
wished I had a self driving car to make the drive easier).</p>
<p>Schwartz refutes some ideas about AVs that I thought were a given. First is
that AVs, and AVs alone, will reduce traffic fatalities to nil. It's true that
AVs can introduce technology that will reduce traffic fatalities, but it is also
true that <strong>right now</strong> we have that technology that we can incorporate in non
self driving vehicles. Technologies like front collision warning (reduces accidents
by 7%), automatic front collision braking (reduces accidents by 15%), and lane
detection are available today. These technologies are shown to make driving
safer, but they are only available on high end luxury vehicles. We have the
power today to make the roads vastly safer overnight by passing legislation to
make these safety features standard.</p>
<p>The book warns of a dystopia future where everyone ones their own AV. In this
world traffic is continually clogged and up to half of the cars in traffic are
empty. People drive much more because it is that much easier. They sacrifice the
exercise they would have gained in biking or walking by choosing the easier
option of watching a movie while waiting in deadlock traffic. In this future,
AVs are more like phones. A new model coming out every year or so, and people
upgrading just as often. All the new production would have an obvious adverse
effect on climate. Likewise the increase in traffic would be terrible for the
environment.</p>
<p>The happy future Schwartz talks about is a future where AVs work with traffic
engineers to complement existing transit infrastructure. There could be AV
buses that could run much more frequently on more routes 24/7 that would extend
existing public transit routes. Autonomous mini buses can be used to create a
route on demand for the rush hour commuters to bring them to a bigger transit
hub (e.g. a subway). Smaller AVs can then be used in the off hours to get people
to their nearest transit hub, thus solving the last mile problem.</p>
<p>Schwartz sprinkles his book with parallels from the time cars where first
introduced. Roads where first made for cyclist. Cyclist were lobbying for more
roads around the time cars were being introduced. As cars became more ubiquitous
and faster, bicyclist and pedestrians were kicked off the roads and
criminalized. Roads took over and changed our cities into desolate avenues with
drivers careening through city centers at 40mph. Schwartz argues, and I'm
inclined to agree, we cannot let AVs push everyone else as second class
citizens. We are at an inflection point where we can define legislation to
prioritize pedestrians and city life.</p>
<p>I would recommend <a href="https://smile.amazon.com/dp/B07B8JGG5M/ref=dp-kindle-redirect?_encoding=UTF8&btkr=1&sa-no-redirect=1">this book</a>.</p>
Thoughts on _The Craftsman_ by Richard Sennet2019-02-21T00:00:00+00:002019-02-21T00:00:00+00:00https://marcopolo.io/books/the-craftsman-sennet/<p>I've finished my second read through of <em>The Craftsman</em>. It's an argument that
craftsman are more capable than society gives than credit for. The notion of
<em>animal laborens</em>, that man is simply a thoughtless animal, is wrong in Sennet's
view. He argues that we think deeply when we work with our hands. The woodworker
isn't just idly planing away at the wood she's working on. She looks at the
grain, analyzes the constraints of the material, and is constantly problem
solving <em>and</em> problem finding. The same applies to programmers. Fixing bugs is
never a simple tasks. Often it challenges both our problem solving and problem
finding abilities.</p>
<p>Sennet decorates his book with interesting case studies. The skill involved in
using a Chinese cleaver or the expressive metaphorical recipe of how to cook a
specific chicken dish are a couple of memorable examples. Sennet analyzes the
history of the craftsman from ancient times to today, and how the industrial
revolution has changed the scene. He asks Voltaire, with his encyclopedia, for
his thoughts to accompany his own. Voltaire understood the superiority of machines
in certain craft tasks, but felt it wasn't fair to the craftsman to focus only
on that superiority. The craftsman has the ability through problem solving and
problem finding to use these new machines as tools.</p>
<p>I found the story of "honest" bricks especially interesting. There was a time
when the only way to make bricks was the artisanal and very labor intensive way.
The bricks in these cases all came out slightly different and imbued with colors
that represented the geographic origin and creators. Then came a more industrial
form of creating bricks. This process created many bricks at uniform sizes and
colors very cheaply. At this point people started to personify the bricks. They
called bricks "honest" when they were hand made with blemishes that reflect
their creation. No one doubted that the industrial bricks where objectively
better, but there was still a considerable demand for these honest bricks. The
industrial process eventually came up with ways to emulate more characteristics
of the honest brick, such as modifying the colors of the bricks to mirror those
of bricks from smaller towns. Even though this happened several hundred years
ago and in another country, I think it's very easy to relate to this story.
We are essentially reliving this, but with everything.</p>
<p>The myth of Pandora and Hephaestus are brought again and again throughout the
book. The myths highlight the paradox in craft and society. Hephaestus was a
skilled craftsman but an ugly person (born with a club foot). Pandora, a
beautiful goddess with a box of ugly and evil things. The myths are the original
prototype of society's fears and beliefs on crafts. Frankenstein is a more modern
version of Pandora's box. Parts of the movie <em>Jiro Dreams of Sushi</em> is our
modern <em>Hymn to Hephaestus</em>.</p>
<p>I think the book is unbelievably fascinating in so many ways. It reminds me of
when I heard <em>Marian Hill</em> for the first time. I didn't even realize music could
sound that way. I was so happy to have discovered a whole new facet of life that
I didn't even know existed. The book is the same way. Every chapter introduces
some thought or history that I never knew existed. Each page was like Christmas.
A nice gift from Sennet.</p>
<p>I only have a few issues with the book. The biggest one is likely it's weight
on the idea that a society of skilled craftsmen could result in a society closer
to self governance. An idea close to Jefferson. Perhaps it's because I've been
reading Ron Chernow's Hamilton biography, but I'm primed to be suspect of
Jefferson's political theories. I'm surprised by Sennet's belief here as he
classifies a lot of somewhat surprising things as crafts (parenting, experience,
etc.), but doesn't seem to classify governance and politics as a craft.</p>
<p>It's a great book, and I'd recommend it. I was introduced to it from Paul
Sellers. Here's an excerpt of his thoughts on it.</p>
<blockquote>
<p>I read an article about skilled artisans producing designs and selling their
wares to mainstream in a recent weekend issue of the Observer written by Emma
Love. In two minutes I thought to myself this surely has to be a joke. ... It
was filled with journalistic drivel where she used stock phrases of the day like
“Traditional skills”, “Tapping into the wider trend for natural materials,”.
What she didn’t see was that most of the goods were highly over priced and none
of them used traditional skills anywhere.</p>
<p>I wish that this journalist read a little more. I wish she had read this book
beforehand, perhaps she might have at least understood she didn’t understand;
that she couldn’t understand. Richard Sennett’s book is a well researched
masterpiece, unpretentious in every sphere and should be mandatory reading for
any and all craft students and mentored apprentices true to their craft. Sennett
carefully crafts his sentences with depth and meaning of an artisan, and whereas
I might not agree with every single word, I do see that he worked extremely
diligently to open the world Pandora left to get a point of view in place first
and then blasted through the myths and mysteries of why we do what we do, why
penchants exist as yearnings beyond our understandings and why, when we are
young, we should indeed listen to our hearts.</p>
</blockquote>
<p>from <a href="https://paulsellers.com/2018/03/theres-a-book-you-should-read/">There's a book you should read</a></p>
An Intro to Functional Reactive Programming in UIs2014-11-16T00:00:00+00:002014-11-16T00:00:00+00:00https://marcopolo.io/code/frp/<p>Maybe you've heard of <a href="https://facebook.github.io/react/">React</a>, <a href="https://github.com/swannodette/om">Om</a>,
or <a href="http://elm-lang.org/">Elm</a>, and wondering: what's the deal with
functional reactive programming (FRP)?</p>
<p>This post will act as primer on FRP using vanilla JS, but the ideas presented
here translate pretty easily in any language and UI system.</p>
<p>So let's start with an informal, pragmatic definition of FRP:</p>
<blockquote>
<p>Use streams of data to create the application state (data)</p>
</blockquote>
<p>And</p>
<blockquote>
<p>Build a UI given only the application state with pure functions (view)</p>
</blockquote>
<h2 id="streams-and-arrays">Streams and arrays</h2>
<p>You can imagine streams of data as a set of values over time.</p>
<p>A stream of numbers representing a counter would look like:</p>
<pre><code>[0,1,2,3,4,5,6,...]
</code></pre>
<p>Each number is essentially a snapshot of the value at that time.</p>
<p>Streams are similar to arrays, but the main difference is time.
An immutable array has all the values it will ever have when it is created, while a stream represents all the values that have happened and will
happen.</p>
<p>Here's a concrete example: You are an owner of an exclusive restaurant.
It's so exclusive that people have to make reservations months in advance.
Every night you have a list of people at your restaurant (because they've
already made reservations). Imagine the list being <code>[amy, sally, bob]</code>.
To count the number of guests, we would just reduce over the list
adding 1 for every guest. If we wanted to know how much each guest spent
we would map against a function that tells us the guest's bill.</p>
<p>This is just a normal array with normal map/reduce construct.
For completeness here's the equivalent code.</p>
<pre data-lang="javascript" class="language-javascript "><code class="language-javascript" data-lang="javascript">var guests = ["amy", "sally", "bob"];
var bills = { amy: 22.5, sally: 67.0, bob: 6.0 };
// Count the guests
guests.reduce(function(sum, guest) {
return sum + 1;
}, 0);
// => 3
// Get the bills
guests.map(function(guest) {
return bills[guest];
});
// => [22.5, 67, 6]
</code></pre>
<p>Unfortunately Sally had some bad fish and died after eating at your
restaurant, so everyone has cancelled their reservations and you are
now a fast food place. In this case you don't have a list of guests,
instead you have a <em>stream</em> of people who come in and order food.
<code>Frank</code> might come in at 10 am, followed by <code>Jack</code> at 2 pm. To get
similar data as before we would again map/reduce over the stream,
but since we are operating over a stream that never ends, the values
from map/reduce themselves become streams that never end.</p>
<p>Here is some equivalent pseudo code for streams that calculates
the <code>guestCounts</code> and the <code>guestBills</code>.</p>
<pre data-lang="javascript" class="language-javascript "><code class="language-javascript" data-lang="javascript">guests = [... time passes ..., Frank, ... time passes ..., Jack, ... ]
guestCounts = [... time passes ..., 1, ... time passes ..., 2, ... ]
guestBills = [... time passes ..., 5.50, ... time passes ..., 6.50, ... ]
</code></pre>
<p>So a stream is just like an array that never ends, and represents
snapshots of time.</p>
<p>Now that we have an intuitive idea what streams are, we can actually
implement them.</p>
<h2 id="streams-of-data">Streams of data</h2>
<p>A stream of numbers representing a counter would look like:</p>
<pre><code>[0,1,2,3,4,5,6,...]
</code></pre>
<p>If we wanted to keep track of how long someone was on our page,
we could just display the latest value of the counter stream
in our UI and that would be enough.</p>
<p>A more involved example: Imagine we had a stream of data
that represents the keys pressed on the keyboard.</p>
<pre><code>["p","w","n","2","o","w","n",...]
</code></pre>
<p>Now we want to have a stream that represents the state of the system,
say the amount of keys pressed.</p>
<p>Our key count stream would look like:</p>
<pre><code>["p","w","n","2","o","w","n",...]
=>
[ 1, 2, 3, 4, 5, 6, 7, ...]
</code></pre>
<p>This transformation would happen with a reducing function.</p>
<pre data-lang="javascript" class="language-javascript "><code class="language-javascript" data-lang="javascript">var keyCountReducer = function(reducedValue, streamSnapshot) {
return reducedValue + 1;
};
</code></pre>
<p>This function takes in the stream value, and a reduced value so far, and
returns a new reduced value. In this case a simple increment.</p>
<p>We've talked about streams for a while now, let's implement them.</p>
<p>In the following code, we create a function that will return an object with two
methods: <code>observe</code> for registering event listeners and <code>update</code> for adding a value
to the stream.</p>
<pre data-lang="javascript" class="language-javascript "><code class="language-javascript" data-lang="javascript">// A function to make streams for us
var streamMaker = function() {
var registeredListeners = [];
return {
// Have an observe function, so
// people who are interested can
// get notified when there is an update
observe: function(callback) {
registeredListeners.push(callback);
},
// Add a value to this stream
// Once added, will notify all
// interested parties
update: function(value) {
registeredListeners.forEach(function(cb) {
cb(value);
});
}
};
};
</code></pre>
<p>We also want to make a helper function that will create a new reduced stream
given an existing <code>stream</code>, a <code>reducingFunction</code>, and an <code>initialReducedValue</code>:</p>
<pre data-lang="javascript" class="language-javascript "><code class="language-javascript" data-lang="javascript">// A function to make a new stream from an existing stream
// a reducing function, and an initial reduced value
var reducedStream = function(stream, reducingFunction, initialReducedValue) {
var newStream = streamMaker();
var reducedValue = initialReducedValue;
stream.observe(function(streamSnapshotValue) {
reducedValue = reducingFunction(reducedValue, streamSnapshotValue);
newStream.update(reducedValue);
});
return newStream;
};
</code></pre>
<p>Now to implement the keypress stream and count stream.</p>
<pre data-lang="javascript" class="language-javascript "><code class="language-javascript" data-lang="javascript">// Our reducer from before
var keyCountReducer = function(reducedValue, streamSnapshot) {
return reducedValue + 1;
};
// Create the keypress stream
var keypressStream = streamMaker();
// an observer will have that side effect of printing out to the console
keypressStream.observe(function(v) {
console.log("key: ", v);
});
// Whenever we press a key, we'll update the stream to be the char code.
document.onkeypress = function(e) {
keypressStream.update(String.fromCharCode(e.charCode));
};
// Using our reducedStream helper function we can make a new stream
// That reduces the keypresses into a stream of key counts
var countStream = reducedStream(keypressStream, keyCountReducer, 0);
countStream.observe(function(v) {
console.log("Count: ", v);
});
</code></pre>
<p>Now with the new stream we can display it like we did before.</p>
<p>Which leads us into our next point...</p>
<h2 id="rendering-uis-given-data">Rendering UIs given data</h2>
<p>Now that we have a system for generating state through streams,
let's actually show something off.</p>
<p>This is where React.js shines, but for the purpose of this post we'll
build our own system.</p>
<p>Let's say at one point in time our data looks like:</p>
<pre data-lang="javascript" class="language-javascript "><code class="language-javascript" data-lang="javascript">{"Count":1}
</code></pre>
<p>And we want to render a UI that represents this information.
So we'll write a simple piece of JS that renders html directly from the map.
To keep it easy, we'll use the keys as div ids.</p>
<pre data-lang="javascript" class="language-javascript "><code class="language-javascript" data-lang="javascript">//Pure Function to create the dom nodes
var createDOMNode = function(key, dataMapOrValue) {
var div = document.createElement("div");
div.setAttribute("id", key);
// Recurse for children
if (typeof dataMapOrValue === "object" && dataMapOrValue !== null) {
Object.keys(dataMapOrValue).forEach(function(childKey) {
var child = createDOMNode(childKey, dataMapOrValue[childKey]);
div.appendChild(child);
});
} else {
// There are no children just a value.
// We set the data to be the content of the node
// Note this does not protect against XSS
div.innerHTML = dataMapOrValue;
}
return div;
};
// Render Data
var render = function(rootID, appState) {
var root;
// Check if the root id is even defined
if (document.getElementById(rootID) === null) {
// We need to add this root id so we can use it later
root = document.createElement("div");
root.setAttribute("id", rootID);
document.body.appendChild(root);
}
root = document.getElementById(rootID);
// Clear all the existing content in the page
root.innerHTML = "";
// render the appState back in
root.appendChild(createDOMNode(rootID, appState));
};
render("counter", { Count: 1 });
</code></pre>
<p>After running this code on a <a href="about:blank">blank page</a> we have a page
that says <code>1</code>, it worked!</p>
<p>A bit boring though, how about we make it a bit more interesting by updating
on the stream.</p>
<h2 id="rendering-streams-of-data">Rendering Streams of data</h2>
<p>We've figured out how streams work, how to work with streams, and how to
render a page given some data. Now we'll tie all the parts together; render
the stream as it changes over time.</p>
<p>It really is simple. All we have to do is re-render whenever we receive
a new value on the stream.</p>
<pre data-lang="javascript" class="language-javascript "><code class="language-javascript" data-lang="javascript">// Let's observe the countstream and render when we get an update
countStream.observe(function(value) {
render("counter", value);
});
// And if we wanted to render what the keypress stream tells us, we can do so
// just as easily
keypressStream.observe(function(value) {
render("keypress", value);
});
</code></pre>
<h2 id="single-app-state">Single App State</h2>
<p>A single app state means that there is only one object that encapsulates the
state of your application.</p>
<p>Benefits:</p>
<ul>
<li>All changes to the frontend happen from this app state.</li>
<li>You can snapshot this state and be able to recreate the
frontend at any point in time (facilitates undo/redo).</li>
</ul>
<p>Downsides:</p>
<ul>
<li>You may conflate things that shouldn't be together.</li>
</ul>
<p>Having a single place that reflects the whole state is pretty amazing,
how often have you had your app get messed up because of some rogue event?
or hidden state affecting the application, or an ever growing state
scattered around the application.</p>
<p>No more.</p>
<p>A single app state is a natural end to the directed acyclic graph that
we've created with streams.</p>
<pre><code>stream1 -> mappedStream
\
mergedStream -> appStateStream
/
stream2 -> reducedStream
</code></pre>
<h2 id="implementing-single-app-state">Implementing Single App State</h2>
<p>In our previous example we had two pieces of state,
the counter and the keypress. We could merge these together into one stream, and
then form a single app state from that stream.</p>
<p>First let's make a helper function that will merge streams for us. To keep it
general and simple we'll take only two streams and a merging function.
It will return a new stream which is the merge of both streams with the mergeFn.</p>
<pre data-lang="javascript" class="language-javascript "><code class="language-javascript" data-lang="javascript">// A merge streams helper function
var mergeStreams = function(streamA, streamB, mergeFn) {
var streamData = [null, null];
var newStream = streamMaker();
streamA.observe(function(value) {
streamData[0] = value;
newStream.update(mergeFn.apply(null, streamData));
});
streamB.observe(function(value) {
streamData[1] = value;
newStream.update(mergeFn.apply(null, streamData));
});
return newStream;
};
</code></pre>
<p>This implementation will call the merge function with the latest value from the
streams or null if the stream hasn't emitted anything yet. This means the output
can return duplicate values of one of the streams.</p>
<p>(As a side note, the performance impact of duplicate values can be mitigated
with immutable datastructures)</p>
<p>We want to put both the keypress and the counter in one object, so our
merge function will do just that.</p>
<pre data-lang="javascript" class="language-javascript "><code class="language-javascript" data-lang="javascript">var mergeIntoObject = function(keypress, counter) {
return { counter: counter, keypress: keypress };
};
</code></pre>
<p>Now to create the single app state stream, and render that single app state.</p>
<pre data-lang="javascript" class="language-javascript "><code class="language-javascript" data-lang="javascript">var appStateStream = mergeStreams(keypressStream, countStream, mergeIntoObject);
appStateStream.observe(function(value) {
render("app", value);
});
</code></pre>
<h2 id="final-code">Final Code</h2>
<p>Most of these functions are library functions that you wouldn't need to implement
yourself. The final application specific code would look like:</p>
<pre data-lang="javascript" class="language-javascript "><code class="language-javascript" data-lang="javascript">// Create the keypress stream
var keypressStream = streamMaker();
// Whenever we press a key, we'll update the stream to be the char code.
document.onkeypress = function(e) {
keypressStream.update(String.fromCharCode(e.charCode));
};
var keyCountReducer = function(reducedValue, streamSnapshot) {
return reducedValue + 1;
};
// Using our reducedStream helper function we can make a new stream
// That reduces the keypresses into a stream of key counts
var countStream = reducedStream(keypressStream, keyCountReducer, 0);
var mergeIntoObject = function(keypress, counter) {
return { counter: counter, keypress: keypress };
};
var appStateStream = mergeStreams(keypressStream, countStream, mergeIntoObject);
appStateStream.observe(function(value) {
render("app", value);
});
</code></pre>
<p>You can see a running version of this code <a href="http://jsfiddle.net/Ld3o1Lm5/2/">here</a></p>
<h2 id="the-render-a-closer-look">The render, a closer look</h2>
<p>So what does the render actually do?</p>
<p>Well, it clears the inner html of a containing div and adds an element inside of it.
But that's pretty standard, how are we defining what element is created?
Why yes, it's the createDOMNode function. In fact, if you wanted your data displayed
differently (e.g. in color, or upside down) all you'd have to do is write your own
createDOMNode function that adds the necessary styles or elements.</p>
<p>Essentially, the <code>createDOMNode</code> function controls what your UI will look like.
createDOMNode is a pure function, meaning for the same set of inputs, you'll
always get the same set of outputs, and has no side effects (like an api call).
This wasn't a happy accident, FRP leads to a
design where the functions which build your UI are pure functions!
This means UI components are significantly easier to reason about.</p>
<h2 id="time-travel">Time travel</h2>
<p>Often when people talk about FRP, time travel is bound to get brought up.
Specifically the ability to undo and redo the state of your UI. Hopefully, if
you've gotten this far, you can see how trivial it would be to store the data
used to render the UIs in an array and just move forward and backward to
implement redo and undo.</p>
<h2 id="performance">Performance</h2>
<p>If you care about performance in the slightest, you probably shuddered when
I nuked the containing element and recreated all the children nodes. I don't
blame you; however, that is an implementation detail. While my implementation
is slow, there are implementations (e.g. React) that only update the items and
attributes that have changed, thus reaping performance benefits with no cost
to the programmer! You are getting a better system for modeling UIs and
the performance boosts for free! Furthermore a lot of smart people are working
on React, and as it gets faster, so will your app. Without any effort on your
part.</p>
<h2 id="now-with-actual-libraries">Now with actual libraries</h2>
<p>A lot of what we wrote was the library to get streams up and running,
however those already exists (e.g. <a href="http://baconjs.github.io/">Bacon.js</a> and <a href="https://facebook.github.io/react/">React.js</a>)</p>
<p>A couple quick notes if this is your first time looking at React.js or Bacon.js.</p>
<p><code>getInitialState</code> defines the initial local state of the component.
<code>componentWillMount</code> is a function that gets called before the component
is placed on the DOM.</p>
<p><code><stream>.scan</code> is our reducing function in Bacon.js parlance.</p>
<pre data-lang="javascript" class="language-javascript "><code class="language-javascript" data-lang="javascript">// Our streams just like before
var keypressStream = Bacon.fromEventTarget(document.body, "keypress").map(
function(e) {
return String.fromCharCode(e.charCode);
}
);
var countStream = keypressStream.scan(0, function(count, key) {
return count + 1;
});
var KeyPressComponent = React.createClass({
getInitialState: function() {
return { count: 0, keypress: "<press a key>", totalWords: "" };
},
componentWillMount: function() {
this.props.countStream.onValue(
function(count) {
this.setState({ count: count });
}.bind(this)
);
this.props.keypressStream.onValue(
function(key) {
this.setState({ keypress: key });
}.bind(this)
);
// Add something extra because why not
this.props.keypressStream
.scan("", function(totalWords, key) {
return totalWords + key;
})
.onValue(
function(totalWords) {
this.setState({ totalWords: totalWords });
}.bind(this)
);
},
render: function() {
return React.DOM.div(
null,
React.createElement("h1", null, "Count: " + this.state.count),
React.createElement("h1", null, "Keypress: " + this.state.keypress),
React.createElement("h1", null, "Total words: " + this.state.totalWords)
);
}
});
React.render(
React.createElement(KeyPressComponent, {
keypressStream: keypressStream,
countStream: countStream
}),
document.body
);
</code></pre>
<p>jsfiddle for this code <a href="http://jsfiddle.net/jf2j62wj/10/">here</a>.</p>
<h2 id="closing-notes">Closing Notes</h2>
<p><a href="https://facebook.github.io/react/">React</a> is great for reactively rendering the ui.
<a href="http://baconjs.github.io/">Bacon.js</a> is a great library that implements these streams.</p>
<p>If you're looking to really delve into FRP:
<a href="http://elm-lang.org/">Elm</a> has a well thought out FRP system in a haskell like language.</p>
<p>If you're feeling adventurous give Om & Clojurescript a shot.
<a href="https://github.com/swannodette/om">Om</a> is a great tool that adds immutability
to React, and brings React to Clojurescript</p>
<p>Finally, Evan Czaplicki (Elm creator) did a <a href="https://www.youtube.com/watch?v=Agu6jipKfYw">great talk on FRP</a></p>
Introducing Servant, a Clojurescript library for web workers2013-10-01T00:00:00+00:002013-10-01T00:00:00+00:00https://marcopolo.io/code/servant-cljs/<h1 id="concurrent-programming">Concurrent Programming</h1>
<p>Javascript by default is single threaded, but web workers introduce
OS level threads. Concurrent programming is hard enough (in imperative
languages), so the webworker designers decided to circumvent a bunch of
concurrency problems by forbidding any shared data between threads. There are
better ways of doing this (read <em>immutability</em>), but we work with what we got.</p>
<p><br></br></p>
<h1 id="problems-with-web-workers">Problems with web workers</h1>
<p>I've done a couple projects with web workers. The biggest project being
<a href="http://cryptic.io">Cryptic.io</a>, which uses webworkers to efficiently
encrypt/decrypt large (GBs) files, and parallel {down,up}load file chunks. Here
are problems I've stumbled across:</p>
<ul>
<li>Everything about the web worker needs to be asynchronous, meaning callback hell</li>
<li>You need to think in a separate context for the web worker, you can't call any functions defined with the rest of your code.</li>
<li>Distributing workload effectively.</li>
<li>The problems only gets worse the more web workers you bring in.</li>
</ul>
<br />
<h1 id="enter-servant">Enter Servant</h1>
<p><a href="https://github.com/marcopolo/servant">Servant</a> is a super small (literally ~100 lines) library that solves all the
problems above, allowing you to write clean, multithreaded, ClojureScript. Even
though it's small, it does a lot.</p>
<ul>
<li>It allows you to define servant functions alongside the rest of your code, even using functions already defined in your
namespace.</li>
<li>It automatically balances work across webworkers.</li>
<li>It provides simple ways to do a normal (copy of arguments) or efficient (arraybuffer transfer) call
to webworkers, easily.</li>
</ul>
<h1 id="sharing-functions-and-predefined-variables">Sharing functions, and predefined variables</h1>
<p>This was the trickiest part of the library. I wanted the ability to define
useful functions, and use them in the webworker without having to copy it over
to a separate worker.js file. I solved it by using the same exact file for both
the main page (browser context) and the web worker. That, however, came with one
problem; you have to explicitly declare code that should run on the webworker
and code that runs in the browser. Like so:</p>
<pre data-lang="clojure" class="language-clojure "><code class="language-clojure" data-lang="clojure">(ns servant.demo
(:require [servant.core :as servant]
[servant.worker :as worker]))
(defn window-load [] (.log js/console "this runs in the browser"))
(if (servant/webworker?)
(worker/bootstrap) ;;Sets the worker up to receive messages
(set! (.-onload js/window) window-load))
</code></pre>
<p>As part of that caveat, the webworker can only see code that it can get to.
Anything defined in window-load would not be visible to the webworker. Now let's
take a look at how we can define a servant function, using the <code>defservantfn</code>
macro.</p>
<p>We need to use a special function, <code>defservantfn</code> to define functions that will
serve as our "access points" to the web worker.</p>
<pre data-lang="clojure" class="language-clojure "><code class="language-clojure" data-lang="clojure">(ns servant.demo
(:require-macros [servant.macros :refer [defservantfn]]))
(defn make-it-funny [not-funny]
(str "Hahahah:" not-funny))
(defservantfn servant-with-humor [your-joke]
(make-it-funny your-joke))
</code></pre>
<p>The <code>defservantfn</code> macro simply calls a defn with the
same arguments, and registers that function with a hashmap atom for the
webworker. The key is the hash of the function and the value is the function
itself. The webworker needs to be able to know what function the browser is
referring in a message, so I use the function's hash as a token that
the browser context and webworker can both agree on. The function's
<code>.toString()</code> value could have worked just as well.</p>
<p>I should also mention, for efficiency reasons, Servant keeps a pool of N
webworkers (you specify N) alive (until you explicitly kill them) so you only
pay for the webworkers once. You control when the webworkers are created with
<code>servant/spawn-servants</code>.</p>
<h1 id="workload-balancing">Workload Balancing</h1>
<p>Core.async is simply amazing, it took this tricky problem and made it trivial. The solution is 4 lines.
The solution for Servant is:</p>
<ul>
<li>spawn N number of workers and place them in a buffered (of size N) channel.</li>
<li>Take workers from the channel as you use them.</li>
<li>Put them back when you get your result.</li>
</ul>
<p>This is so beautifully simple. I just write the behavior I want, and core.async
handles the messy state. If all the webworkers are busy the code will "block"
until a webworker is free. What this means for you as a user, is you don't have
to think about which worker is available to run your code.</p>
<h1 id="configurable-message-types">Configurable message types</h1>
<p>Now the whole point of using webworkers is to be as fast as possible. Sometimes
you can't even afford copying data to the webworker (especially if the data is
big, like at <a href="http://cryptic.io">Cryptic.io</a>). Servant provides a way to access
webworkers' nifty <a href="https://developer.mozilla.org/en-US/docs/Web/Guide/Performance/Using_web_workers#Passing_data_by_transferring_ownership_(transferable_objects)">arraybuffer transfer context ability</a>.
Take for example:</p>
<pre data-lang="clojure" class="language-clojure "><code class="language-clojure" data-lang="clojure">(defservantfn get-first-4bytes-as-str [arraybuffer]
(let [d (js/DataView. arraybuffer)]
(.toString (.getUint32 d 0) 16)))
</code></pre>
<p>That function expects an arraybuffer and returns a string. If we wanted to be
efficient about it (and didn't care about getting rid of the arraybuffer) we can
make the call using the <code>servant/array-buffer-message-standard-reply</code> fn instead
of the <code>servant/standard-message</code>. So the efficient result would be:</p>
<pre data-lang="clojure" class="language-clojure "><code class="language-clojure" data-lang="clojure">(def arraybuffer (js/ArrayBuffer. 10))
(def d (js/DataView. arraybuffer))
(.setUint32 d 0 0xdeadbeef)
(def result-channel
(servant/servant-thread
servant-channel
servant/array-buffer-message-standard-reply
get-first-4bytes-as-str arraybuffer [arraybuffer]))
</code></pre>
<p>The arguments to servant-thread are:</p>
<ul>
<li><code>servant-channel</code> - channel that contains the available workers</li>
<li><code>servant/array-buffer-message-standard-reply</code> - A function that defines how the <code>.postMessage</code> function will be called (a.k.a mesage-fn)</li>
<li><code>get-first-4bytes-as-str</code> - The servant function we defined earlier</li>
<li><code>arraybuffer</code> - our argument to the function</li>
<li><code>[arraybuffer]</code> - a vector of arraybuffers that are going to be transferred</li>
</ul>
<p>The message-fn can be anything, but I think servant has you covered with:</p>
<ul>
<li><code>standard-message</code> : Copies all the data both ways</li>
<li><code>array-buffer-message</code> : <em>Can</em> transfer the context both ways</li>
<li><code>array-buffer-message-standard-reply</code> : <em>Can</em> transfer the context when making the call, <em>won't</em> transfer the context coming back</li>
</ul>
<p>There is a reason why array-buffer-message isn't just the standard. You need to
explicitly tell the postMessage call that you want to transfer arraybuffers. So
to transfer context you need an additional argument, an array of arraybuffers.
You also need to make sure the defservantfn returns a vector of results and an
array of arraybuffers [result [arraybuffer1]] if you want to transfer the
arraybuffer from the worker to the browser context. I figured if you wanted
that you could use it and deal with the extra argument, if you didn't you could
write your functions how you normally would.</p>
<h1 id="examples">Examples</h1>
<p>I wrote two examples using the servant library:</p>
<ul>
<li>The first is a <a href="https://github.com/MarcoPolo/servant-demo">simple demo</a> showing several use cases.</li>
<li>The next is more featured demo that can encrypt/decrypt large files efficiently using webworkers.</li>
</ul>
<h1 id="last-thoughts">Last thoughts</h1>
<p>I used to curse the name webworkers. They brought gifts of speed at the cost of
complexity. Servant is different, it doesn't sacrifice simplicity or
efficiency. I'm pretty excited at the ease of using webworkers with servant, and
I hope you have fun making an amazing, multithreaded Clojurescript application!</p>
<br />