Robert Ou Labs

Unofficial open-source place-and-route for Xilinx Coolrunner-II CPLDs

2018-05-16T01:00:00-07:00

Recently, I have been working on an open-source tool for producing bitstreams for Xilinx Coolrunner-II CPLDs that I have named xc2par. It is finally at a point where I no longer feel embarrassed to have other people try and use it.

Quick start

The first tool you will need is yosys. You need a version that contains commit 14e49fb05737cfb02217b2aaf14d9f5d9e1859da, but it is recommended to use the latest master branch. Follow the instructions in the yosys README to install it (normally I would point people to my pre-built binaries, but those are currently broken).

Next, you will need to install the compiler for the Rust programming language.

After installing Rust, clone the "openfpga" repository. Change directories into the src/xc2par directory and run cargo build --release. When this completes, the final command-line tool will be in target/release/xc2par.

At this point you are ready to try compiling some HDL! Unfortunately, xc2par does not currently accept exactly the same code as the proprietary Xilinx tools, and there is also currently no documentation on what is or is not accepted. You will have to either guess or read the source code to figure out what currently works. Most notably, .ucf constraint files are not supported (you must use LOC attributes, and they must use the function block and macrocell number rather than a package pin), and automatic insertion of BUFGs is not supported. However, to get started, the following is some example Verilog for an LED blinker:

module top(led0, led1, led2, led3, clk_);

// NOTE: Must use LOC attributes rather than a .ucf file
(* LOC = "FB1_9" *)
output led0;
(* LOC = "FB1_10" *)
output led1;
(* LOC = "FB1_11" *)
output led2;
(* LOC = "FB1_12" *)
output led3;
(* LOC = "FB2_5" *)
input clk_;

// NOTE: Must manually instantiate BUFG
wire clk;
BUFG bufg0 (
    .I(clk_),
    .O(clk),
);

reg [23:0] counter;

assign {led3, led2, led1, led0} = counter[23:20];

always @(posedge clk)
    counter <= counter + 1;

endmodule

To compile this code, use the following commands:

yosys -p "synth_coolrunner2 -json blinky.json" blinky.v
./target/release/xc2par -p xc2c32a-4-vq44 blinky.json

At this point, if everything worked correctly, there will be a blinky.jed file in the same directory as the blinky.json file. It should now be possible to program this file into a CPLD to test it out. However, embarrassingly, there currently isn't any code in the xc2par project that can help with that. You will either need to use

ISE/iMPACT (either to generate a .svf file or to program parts directly)
Andrew Zonenberg's jtaghal (only supports the 32A and 64A parts)
Some other JTAG programmer that can read Xilinx .jed files

The Origins of xc2par

In the beginning, there was only Andrew Zonenberg. Back in 2014, Andrew took some CPLDs and exposed them to a slew of nasty chemicals and some high-energy electrons. This culminated in a talk presented at REcon 2015. At this point in time, the PLA (AND and OR gates) and interconnect were understood but the macrocell configuration was not. Although Andrew had always wanted to write an open-source compiler for these chips, yosys did not have support for sum-of-products architectures at this point in time. This combined with Life™ led to the project being temporarily shelved.

Robert Has Joined the Party

Being a "hacker" with extremely varied interests, I had been lurking Andrew's work for years (since at least 2012). I had an interest in both programmable logic devices and reverse engineering. However, since I was pretty busy with Life™ of my own, I had never considered ever contacting Andrew or collaborating. However, one of Andrew's projects combined with perfect timing changed this.

Quick Detour: Introducing GreenPak

I first heard about these interesting tiny programmable logic devices from an article on Hackaday. An interesting feature of these parts is that the bitstream format is completely documented in the datasheet. After reading the article, I purchased a GreenPak development kit with the idea of making "some kind of domain-specific-language" for programming these parts (I was not ambitious enough to consider supporting a traditional HDL, and I was not yet aware that yosys existed at the time). However, since I was still working on Life™ (being a university student), this project also got shelved.

However, in the meantime, Andrew was working on his open-source Verilog flow for GreenPak. This tool just so happened to be released almost exactly around the time that I finished my undergrad degree. Since I had planned to take a year off to remain "funemployed," I decided to introduce myself to Andrew and joined the ##openfpga IRC community on Freenode somewhere around this time. Andrew eventually suggested that I could work on a place-and-route for Coolrunner-II parts (yosys support for sum-of-products had also gotten added around this time).

Finishing the Reverse Engineering

When Andrew had originally set aside the Coolrunner-II toolchain project back in 2015, the macrocell configuration was not yet understood. Andrew suggested that I ignore the macrocell-related parts and write tooling to handle the other parts of the device first. Macrocells would be configured by copying the entire block of bits from an existing ISE-generated design. Dissatisfied with this answer, I decided to schedule an IRL hackathon with Andrew to figure out all of the remaining bits. This resulted in the following:

Note that this diagram is not accurate and has errors. A more accurate diagram needs to be drawn eventually.

Now with all the bits understood, we could begin writing tooling that did not require any strange hacks.

xc2bit and xc2par are Born

At the end of May 2017, the first commit of the "xc2bit" code is created. This is a Rust library for performing low-level manipulations of Coolrunner-II bitstreams. This is analogous to Andrew's never-finished libcrowbar library. The first commit of xc2par is created a few days later.

First Attempt: xbpar

The first version of xc2par was supposed to use xbpar, Andrew's generic C++ framework for writing place-and-route tools for "crossbar interconnect" architectures. The internals of xbpar are described in Andrew's blog post about the GreenPak Verilog tooling, but the general idea is that the algorithm takes as input two graphs, one modeling the target device and one modeling the user's design, and attempts to check if the user's design graph is a subgraph of the device structure graph. It does this by attempting to "pair up" nodes of the two graphs and then checking if the edges between the nodes of the design graph exist in the device graph. If any edges do not, it attempts to find something better using simulated annealing.

Trying to use this library for xc2par had a number of issues. The first issue was "ideological." I wanted to use Rust for xc2par because I did not like C++ but wanted more powerful libraries and abstractions than are available in C. Unfortunately, xbpar is a C++ library that isn't very friendly to being interfaced with a different programming language. Among other things, device-specific functionality is implemented by deriving from an xbpar base class and overriding some methods. However, I persevered and attempted to write glue code between the two languages. Unfortunately, all of this glue code ended up being more lines of code than xbpar itself.

A second issue was that the specific mechanism I had used for describing graphs for xbpar caused extremely poor behavior. I had described every "enable-able or disable-able" feature of the macrocell as an individual node in xbpar (so the XOR gate of a macrocell would be a node, the register would be a different node, and the IO pad would be yet another different node). However, this would cause the following behavior:

The greedy initial placement algorithm packs together "the wrong" set of "macrocell pieces." For example, it would place the IO pad for pin a and the XOR gate for pin b both at function block 1 macrocell 1. Meanwhile, the XOR gate for pin a and the IO pad for pin b both get placed at function block 1 macrocell 2.
The algorithm would notice that there are not proper edges between the nodes. In the above example, there is an edge in the device graph between the XOR and IO pad of FB1_1, but there are no edges between the IO pad of FB1_1 with the XOR gate of FB1_2 or between the XOR gate of FB1_1 with the IO pad of FB1_2. Therefore, all components of pins a and b will get added to the list of nodes contributing to the bad scores.
The simulated annealing step picks one of the "bad" nodes and tries to move it. It ends up moving e.g. the XOR gate of one pin to a completely different random spot. This does not improve the situation since the XOR gate needs to be in the same location as all of the other "macrocell bits" for the same pin (but it has just been moved somewhere random and unrelated).
The cycle repeats without ever making progress.

One possible solution for this problem is "why not just model the entire macrocell as one giant node with a bunch of attributes?" This solution might work, but it has difficulties with optimal packing of buried nodes. The Coolrunner-II architecture has the following quirk:

A macrocell that needs to output to a pin cannot share a macrocell site with anything else
A buried combinatorial feedback node can share a macrocell site with both a direct pin input as well as a pin input that goes through the register
A buried registered feedback node can share a macrocell site with a direct input pin only, and it can only share this site if the registered feedback node is not also simultaneously a combinatorial feedback node.

It was not clear to me how to express this quirk in a way that was actually useful at fixing the above behavior. Since this also required me to write a lot of device-specific code, I was wondering if I could somehow avoid having to do that.

Second Attempt: SAT/SMT

While I was encountering the above problems, I discussed them with one of my housemates. They suggested that I should try feeding the place-and-route problem into a SAT/SMT solver. The idea behind this was that:

Modern SAT/SMT solvers have quite a few advanced algorithms in them
The problem size seemed "small"

With this suggestion, I quickly hacked together a naive lowering of the place-and-route problem into a SMT problem. Since I added this hack into xbpar, it was possible to test it for both GreenPak designs and Coolrunner-II designs.

Unfortunately, although I did not keep very detailed measurements, this approach did not work very well. A major problem was that this was my first attempt at using SMT solvers. Firstly, I selected the QF_LIA theory because I was expecting that I might use it to eventually implement timing-driven place-and-route. Unfortunately, this restricted me to using the SMT solvers that tended to be relatively slower. Secondly, the particular lowering that I chose was relatively naive and probably did not help the SMT solver to try to optimize the problem. Overall, the results were that Coolrunner-II designs never completed in a reasonable amount of time. GreenPak designs would complete relatively quickly if they did indeed fit in the device (solver returns SAT) but would take forever if they did not actually fit in the device (solver returns UNSAT).

Second-and-a-Half Attempt: Naive backtracking search

Since using a SMT solver did not appear to be working, I decided to try other "brute-force" techniques. As part of this, I wrote a naive backtracking search solver to assign design graph nodes to device graph nodes. I implemented only the "min-remaining-values" optimization. This actually completed incredibly quickly for GreenPak designs that fit. However, GreenPak designs that did not fit were still somewhat slow, and Coolrunner-II designs still did not work at all.

Since this solver was a completely standalone program and written in Python, I could quickly add hacks to it and observe how it behaved. This ended up giving me the following insights:

There are symmetries or "shortcuts" that naive solvers cannot easily take advantage of (even with methods like forward checking or arc consistency). For example, every product term in a PLA is accessible to every OR gate, and so it does not "really matter" where these product terms are placed. However, backtracking search cannot easily discover this. The same goes for the ZIA - every ZIA row mux output is accessible to every product term, and it does not matter which specific rows are chosen.
Following on from the above, the only placements that "really matter" are the placements of the macrocells. Everything else can be "mostly" done with greedy assignments. (However, note the scare quotes. Read below a less simplified explanation)
Backtracking search cannot be used even for just placing macrocells. An approximation must be used here. This is because, at worst, backtracking search can take exponential time and explore every single possibility. If the algorithm happens to be working on the XC2C512 and is trying to place every macrocell into every possible site, this is $512!\approx2^{3875}$ possibilities. Even if we try to get a much lower bound by pretending that all macrocells in the user design are identical and can be freely swapped, we still have too many worst-case possibilities. To count how many, we would like to know the "number of ways to put identical (unlabeled) objects into labeled bins, where all of the bins have a maximum size." Unfortunately my combinatorics is a bit rusty, but @eggleroy managed to help me out with the following formula:
$$ f(N, X, S) = \begin{cases} 0 & \mbox{if } N < 0\\ 1 & \mbox{if } N = 0\\ 0 & \mbox{if } N > 0, X = 0\\ \sum_{k=0}^S f(N-k, X-1, S) & \mbox{otherwise} \end{cases} $$
where $N$ is the number of objects (macrocells), $X$ is the number of bins (function blocks), and $S$ is the maximum size of each bin (macrocells per function block). $N$ depends on the size of the user design while $X$ depends on the specific CPLD device chosen. $S$ is fixed for all devices in the family. Somewhat unintuitively (at least to me initially), this function is maximized (for our purposes) when $N=256, X=32, S=16$. Computing this yields $33926222943232064651934548544483314385 \approx 2^{125}$.

If anybody who is more familiar with SAT/SMT solvers would ever like to continue to play with these ideas, there is some code here.

Detour: xc2jed2json

While working on reverse engineering efforts, I wrote a tool that can "decompile" Coolrunner-II bitstreams back into a .json netlist that can be loaded into yosys. After some (not yet ready for production) steps, this can be turned into something vaguely understandable. Andrew has given a talk about this. More on this in the future.

Final Attempt: Explicitly Modeling Shortcuts

At this point I was rather frustrated with working with xbpar. Given the insights I obtained from the backtracking search implementation, I decided to write a completely new place-and-route tool. This tool explicitly models all of the "shortcuts" that can be taken within an individual function block but still uses heuristics for placing macrocells. The algorithm works mostly as follows:

There is a subroutine that takes as input an assignment of macrocells to macrocell sites and outputs a placement of product terms and a map of ZIA usage. For each function block, this subroutine does:
1. Finds all of the product terms referenced by macrocells in this function block. The algorithm knows about two categories of product terms: those that have some kind of constraint on their placement (because they go into the dedicated product terms (either the per-macrocell PTA/B/C or the per-function-block CTx control terms) or because they have an explicit LOC constraint), and those that have no such constraints and just go into the OR array. The "shortcut" being taken here is that only those product terms that have a constraint need to be specifically handled. Everything else can be greedily assigned.
2. For the first type of product terms (those that have restrictions on their placement), store all the candidate locations for each of them. Possibly filter by the explicit LOC constraints.
3. Use backtracking search to assign the locations of these product terms. If it does not succeed, return an error. This backtracking search is much smaller because macrocells will almost always use the per-macrocell PTA/B/C terms. The only terms that can be contended are the per-function-block CTx control terms. There are only 4 of them, and at worst the algorithm can try assigning one product term from each macrocell into each of the control terms. This is $16^4=2^{16}$ which can complete almost instantly.
4. Take the remaining product terms (that have no restrictions on their placement) and assign each one to the first open site. If there are no open sites left, return an error.
5. Independently of the product term placement logic, gather up all inputs into the product terms. The ZIA muxes need to be programmed to provide these as inputs into the function block. Because we are starting with a complete macrocell placement, we know which specific choices we want to search for in the ZIA.
6. Search the ZIA data table for all of the rows that can possibly give each output. Then use backtracking search to pick a feasible assignment. I do not currently have a complexity estimate for this step, but the ZIA was originally designed such that "almost all" combinations should be able to be routed. The "shortcut" here (that was also the insight originally used by Xilinx to design the ZIA) is that it does not matter which row of the ZIA is used to obtain each specific input. It only matters that the entire set of inputs is somehow present (because AND is commutative).
The actual PAR tool first reads a yosys .json file and parses it.
We create a data structure consisting of "nodes" and "nets" connecting them. The nodes are of a type such as "AND gate," "OR gate," "register," or "XOR gate," but they are not distinguished from each other (they are all a generic "node" data type). Net objects store their source and sink nodes, but nets can refer to any nodes whatsoever. Essentially, this step takes the yosys data model and filters/parses the individual cell types without any checking of connectivity between them. It turned out that this data structure was too difficult to work with in subsequent steps, so there is also a second data structure.
The second data structure no longer models nets. Instead, there are multiple distinguished types of nodes that directly refer to each other. For example, a node corresponding to an OR gate will directly refer to the AND gates that feed it. All of the functionality related to macrocells is also packed into a large "macrocell" node with subcomponents for "the register part," "the IO part," and "the combinatorial (XOR) part." This step does most of the checking for proper connectivity.
Macrocells are initially placed with a greedy placement algorithm. One observation from earlier experiments was that most simple designs can be placed with just greedy placement.
Run the subroutine described in 1. If it succeeds, the place-and-route is done!
If it did not succeed, something needs to be improved. Instead of using simulated annealing, xc2par uses "min-conflicts." When the subroutine described in part 1 fails, it also tries to calculate which macrocells in the placement contribute most to the failing. To do so, it simply brute-force deassigns each macrocell in the design and checks how many conflicts are left. The min-conflicts algorithm then chooses a "bad" macrocell to move weighted by the number of conflicts each macrocell causes.
Once a macrocell to move is chosen, the algorithm tries every location in the device and tries swapping the macrocell into that location (ignoring those that have LOC constraints on them). The algorithm is looking for a site that improves the number of conflicts the most.
If such a site is found, commit to the given swap. If not, randomly perform a swap. If an iteration limit is exceeded, give up.

After this is complete, xc2par then outputs a programming file.

Future work

Since this is an early prototype, there is always room for improvement. Hop into ##openfpga on Freenode to discuss and report bugs.

Notable areas needing improvement are:

More tests
Support for IO standards (currently hardcoded)
Directly generating .svf files (or other means of programming chips)
Ability to use the XOR-with-PTC functionality more effectively
Extra passes to handle Xilinx-isms (e.g. things described in the "Xilinx CPLD Libraries Guide")
Devices other than the XC2C32/A

Lab notes on decapping ICs with sulfuric acid

2017-10-09T05:00:00-07:00

Recently, I've been working on setting up my lab for decapping ICs for the purpose of reverse engineering and future invasive attacks. Here are some notes from my initial round of using sulfuric acid to perform the decap.

DISCLAIMER: I am not trained as a chemist. This experiment is dangerous. Perform your own risk assessment before repeating. Repeat this experiment at your own risk.

Materials

Glass beakers (one 50mL, one 250mL, and one 1000mL)
Watch glass
Sulfuric acid drain cleaner. I used "Liquid Lightning" brand which was purchased at a local Walmart. This product is a clear liquid with no dyes. It is thick and viscous and probably at least 90% concentration. I have not attempted to determine the exact composition (including possible inhibitors).
A cheap kitchen hotplate with no stirring capability, purchased from Amazon

That's it! There is actually very little equipment needed if you only want to completely dissolve the plastic package on a chip and leave only a bare die. However, the following additional items are recommended:

Sand
Baking soda
Type-K thermocouple
Beaker tongs

And finally, the following safety equipment is required:

A cheap box fan used to exhaust fumes.
A full face respirator. These two items are used in lieu of having a proper fume hood.
Chemical-resistant gloves

Procedure

Prepare the 1000mL beaker by filling it with about 250mL of tap water. We will need this later.
Place the chips to be decapped into the 50mL beaker. Caution: If the chip package is physically very large, it is strongly recommended to mechanically trim it to just the material around the die. I currently have not tried this, but others have recommended either carefully grinding until bond wires are visible or grinding the back side of the package until the paddle the die is attached to is visible.
Pour some sulfuric acid drain cleaner into the beaker. Caution: Do not fill the beaker more than one-quarter with acid. Otherwise there will be a large risk of the acid bubbling over and overflowing out of the beaker.
VERY VERY IMPORTANT: Place the 50mL beaker containing the ICs and acid inside the 250mL beaker. Optionally place some sand into the large beaker to improve thermal transfer.

If you do not do this, you run the risk of hot sulfuric acid overflowing out of the beaker and landing on the surface of the hotplate. Because this cheap hotplate is intended for the kitchen, this hot acid will instantly react with some plastic coating on the hotplate and produce a huge cloud of probably-toxic white smoke. This will leave a detectable odor even after several days of blowing air through the room with fans. Depending on your living situation, this will also escalate the problems with that one other housemate that you are not on speaking terms with.
Place the two nested beakers slightly off the center of the hotplate. Cover with a watch glass.
Ensure the fume exhausting box fan is turned to max speed. Turn the hotplate temperature to "MAX" and turn the hotplate on. Occasionally monitor the temperature using a thermocouple. I was monitoring the temperature by poking a thermocouple into the gap between the beaker and the hotplate (which probably overestimates the actual temperature of the solution).
Initially, not much will appear to be happening. However, as the acid nears its boiling point (with the hotplate temperature almost at 400 degrees C), the mixture will suddenly begin bubbling vigorously. If it looks to be going out of control, immediately shut of the heating. Otherwise, leave the mixture heating for about five minutes. During this process, white fumes are emitted. These are likely sulfur trioxide. Avoid coming into contact with them. At this point, you must be absolutely sure that the respirator is properly sealed against your face and that the box fan is blowing the fumes away. If you somehow determine that this is not the case, abort the experiment and run away as fast as you can.
Once the chips look sufficiently dissolved, shut off the hotplate. Wait for the apparatus to cool.

If you are in a hurry, you can remove the beakers from the hotplate once touching the top of the large beaker no longer burns your fingers. This will make cooling slightly faster because the surface of the hotplate has quite a bit of thermal mass. Absolutely do not attempt to do this unless the hotplate temperature is below about 100 degrees C.
Once everything is sufficiently cooled, carefully pull the small beaker out of the large beaker. You should definitely use beaker tongs for this. If like me you do not, you risk spilling the contents of the beaker onto the carpet because you did not get a good grip on the beaker. If this happens, quickly pour baking soda on it to neutralize it.

Very slowly pour out the spent acid into the 1000mL beaker that was filled with water. Make sure to avoid any splashes. Because we are pouring a concentrated acid into water, use a huge beaker with a huge amount of water (but still significantly less than the maximum volume of the beaker).
Because the waste water and acid solution is still acidic, it is recommended to neutralize it with baking soda. If you do this, make sure to add the baking soda slowly or else the solution will spill out of the beaker. (This is another reason why this waste beaker is 1000mL.) However, if you have an antagonistic relationship with your landlords like me, you might skip this step and just pour the waste solution down the drain with the tap turned on to further dilute.
It should now be possible to rinse out the small 50mL beaker with water and search for the freed silicon dies. If for some reason some dies did not get freed, repeat the decap procedure until they are.

Results

(Re-)Hacking the Sega Saturn

2017-07-14T18:20:00-07:00

A while ago, I decided to duplicate the existing hack to extract the SH-1 CD-ROM controller firmware from the Sega Saturn. I gave a short presentation about this at the Mountain View Reverse-Engineering Meetup a few days ago. You can find the slides here.

TPM2-sealed LUKS encryption keys

2017-03-02T03:00:00-08:00

Introduction

Recently, I upgraded my NAS machine and decided I wanted to set up full disk encryption with the disk encryption key sealed inside a TPM. This setup is very similar to Microsoft's BitLocker disk encryption. Just to make it more difficult for myself, I decided to use a TPM2 device rather than an old TPM1.2 device. There are existing projects that implement this functionality for Linux using TPM1.2, but I did not find one for TPM2.

TPM? TPM2?

A TPM, or Trusted Platform Module, is a small and supposedly tamper-resistant chip that can be added to a computer in order to safely store some secret information. Among other features, a TPM can be programmed such that it only allows access to its secret information if the computer has booted up in the correct state (e.g. has not been booted from an external USB thumb drive). If a disk encryption key is stored in a TPM, then it can be configured to automatically unlock the root disk during a normal boot but not unlock it when something in the boot configuration changes.

There are two major versions of TPM devices, 1.2 and 2.0. A good description of the changes in 2.0 can be found here.

Hardware setup

I am using an ASRock E3V5 WS motherboard with ASRock's TPM-S 2.0 device which is based on a Nuvoton NPCT650. The TPM module just needs to be plugged into the matching connector near the bottom of the motherboard.

In the UEFI configuration for the motherboard, I ensured that the TPM module was detected, that the software/Intel ME emulated TPM was disabled, and that the TPM was set up to use SHA-2 hashes (which actually causes both SHA-1 and SHA-2 hashes to be available).

Software setup

On the software side, I am using IBM's TPM 2.0 TSS along with some custom shell scripts. Unfortunately, despite claims on James Bottomley's linked blog, Ubuntu 16.04 LTS did not seem to have a package for the IBM tools. I built the tools from source after applying this patch that disables the building of a shared object for the common code (preferring to statically link it instead). After compiling, I followed James Bottomley's instructions for creating a 81000001 key handle.

At this point, I created the /opt/tpmdisk directory and placed the .sh files from my repository into it. Finally, I copy the zzz-tpmdisk file into /usr/share/initramfs-tools/hooks/.

Disk setup

At this point, I create a 32-byte file of random data and store it at /keys/rootkey.bin. Ensure that this file is readable only to root. The purpose of this file is to allow the shell scripts to add/remove LUKS key slots without needing the "recovery" password. Because this file is being stored on the disk that is to be encrypted, it should not introduce an extra security vulnerability. I use the cryptsetup utility to set key slot 7 to the randomly generated file and key slot 6 to a "recovery" password. /opt/tpmdisk/new-disk-key.sh can now be run (as root) to create a third disk encryption key that will be sealed to the TPM. The sealed key will be stored in /opt/tpmdisk/sealedkey_{priv,pub}.bin

Finally, ,keyscript=/opt/tpmdisk/unseal-disk-key.sh is appended after the luks option in /etc/crypttab and the initramfs is regenerated.

Code walkthrough

new-disk-key.sh

This script is relatively straightforward. It creates a temporary directory and then creates inside it a temporary file with 32 random bytes. This will be the new sealed-to-the-TPM encryption key, but it is currently in the clear. The script removes the existing data in the appropriate LUKS key slot and assigns the newly-generated key to that key slot. The script also invokes seal-to-pcrs.sh to perform the actual TPM sealing operation. The script finally cleans up and removes the cleartext copy of the new key.

seal-to-pcrs.sh

This script performs the bulk of the work to seal secret data to a TPM. The data is sealed such that "PCR values" in the TPM must match certain values in order for the data to be unsealed. "PCR values" are special append-only registers that log various steps of the boot process. The PCRs that are used are specified (redundantly) by the PCRS and PCR_BITS variables in the script. The PCR_BITS variable is a hex value that contains 1 bits in the bit position corresponding to values in the PCRS variable.

The PCR values chosen here are modified from the BitLocker recommendations. However, note that all PCRs 8 and above are created by the operating system rather than by the BIOS/UEFI, and Linux does not seem to use any of them. Also, for a TPM2 system, PCR 7 contains the UEFI Secure Boot configuration, so that is included in my list of PCRs.

This script first gathers the current value of all the PCRs into a temp file (one line for each value). These values are then combined into a textual policy file. Note that TPM2 policies are rather complicated and can contain an arbitrary amount of AND and OR terms. Here we are just creating a single AND term with all of the PCR values. The textual policy is then converted into a binary policy. The data is finally sealed to the TPM using that policy file.

unseal-disk-key.sh

This simple script tries to use unseal-from-pcrs.sh to unseal the disk encryption key. If this fails for any reason (such as the kernel having been modified/upgraded), it falls back to the normal askpass utility that belongs to cryptsetup (where the "recovery" password can be entered). This is necessary because otherwise the Ubuntu initramfs will loop forever (or a very long time) repeatedly trying and failing to unlock the disk.

unseal-from-pcrs.sh

This script performs the bulk of the work to unseal data from a TPM. First, it loads the sealed data from files on disk into the temporary storage of the TPM. It then starts a session where policy verification can be performed. It then instructs the TPM to try to check the policy using (hopefully) the same list of PCRs that was used to create it. It then calls the unseal command and, if the PCR values match, the TPM will unseal the data. The script finally cleans up.

My First Rust Program - Dynamic DNS Updater

2017-03-01T20:00:00-08:00

I recently needed a dynamic DNS updater client. Being rather unsatisfied with the existing ones, I decided to write my own in Rust. This is the first actual Rust program I have written. If you attempt to use this on your own machine, do be aware that it has very limited error handling.

The primary advantages of my dynamic DNS updater are:

Does not require polling. It instead uses Netlink to get actively informed of IP address changes (and as such only works on Linux).
Supports both IPv4 and IPv6
- Supports updating additional extra hosts that share the same IPv6 /64 prefix

However, this program does require nsupdate as an external runtime dependency to actually perform the DNS updates.

This tool takes three command-line arguments: the "upstream" network interface, the "downstream" network interface, and a configuration file. The "upstream" network interface is used to update the IPv4 and IPv6 address of the main host. The "downstream" network interface will have its IPv6 /64 prefix extracted and combined with suffixes specified in the config file to update the additional extra hosts. The config file must be formatted exactly as follows:

<path to nsupdate key file>
<main hostname>
<extra hostname> <extra hostname address suffix>
...
<extra hostname> <extra hostname address suffix>

For example, the following is a working configuration file:

/etc/rqou-ddns/dyn.rqou.com.key
testhostname.dyn.rqou.com
auxhostname1.dyn.rqou.com abcd:ef00:1234:5678
auxhostname2.dyn.rqou.com abcd:ef00:1234:5679

Yet Another Minecraft⟺IRC Bridge

2017-03-01T19:00:00-08:00

A while ago, I decided to write yet another server wrapper for Minecraft that would bridge chat messages between Minecraft and an IRC channel. The program can be found here. This has been done a decent number of times before. The primary reason I created yet another one was because I wanted an excuse to use Python's somewhat-new asyncio functionality. Unlike many of the existing implementations, this implementation works by wrapping stdin/stdout of the server process rather than by modifying the jar directly. It therefore needs minimal updating as Minecraft itself updates. It also does not implement most of the complexity needed to connect to "real" IRC networks. Instead, it is designed to connect via localhost to a bouncer program such as ZNC.

The wrapper program is configured using a single JSON file that is passed as a command line argument. An example config file (live on ##openfpga on Freenode right now) is below:

{
    "cmdline":      ["java", "-jar", "forge-1.7.10-10.13.4.1614-1.7.10-universal.jar", "-Xmx2G", "nogui"],

    "irc_server":   "2600:3c01:e000:1ab::1:1",
    "irc_nick":     "fpgacraft1",
    "irc_port":     6667,
    "irc_channel":  "##openfpga",
    "irc_password": "test",

    "enable_irc_bridge":    true,
    "use_tellraw":          true,

    "backup_interval":  3600,
    "num_backups":      5,

    "enable_sig_verify":    true,

    "users": {
        "rqou":     "NyXS8NwA7NkpgHPlFhsh2X1lTXrdO/VQwx7a8HYf81U",
        "rqou_":    "NyXS8NwA7NkpgHPlFhsh2X1lTXrdO/VQwx7a8HYf81U"
    }
}

Most of the fields should be reasonably self-explanatory. The only part of note is how the IRC server authentication works. This authentication is done unencrypted using the PASS command. This is intended to connect only to a local server over an internal loopback/bridge interface rather than the global Internet. The server is a ZNC instance configured without any of the normal logging or message replaying functionality but configured with the "reconnect" and "keep nickname" functionality.

Lastly, commands to the bridge are supplied via special messages in the IRC channel. They are authenticated by checking both the IRC nickname and an Ed25519 signature. Read the source code for details.

Yet Another Let's Encrypt ACME Client

2016-10-26T02:45:00-07:00

A while ago, I wrote (but forgot to document) Yet Another ACME client in Python. You can see the code for this client here.

ACME is a protocol invented by the Let's Encrypt project for automatically requesting TLS certificates from a certificate authority. Other website describe both the protocol and the Let's Encrypt project in much more detail, but in essence they are an attempt to standardize the ad-hoc natural language instructions that most current certificate authorities use.

Let's Encrypt has an unusual feature in that its certificates have a short 90-day validity period and are designed to be automatically issued and renewed. To help with this automation, a huge number of ACME clients have been written by various people. However, various small issues made me unhappy with all of the clients I looked at, so I programmed my own.

This client is designed to only work with the way my particular web server is set up. The reason I felt I needed yet another client was because existing clients all seemed to be a little awkward to use with my particular setup involving many many virtual hosts that do not actually serve files but only redirects (and thus don't have a "web root" in the normal sense).

On my server, this script is run from a per-user crontab belonging to a special user acme-cert dedicated for running this script. The crontab is set up to invoke the following small shell script once a month (giving two whole months to notice that the script isn't working):

#!/bin/bash

set -eo pipefail
. /storage/certrenewal/venv/bin/activate
set -u

python /storage/certrenewal/le-rqou.py
sudo /bin/systemctl reload nginx.service
sudo /usr/sbin/postfix reload

This script loads a Python virtualenv containing the dependencies for this script, runs the script itself, and then reloads servers that use the certificate. Currently, this is only nginx and Postfix. This is allowed because the sudoers file contains the following lines:

acme-cert ALL=(ALL) NOPASSWD: /bin/systemctl reload nginx.service
acme-cert ALL=(ALL) NOPASSWD: /usr/sbin/postfix reload

If you examine the source code for this script, you will notice that it has the following constants at the top:

ACCOUNT_KEY_PATH - Account key JSON file. This file is created by the official Let's Encrypt client certbot. Only RSA keys are supported in this client. This client does not have the ability to generate this file.
CSR_PATH - CSR file. This file should be in PEM format and should contain a list of domains to issue for in the CN/subjectAltName fields.
REGISTRATION_EMAIL - The email address to associate with the account.
ACME_CHALLENGE_DIR - Common directory used for writing challenge files. This client only supports the http-01 challenge, and it requires all domains to server the .well-known/acme-challenge path from this directory. More on this later.
CERT_PATH_TMPL - A template to create a filename to save newly-issued certificates. The argument {} will be filled in with a date stamp. Old certificates are not automatically deleted in order to allow for easy rollback. There is currently not a mechanism to automatically prune old certificate files.
CERT_PATH_SYMLINK - A path to a symlink that will be updated to point to the latest issued certificate.
CHAIN_PATH - A path to a file where the certificate chain (issuers) will be saved.

As mentioned above, my server is set up to expect all virtual hosts to serve the .well-known/acme-challenge path from the same directory. This is achieved by a fragment similar to the following in the nginx configuration:

server {
    listen 80;
    listen [::]:80;

    server_name robertou.com  www.robertou.com
                robertou.net  www.robertou.net
                robertou.org  www.robertou.org
                robertou.mobi www.robertou.mobi
                robertou.me   www.robertou.me
                robertou.info www.robertou.info
                robertou.us   www.robertou.us;

    location = / {
        return 301 https://robertou.com;
    }

    location / {
        return 404;
    }

    location /.well-known/acme-challenge/ {
        alias /var/www/acme-challenge/;
    }
}

This fragment redirects the HTTP root of all domain variants to the HTTPS root of the "canonical" variant, refuses all paths other than the root (because there should never be a case where these paths get used as the site is HTTPS-only), and serves the .well-known/acme-challenge path from a fixed location in the filesystem for all domains.

The failure of "one post a day"

2016-10-26T02:00:00-07:00

As is obvious from the lack of posts, I have more-or-less given up on the idea of writing one blog post per day. Often, I don't accomplish enough work to write a detailed, thorough blog post. Instead of cluttering this blog with short snippets, I have decided to move progress updates to Twitter and keep this blog for detailed write-ups.

Catch-up post for the last week

2016-09-06T19:30:00-07:00

Unfortunately, I have failed to write a blog post on any day for the entirety of last week. This is a catch-up post to show off what I have not done. I spent the 4th and 5th out with friends. Before that, I:

did a number of uninteresting sysadmin-type tasks not worthy of detailed write-ups (e.g. deleting old files, updating software, etc.)
read the Rust book
distracted myself with ##openfpga
did a bunch of thinking about various projects in the pipeline (but didn't invest "actual effort" into any of them)

So far for today, I have:

finally replaced the filter on my camera lens that I broke weeks ago
performed miscellaneous errands
did an insufficient amount of cleaning of my room

Unfortunately, I have exactly fallen into the "waffle about on different projects without making substantial progress" trap that I am trying to avoid.

Jenkins proxied behind nginx with TLS client certificates

2016-08-30T23:30:00-07:00

I just finished reinstalling Jenkins on my server. The reason for reinstalling Jenkins is to move it inside an LXC container. Hopefully, this will result in a reduced attack surface as well as making it easier to keep track of the global state needed to build my projects.

I use TLS client certificates on all the services I host that will only be used by me or a very-limited group of technically-savvy people. I prefer client certificates because it removes the need to use passwords, session cookies, or other methods to track authentication state. Once the initial setup is done, client certificates are almost fully transparent to the user. Unfortunately, making client certificate authentication work with Jenkins behind a proxy is tricky and requires some hacks.

Jenkins has a plugin that authenticates users via client certificates. However, because nginx terminates TLS and sends plaintext requests to Jenkins, Jenkins obviously can't know about client certificates directly and must be passed this information some other way. From its source code, this plugin appears to get information about client certificates by querying an attribute javax.servlet.request.X509Certificate that is apparently somehow set by Jenkins' surrounding servlet container. Unfortunately, some extended Googling seemed to suggest that the standard way to forward this information is to use a protocol called AJP, and recent versions of Jetty, the servlet wrapper for Jenkins, no longer supports AJP. Supposedly, AJP support can be reobtained by making Jenkins use Apache Tomcat instead. However, because I am not really familiar with how servlets are supposed to work and because there are various claims on the Internet that AJP is slower than HTTP, I decided to look for an alternative solution instead.

The alternative that I decided to use instead is the Reverse Proxy Auth Plugin . This plugin is intended to be used with a reverse proxy that performs some form of HTTP authentication and then forwards a username to Jenkins via an extra X-Forwarded-User header. However, this plugin doesn't actually care how the reverse proxy determines a username as long as the proxy does send one, so we can use this plugin as long as we can get nginx to extract a username from the client certificate.

The certificates I use for my services have a subject like /OU=<computer>/CN=<username>/emailAddress=<email>. We therefore need to extract the CN component from the subject. Fortunately, we can use this snippet from here:

map $ssl_client_s_dn $ssl_client_s_dn_cn {
    default "";
    ~/CN=(?<CN>[^/]+) $CN;
}

This snippet needs to go outside a server block. Finally, to actually proxy to Jenkins, add this inside a server block:

ssl_client_certificate <ca for client certs>;
ssl_verify_client optional;

# Jenkins proxy
location /jenkins {
    if ($ssl_client_verify = FAILED) {
        # BAD cert (rather than no cert at all)
        return 403;
    }
    proxy_set_header X-Forwarded-User $ssl_client_s_dn_cn;

    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Host $http_host;
    proxy_redirect http:// https://;
    proxy_pass http://<where Jenkins is>:8080;
}

(The if block doesn't seem to be strictly necessary as nginx seems to generate a 400 error for client certificates that don't verify, but I am keeping it just in case.)

Finally, set up Jenkins, change its location under "Configure System", ensure that it doesn't complain about the proxy setup, and change the "Security Realm" to "HTTP Header by reverse proxy" under "Configure Global Security". If it doesn't seem to be working, you can debug by navigating to the /jenkins/whoAmI/ URL.

Configuring uWSGI Emperor, nginx, and git-based deploys

2016-08-30T01:00:00-07:00

After having uWSGI installed on my server but not configured for years, I have finally decided to go set it up properly. This setup was done on Ubuntu 16.04 and is mostly based off of these two articles along with a large collection of various StackOverflow and ServerFault questions that I unfortunately didn't keep track of. These notes assume that uWSGI will be used to deploy apps written in Python. You will need to adjust some of the steps if you are deploying other languages.

In general, this setup was as painless as I would reasonably expect. Most of the difficultly resolved around my somewhat-unusual requirements — namely that I wanted to have more than one WSGI application per virtual host, and I wanted the ability to add/remove WSGI applications via the git push mechanism I documented earlier. I can accept that it does seem like I am forcing uWSGI to conform to my own deployment method rather than going for a more "normal" site organization strategy. However, I believe that the setup I have here is more suitable for the organization of my websites (a huge dumping ground of small components rather than one large app), and the fact that it mostly Just Worked™ is a testament to uWSGI's flexibility.

The first step in the setup process is to install uWSGI. I opted to install the version from the Ubuntu repository rather than to install the latest stable via pip or from source. I wanted the convenience of being able to add support for a new language later by simply using apt-get install, and I didn't feel I was likely to require the latest and greatest features. To install from the repositories, install the uwsgi, uwsgi-emperor, and uwsgi-plugin-python packages. If you do not already have them, also install python-pip and python-virtualenv. (This installs the Python 2 version of the packages. Replace instances of python with python3 to get the corresponding Python 3 versions.)

Next, I chose to replace the Ubuntu-provided init scripts for uWSGI with a systemd unit file. There was no real reason for this other than because the two tutorials these notes are based on decided to use systemd, and because systemd is supposed to be The Future™.

(I will now insert a mini-rant here and say that Ubuntu's "halfway adopt systemd but keep around a bunch of older init stuff" is probably the worst of the possible choices of init systems. Some of the issues this has caused me include:

losing the getty on ttyS0/ttyHVC0 (it needs to be explicitly reenabled in systemctl)
a number of warnings mentioning Upstart jobs and runlevels whenever a large dist-upgrade is performed (I have just been ignoring these for now because nothing detrimental seems to happen as a result.)

All of these issues surfaced after an upgrade from Ubuntu 14.04 to 16.04, and IMHO a better transition would have been appreciated.)

Anyways, place the following contents into /etc/systemd/system/uwsgi-emperor-systemd.service:

[Unit]
Description=uWSGI Emperor
After=syslog.target

[Service]
ExecStartPre=/bin/bash -c 'mkdir -p /run/uwsgi; chown www-data:www-data /run/uwsgi'
ExecStart=/usr/bin/uwsgi --ini /etc/uwsgi-emperor/emperor.ini
Restart=always
KillSignal=SIGQUIT
Type=notify
StandardError=syslog
NotifyAccess=all

[Install]
WantedBy=multi-user.target

Now run the following commands to disable the Ubuntu init scripts and activate this new one:

sudo systemctl stop uwsgi.service
sudo systemctl stop uwsgi-emperor.service
sudo systemctl disable uwsgi.service
sudo systemctl disable uwsgi-emperor.service
sudo systemctl enable uwsgi-emperor-systemd.service
sudo systemctl start uwsgi-emperor-systemd.service

Note that unlike the Ubuntu-provided init scripts, this unit file will cause uWSGI to log to journalctl rather than its own log file.

The next and final "global system" configuration step is to create symlinks in /etc/uwsgi-emperor/vassals/ that point to /var/www/<site>/conf/uwsgi-master.ini (or some other filename of your choice).

$ ls -l /etc/uwsgi-emperor/vassals/
lrwxrwxrwx 1 root root  43 Aug 30 05:25 robertou.com.ini -> /var/www/robertou.com/conf/uwsgi-master.ini
...

In order to potentially contain multiple apps in one virtual host without needing to create additional symlinks, the file conf/uwsgi-master.ini in each site's git repository contains only the following two lines:

[uwsgi]
emperor = /var/www/<site>/conf/apps

(Note that the %d magic variable will not work here because uWSGI doesn't follow the symlink before computing %d .)

Although I didn't find any explicit documentation that this would do what I expected, uWSGI is indeed smart and understands the intention of creating a sub-Emperor as can be observed in the logs:

Aug 30 05:04:21 <snip> uwsgi[4882]: *** starting uWSGI sub-Emperor ***

An individual configuration file for each app can now be placed in the conf/apps/ directory in each virtual host's repository. An example of one is below:

[uwsgi]
socket = /run/uwsgi/rqou.com-testapp1.sock
chmod-socket = 660
# XXX do we need the build thing?
chdir = /var/www/rqou.com/build/testapp1
master = true
virtualenv = /var/www/rqou.com/venv
module = testapp1:app
processes = 1
threads = 1
plugins = python

Finally, insert an appropriate proxying command into conf/nginx.conf:

    location = /testapp1 { rewrite ^ /testapp1/; }
    location /testapp1 {
        include /etc/nginx/uwsgi_params;
        uwsgi_pass unix:/run/uwsgi/rqou.com-testapp1.sock;
    }

When these configuration changes are pushed to the webdeploy user, uWSGI seems to automatically detect the change and gracefully reloads. Hopefully the app now works!

Deploying my website using git

2016-08-29T14:00:00-07:00

All of the websites on my server are currently deployed using git push. This ensures that all of the changes are easily traceable and easily reverted if necessary. This deploy mechanism is mostly powered by Jeff Lindsay (progrium)'s gitreceive script. Here are some notes on how to create a similar setup. These notes assume you already have a Linux server that is running nginx with virtual hosts (although the steps can be adapted to other setups as well).

The first step in this setup process is to set up a user to receive git commits via git push and the gitreceive script. Instead of duplicating the instructions, I will simply refer to the README for the gitreceive script instead. The important steps to perform are the "Set up a git user on the server" and "Create a user by uploading a public key from your laptop" steps. Although you might want to test the setup using the default receiver script in the README, you will replace the receiver script with a completely different one in the next step. I will note that the username I use for gitreceive is not git but webdeploy, and you must ensure that the GITUSER environment variable is set appropriately if you are also using a username other than git for your server. You will obviously also have to substitute webdeploy with the appropriate username in the steps below if you are not using the same username in your setup.

The next step is to copy an appropriate receiver script into the home directory of the webdeploy user. The receiver script I use can be found here on my GitHub. However, this script makes many assumptions about how files are laid out in each site's git repository and on the server.

The git repositories for each site on my server are laid out somewhat like this:

<root>
├── build.sh
├── build/
├── conf/
│   └── nginx.conf
├── content/
│   └── ...
└── ...

Notably, the nginx.conf fragment for each virtual host is located within the source itself. This helps ensure that any possible reverse proxying, settings for FastCGI/uWSGI/etc., and other similar configuration are all versioned along with the actual source code. There is also a build.sh script that does any necessary processing of the source code (e.g. minifying). The build/ directory becomes the actual webroot for each site. (Unfortunately this does mean that extra copying/hardlinking is required for static files that do not need to undergo any processing during the build process.) The rest of the repository (e.g. content/) is not served despite eventually also ending up under /var/www (this is controlled by a root directive in each site's nginx.conf fragment).

Because changing the nginx.conf file requires a reload of nginx which normally requires root, the next step is to modify the sudoers file to allow webdeploy to do this. To do so, run sudo visudo and add a line like the following:

webdeploy ALL=(ALL) NOPASSWD: /bin/systemctl reload nginx.service

Unfortunately, the sudoers file is rather confusing. The line here is allowing the webdeploy user to run only the /bin/systemctl binary and only with the arguments reload nginx.service. The webdeploy user can run this command without a password and can run it as any user (i.e. root). However, webdeploy won't be allowed to perform any other actions such as disable or to control any other service.

In order to allow the receiver script to actually create new files and directories under /var/www, we need to change some permissions. One way that this can be done is to give webdeploy ownership of all of /var/www. However, I opted to use extended ACLs to grant permissions instead. Although it makes almost no difference in this case and traditional groups can be used rather than ACLs in many cases, extended ACLs in general help to reduce group proliferation when granting fine-grained access to different parts of the filesystem to different users. If you choose to use ACLs, you can grant the webdeploy user full permissions on /var/www using

sudo setfacl -m "u:webdeploy:rwx" /var/www

My receiver script uses a number of symlinks in order to make site updates less obviously non-atomic. The next step is to set these up. For each domain, there are two symlinks <site> and <site>-prev under /var/www. For example, part of /var/www might look like this:

$ ls -l /var/www
lrwxrwxrwx  1 webdeploy webdeploy   62 Aug 29 09:33 robertou.com -> /var/www/robertou.com-3a8d6216872d83f2d68c82fcb09b6904796bd70a
drwxrwxr-x  1 webdeploy webdeploy  270 Aug 29 09:33 robertou.com-3a8d6216872d83f2d68c82fcb09b6904796bd70a
drwxrwxr-x  1 webdeploy webdeploy  276 Aug 29 09:22 robertou.com-6c19354ac248201a032fab3524602550051f3310
lrwxrwxrwx  1 webdeploy webdeploy   62 Aug 29 09:33 robertou.com-prev -> /var/www/robertou.com-6c19354ac248201a032fab3524602550051f3310
...

Initially, the symlinks can both be set to point to dummy targets. The purpose of the -prev symlink is to facilitate emergency reverts without needing to rerun the build process for the previous version of the site.

Finally, the nginx.conf fragments are included in the main /etc/nginx.conf by (assuming a distribution default configuration that uses sites-enabled and sites-available) symlinking /var/www/<site>/conf/nginx.conf into /etc/nginx/sites-available and then symlinking that symlink into /etc/nginx/sites-enabled like so:

$ ls -l /etc/nginx/sites-enabled/robertou.com 
lrwxrwxrwx 1 root root 31 Mar  2 12:04 /etc/nginx/sites-enabled/robertou.com -> ../sites-available/robertou.com
$ ls -l /etc/nginx/sites-available/robertou.com 
lrwxrwxrwx 1 root root 37 Mar  2 12:03 /etc/nginx/sites-available/robertou.com -> /var/www/robertou.com/conf/nginx.conf

Once this is all set up, doing a push to the webdeploy user should automatically:

Receive and unpack the new version of the site.
Run its build.sh script
Move the "current site" symlink to point to the new version.
Move the "old site" symlink to point to the old version.
Delete the "old old" version.
Reload nginx.

My setup here has a number of known bugs:

A build failure will leave stray files in /var/www. This problem was ignored for my use case because I assumed that I would always run the build locally before pushing and because I can simply use sudo to remove the stray files if a build does unexpectedly fail.
Submodules are not handled. The gitreceive repository has a hint on how to fix this.
It is not resistant against malicious users. Because the webdeploy user runs build.sh under its privileges, a malicious user can grief any site on the same server. This script as such is therefore not suitable for a shared hosting environment. gitreceive itself also apparently has a bug that allows creating repositories anywhere on the system (subject to normal write permission checks). Only give trusted users access to the webdeploy user.

Current purpose of this blog thing

2016-08-29T02:00:00-07:00

I currently feel that I am not progressing on my personal projects to my satisfaction. This blog is my attempt to force myself to rectify that by making my (lack of) progress very explicitly visible. The current plan is to write a daily post documenting the work I accomplish each day. The purpose of this is not solely to publicly shame myself. One important "other" goal of this exercise is to get myself to actually publicly document useful things I work on. I also hope that this daily reflection exercise can help reduce my tendency to waffle about and can help me clarify the direction of my projects.

I will conclude the post for today by noting that my website has finally been updated. There isn't too much notable about the site itself (it's a simple Pelican static site), but tomorrow I will probably document the site deploy system that has been set up.

Hello world

2016-08-29T00:00:00-07:00

Hello world. Apparently Pelican is a thing that makes websites!