Nathan Friend

I built a weird keyboard

2023-06-26T20:00:00-04:00

I spent most of my free time over the last 10 months building this bizarre keyboard from scratch. It’s a Dactyl Manuform - a split keyboard with a highly sculpted design that is somehow simulataneously the ugliest and most eye-catching object I’ve ever seen.

The goal of this keyboard design is to place keys exactly along each finger’s natural axis of motion. The consequences of this approach are downward-sloping (A.K.A “tented”) rows to match the natural angle of the wrists, a deeper middle finger column to compensate for this finger’s relative length, an offset pinky column to minimize stretching, and a thumb cluster with multiple keys to take advantage of its opposable nature.

There’s a reason most keyboards don’t look like this. It’s difficult to mass-produce curved keyboards since they can’t use the stiff, flat PCBs that most keyboards use to wire the keys to the microcontroller. Every Dactyl Manuform is a unique piece of art, painstakingly hand-wired by a human being. Here’s how I built mine:

Build log

The first step was to design and 3D print the case. There are a number of Dactyl Manuform model generators out there; I ended up using this one since it had a few extra features I wanted (e.g. wide pinky keys). Some things I was looking for in my design:

A key layout similar to the ErgoDox EZ keyboards I already own
An aggressive tent angle (I went with π/8 = 22.5°)
Hot-swap sockets

I gave up on the hot-swap sockets after failing to coax the generator script to produce valid hot-swap socket holders. I’m glad I did - in the end I don’t think they would have worked anyway.

Once I had a model that looked good on the screen, I printed a draft version to see it in real life.

Printed with a .8mm nozzle, .32mm layer height, and lightning infill

Overall, I was really happy with this first draft. I only made a few tweaks to the model before printing the real thing:

I enabled the “external microcontroller” option, which let me kick the can on deciding which microcontroller and connection types to use
I added my own screw holes; the holes generated by the script were awkwardly placed

To mount the base plates to the body of the keyboard, I used these heat set inserts. It’s a neat system; the threaded inserts are melted into the 3D print using a soldering gun, producing threaded screw holes much stronger and smaller than anything that would be possible using only 3D printing.

To determine the ideal hole size for these inserts, I made a test print.

A test print with insert holes ranging from 3.8mm to 4.2mm in diameter

I’m glad I tested this; all of the hole options were too small! A second print with bigger holes was more successful.

A second test print with insert holes ranging from 4.2mm to 4.6mm in diameter. I decided to go with 4.5mm

With this last detail resolved, I began printing the real halves. I used my favorite material - wood PLA - which looks (and even smells!) quite similar to real wood, once properly post-processed.

The final STL file being sliced in Cura. 24 hours per half!

The left half, fresh off the printer

Both halves printed and slightly cleaned up

I melted the inserts into the holes, which was nerve-wracking; one bad insert would have likely ruined the whole print. It was awkward to try and hold the keyboard and the insert in place while pressing the insert into the plastic with the soldering gun. Somehow I managed to install all ten without issue!

An installed heat-set insert

I ordered the transparent acrylic base plates from ponoko.com and was quite happy with the result.

Acrylic base plate from ponoko.com

Next up was post-processing. I sanded each half with 80-grit sandpaper, which was a ton of work - there are a lot of nooks and crannies that make this a tedious job.

Both halves, sanded

In order to sand the inside of the key holes, I printed a little attachment that I could wrap with sandpaper and fit on my screwdriver.

My custom SuperSander™ (patent pending)

Normally this would be an unnecessary step - no one sees the inside of the key holes - but the fit was a bit too tight; most of the holes required some sanding before the switch would fit properly. (This is why I’m glad I didn’t bother with hot-swap sockets - the fit is so tight, I’d never be able to get the switches out anyway.)

Another shot, because sanding these took too much time not to show off

The next step was to stain and clear coat the prints. I used this gel stain and glossy polyurethane to give the prints a rich, polished wood color.

The finished product

Some nice faux woodgrain

This was a time consuming step, as each half required three coats of stain (minimum 24 hours to dry per coat) and at least 3 coats of polyurethane (a few hours to dry per coat). I did this in the dead of winter which made drying these in the outdoors challenging (a space heater may have been involved).

The hard work paid off, though - I’m really happy with how these look! The layer lines even give the illusion of a wood grain.

The next decision was which key switches to use. I’m a clicky switch guy; the noisier the better. I bought a Kailh switch tester so I could make an informed decision and decided on Kailh Box Whites.

Regular and low-profile Kailh switch testers. Not pictured: Gateron and Boba testers

Compared to other clicky switches (e.g. Cherry MX Blues), Box Whites are extra clicky (they click twice per key press). Perfect for working remotely! If I ever make a silent/office-friendly version of this board, I’ll go with Boba U4 Silents.

At this point, I was able to set the switches and keycaps in place and get a feel for what it would be like to type on this monstrosity. I’ll admit it felt about as weird as it looks.

Switches and keycaps (temporarily) installed

There was one last detail to work out before I could begin wiring up the halves. I wanted to install a rotary encoder (“volume knob”) on each half, but these don’t click into a standard keyboard hole out of the box. I had to 3D print a special adapter for each.

The end result, with chunks of glass attached to the bottoms

I printed these with PETG and learned the hard way to always use glue stick when printing with PETG. The adapters adhesed to the bed so strongly they took chunks out of my glass bed when I finally pried them off. I was able to salvage the adapters with some sanding, but the printer bed was unfortunately beyond repair.

😭

Despite their rather violent effects on my printer, the adapters did their job quite nicely!

Rotary encoder with and without the cap

Finally, it was time to start wiring it up! First, I wrapped diodes around one pin on each switch.

Diodes wrapped

I soldered the diodes into place and snipped the extra leg.

Diodes soldered and snipped

I soldered the remaining diode legs together to form the rows of the key matrix.

Diode legs soldered together into rows

I used small, individual pieces of insulated wire to form the columns.

Don't look too closely; I'm really bad at soldering

I installed DuPont connectors so I didn’t have to solder directly to the microcontroller. This saved me a lot of headache since it took quite a bit of trial and error to get all pins in the right spot.

DuPont connectors installed

I flashed a basic QMK firmware to the microcontroller and had the incredibly satisfying experience of seeing a letter appear on the screen when I pressed a key. I also got the LED strip working!

IT LIVES!!

I was getting really close at this point. I designed a custom holder for the microcontroller since the one that was supposed to be compatible with my case didn’t fit for some reason.

Custom-designed microcontroller holder

I spliced some wires together since a few of the microcontroller pins had to be shared by more than one connection.

I'm frankly shocked this thing works

After assembling all the pieces, a bit of software configuration, and a lot of trial and error… I had a working keyboard!

Finally. Done.

This was way too much work.

Never again. Probably.

ALERT

How does it feel?

Weird, but good! I’ve only been typing on it for a few work days, so my muscle memory hasn’t fully adjusted. I keep reaching for keys in the wrong places; in particular, my fingers naturally stretch too far when reaching for the bottom row. I also made a few modifications to my QMK layout to take advantage of the more accessible thumb clusters compared to my ErgoDox EZ. I think I’ll really like it once I’m used to it.

Was it worth it?

Umm… I think so? The end result was fantastic, but it was an insane amount of work. I don’t recommend this project to anyone who isn’t interested in the process itself. If you’re just looking for a great ergonomic keyboard, I’d recommend buying an ErgoDox EZ, a Moonlander, a Kinesis Advantage360 or a prebuilt Dactyl Manuform, all of which will cost about the same as this project (see below).

Cost breakdown

Cost of all items, including tax and shipping.

Description	Cost (CAD)	Link
Wood PLA filament for 3D printed case	$40.44	amazon.ca
Kailh switch tester	$21.46	aliexpress.com
Kailh low-profile switch tester	$9.68	aliexpress.com
Pro Micro controller (x2)	$44.98	amazon.ca
Kailh BOX White switches (x90)	$51.26	aliexpress.com
M3 heat-seated inserts (x100)	$12.42	amazon.ca
M3 screws (x100)	$12.02	amazon.ca
EC11 rotary encoder (x4)	$15.80	amazon.ca
LED strip (1m)	$16.37	aliexpress.com
1N4148 Diode (x100)	$8.80	digikey.ca
Reset button (x3)	$10.82	digikey.ca
22AWG Wire (25’)	$7.24	digikey.ca
TRRS jack, female (x3)	$12.65	digikey.ca
Jumper wire (x60)	$11.74	digikey.ca
Soldering iron	$59.87	homedepot.ca
Solder	$28.23	homedepot.ca
Wire stripper	$11.29	canadiantire.ca
Acrylic base plate (x4)	$54.88	ponoko.com
Gel wood stain	$19.93	homedepot.ca
Glossy polyurethane	$27.11	amazon.ca
Keycap set (x2)	$57.32	amazon.ca
Electrical tape	$5.37	amazon.ca
Rubber feet	$13.55	amazon.ca
Only keyboard materials	$450.93	≈ $340 USD
All items (including tools, testers, etc.)	$553.23	≈ $417 USD

As you can see, building your own keyboard is not a good way to save money 💸

Links/resources

All .stl and .svg files: https://www.thingiverse.com/thing:6099418
A helpful Hacker News comment: https://news.ycombinator.com/item?id=23445208
The generator I used to generate the keyboard case model: https://github.com/ibnuda/dactyl-keyboard/tree/refaktor
The firmware that powers the keyboard: https://qmk.fm/
Some helpul build logs:

Wordle Bot

2022-01-25T05:25:10-05:00

Like everyone else, I’ve been obsessed with Wordle lately.

I was really bad at it when I began. My original strategy was to use the first 3 or 4 guesses to uncover as many letters as possible, and then use the last few guesses to unscramble the solution. This turned out to be a terrible approach, and I failed most of the challenges.

This got me thinking: what is the optimal Wordle strategy?

And that immediately led to a second thought: can I automate it?

The answer is a resounding of course! I present to you: Wordle Bot!

How it works

The general strategy is to repeatedly pick the word that is most likely to reveal new letters. To do this, I use the frequency of each letter in the set of possible solutions to assign a score to each potential guess, and choose the option with the highest score.

The set of all possible solutions is conveniently hardcoded in Wordle’s source code, so my first step is to simply hardcode this list into my own script.

Next, I loop through each potential solution and count how many times each letter occurs. This results in a mapping like this:

{ "e": 1233, "a": 979, "r": 899, "o": 754, "t": 729, "l": 719, "i": 671, "s": 669, "n": 575, "c": 477, "u": 467, "y": 425, "d": 393, "h": 389, "p": 367, "m": 316, "g": 311, "b": 281, "f": 230, "k": 210, "w": 195, "v": 153, "z": 40, "x": 37, "q": 29, "j": 27 } 

So, for example, a word like “HELLO” would score 3,814 (389 + 1,233 + 719 + 719 + 754).

Using this method, the bot would always start with the word “EERIE”, due to its high score of 5,269 (1,233 + 1,233 + 899 + 671 + 1,233).

However, this actually isn’t a very useful guess! It only gives us information about 3 letters. A more strategic method is to pick the highest scoring word that contains no repeating letters. Using this approach, the best first play is “LATER”, which still scores quite high (4,559 = 719 + 979 + 729 + 1,233 + 899) and provides information about 5 different letters.

(Side note: Even though I know I should start with “LATER” when playing on my own, I can’t help but start with “ADIEU” - too many vowels to pass up!)

Each subsequent guess is generated using the same algorithm, except that the list of possible solutions is shrunk by eliminating all options that contradict the results of previous guesses. A side effect of this elimination method is that it effectively plays the game on Hard Mode. (Hard Mode = “Any revealed hints must be used in subsequent guesses”).

In later guesses, it’s possible there won’t be a valid option that includes no repeating letters, so in this case the bot falls back to the highest-scoring word regardless of letter repetition.

Demo

Here’s an example of the script working in bookmarklet form:

(Instruction for setting up this script as a bookmarklet can be found here.)

How good is it?

The short answer is: quite good! (Much better than me.) But not perfect!

At the time of writing, Wordle Bot has solved 220 puzzles. Its guess distribution looks like this:

80% of the time, Wordle Bot can solve the daily puzzle in 4 tries or less. Not bad!

But why isn’t it perfect? I was intrigued by the two puzzles it failed to solve. In both cases, Wordle Bot failed in a similar fashion. By guess 3, it had nailed down letters 2 - 5. It spent the remaining guesses cycling through different starting letters, and failed to find the correct one before it ran out of guesses.

Here is its attempt at puzzle 113:

Wordle 113

Even though the correct solution (“HATCH”) scores higher than all the options it tried (“PATCH”, “MATCH”, “BATCH”, and “WATCH”), it chose not to guess “HATCH” since it includes a repeated letter (“H”). As mentioned earlier, the algorithm always prefers guesses without repetition if they exist.

A similar thing happened on day 123:

Wordle 123

In this case, the solution “JAUNT” scored lower than all the options it tried (“DAUNT”, “HAUNT”, “GAUNT”, and “VAUNT”), so it simply ran out of time.

Bringing the bot to life

Turning the script into a Twitter bot was just a matter of automation.

First, I built a Puppeteer script to launch Wordle in an instance of headless Chrome, inject my bookmarklet script, and solve the puzzle. Once it’s finished, the script takes a screenshot, uploads it to Imgur, and tweets the results:

Wordle 220 3/6

⬜🟨⬜⬜🟩
🟩⬜⬜🟩🟩
🟩🟩🟩🟩🟩

Full solution [#SPOILER!]: https://t.co/kXojDt5u3i
— Wordle Bot (@bot_wordle) January 25, 2022

Then, I configured a scheduled GitLab pipeline to run the project’s pipeline every day at 3:00 AM ET.

Even though the end result is fairly simple, I ran into a number of speed bumps along the way:

Wordle shares its results by copying them to the clipboard, but I couldn’t figure out a way to access the clipboard’s contents inside Puppeteer (it always returned my system clipboard contents instead). To get around this, I had to build the “share” text myself.
The Imgur JavaScript library I use required Node 14+, but the Puppeteer Docker image I was using inside my GitLab pipeline was stuck on Node 12. Switching to buildkite/puppeteer fixed this.
The pipeline would randomly fail while trying to connect to the Wordle website: Error: net::ERR_NETWORK_CHANGED at https://www.powerlanguage.co.uk/wordle/. I “solved” this by adding a timeout and retry to my GitLab job.

Absurdal

Oh, did I mention the bot is also surprisingly good at solving the Absurdal variant?

Herding Gits

2021-08-26T19:42:44-04:00

Juggling multiple Git identities can be tricky.

For example, at Stripe, we encourage developers to create a separate GitHub account for Stripe-related open source activity. For me, this means I now own both a nfriend and a nfriend-stripe GitHub profile.

While setting up my dev environment, I had a few goals:

Use both Git identities on the same machine
Sign commits with separate GPG keys
Connect to remotes using different SSH keys
Have all of this 👆 happen automatically without me having to think about it

Good news! This is possible with a little .gitconfig magic ✨

The setup

Note: The instructions below rely on Git’s conditional includes, which are only available in Git 2.13 and beyond.

1. Set up separate SSH and GPG keys for each identity

I won’t go into details since this is already covered in great detail by other tutorials. GitHub’s tutorials are particularly well-presented:

2. Create separate directories for each identity

For example, a ~/github-personal and a ~/github-work directory.

3. Create a `.gitconfig_include` file in each

Inside each of these new directories, create a new file named .gitconfig_include with the following content:

[user] name = Your Name email = your-name@example.com signingkey = 0123456789ABCDEF [core] sshCommand = ssh -i ~/.ssh/id_rsa_example -F /dev/null 

Update each file with the name, email, and signing key for the corresponding Git identity.

Additionally, update the command in the sshCommand option to reference the appropriate key file.

4. Reference these files from the global `.gitconfig`

In your global .gitconfig (i.e. ~/.gitconfig), configure Git to conditionally include the correct .gitconfig_include file based on the current directory:

[includeIf "gitdir:~/github-personal/"] path = ~/github-personal/.gitconfig_include [includeIf "gitdir:~/github-work/"] path = ~/github-work/.gitconfig_include 

5. Test it!

Create a test project with both identities. Ensure you can:

Clone the repository from the remote
Make a commit
Push the commit to the remote

If you’re using a web UI like GitLab or GitHub, check to see that your commits are being signed correctly and are labeled as “Verified”:

Helpful links

Some things I found helpful while setting this up:

Setting .gitconfig settings on a per-directory basis: https://stackoverflow.com/a/48088291/1063392
Using Git’s core.sshCommand configuration to select an SSH key: https://superuser.com/a/912281/144803

Feedback

Thoughts? Let me know in this GitLab issue!

It’s finally here! 🎉

2021-05-07T00:01:01-04:00

It’s been almost seven years since Inspiral Web was released.

For most of those seven years, the site included a “Get the app” button that popped open a message that read “Inspiral Web isn’t mobile friendly… yet” and included a link to join the mailing list.

Over 25,000 people joined that mailing list. The sheer number of interested people, combined with just a bit of pandemic cabin fever, spurred me to finally begin work in earnest on the mobile app.

After five months of feverish, bleary-eyed, nights-and-weekends coding, I’m thrilled to announce that Inspiral is now available for iOS and Android!

Give it a try on your platform of choice:

Or, check out the app’s website, which includes links to the app’s social media presence.

The app’s source code is public. If you find issues or have a feature request, please open an issue!

Capturing Alexa Errors with Sentry and GitLab

2020-11-18T09:22:10-05:00

Diagnosing issues with a live Alexa skill can be tricky.

Most users who run into issues will simply uninstall your skill. A few unusually inspired users may even leave helpful reviews like this:

How do you go about figuring out what’s wrong?

By plugging a few open source tools together, you can get great visibility into what’s going wrong.

1. Implement an `ErrorHandler` in your skill code

First, you’ll need a way to globally catch errors in your Alexa skill. The ASK SDK provides an ErrorHandler interface that does just this. Create a new file for your ErrorHandler implementation:

// lambda/src/handlers/ErrorHandler.ts import * as Alexa from 'ask-sdk-core'; export class ErrorHandler implements Alexa.ErrorHandler { canHandle() { // Handle _all_ exceptions return true; } handle(handlerInput: Alexa.HandlerInput, error: Error) { console.log(`~~~~ Error handled: ${error.stack}`); const speech = 'Sorry, something went wrong! Can you please try again?'; return handlerInput.responseBuilder .speak(speech) .reprompt(speech) .getResponse(); } } 

(I’m using TypeScript in this example, but a vanilla JS implementation shouldn’t be much different.)

Next, register this error handler in your skill’s entrypoint:

// lambda/src/index.ts import * as Alexa from 'ask-sdk-core'; import { ErrorHandler } from './handlers/ErrorHandler'; export const handler = Alexa.SkillBuilders.custom() .addRequestHandlers(/* ...your request handlers here... */) .addErrorHandlers(new ErrorHandler()) // ← add this .lambda(); 

This already gets you pretty close! If anything goes wrong, you’ll have a nice stack trace in your CloudWatch logs, and the user will get a polite message informing them something went wrong.

2. Ship error details to Sentry

Sentry is an open source monitoring platform that does a great job of tracking and organizing software errors.

After creating a free account at https://sentry.io, create a new Sentry project with the “Node.js” platform option. Install the dependencies it recommends:

yarn add @sentry/node @sentry/tracing

Next, set up @sentry/node with the tracking info it needs. You can do this at the beginning of every Alexa request by creating a new request interceptor:

// lambda/src/interceptors/SentryInterceptor.ts import * as Alexa from 'ask-sdk-core'; import * as Sentry from '@sentry/node'; export class SentryInterceptor implements Alexa.RequestInterceptor { async process() { Sentry.init({ dsn: '<your DSN here>', tracesSampleRate: 1.0, }); } } 

(Your Sentry DSN will be provided to you when setting up your Sentry project.)

Don’t forget to register this interceptor, similar to how you registered your ErrorHandler above:

// lambda/src/index.ts import * as Alexa from 'ask-sdk-core'; import { ErrorHandler } from './handlers/ErrorHandler'; export const handler = Alexa.SkillBuilders.custom() .addRequestHandlers(/* ...your request handlers here... */) .addRequestInterceptors(new SentryInterceptor()) // ← add this .addErrorHandlers(new ErrorHandler()) .lambda(); 

Finally, in the error handler you created earlier, send the error to Sentry:

// lambda/src/handlers/ErrorHandler.ts import * as Alexa from 'ask-sdk-core'; import * as Sentry from '@sentry/node'; // ← add this export class ErrorHandler implements Alexa.ErrorHandler { canHandle() { // Handle _all_ exceptions return true; } handle(handlerInput: Alexa.HandlerInput, error: Error) { console.log(`~~~~ Error handled: ${error.stack}`); Sentry.captureException(error); // ← also add this const speech = 'Sorry, something went wrong! Can you please try again?'; return handlerInput.responseBuilder .speak(speech) .reprompt(speech) .getResponse(); } } 

That’s it! Now all that’s left is to…

3. Test it!

In your LaunchRequest handler, do something silly like this:

const anObject: any = {}; anObject.aMethodThatDoesntExist(); 

(I’m explicitly specifying any here, otherwise TypeScript won’t let me get away with this!)

Deploy your skill and give it a spin!

❯ ask dialog User > Open <your skill name here> Alexa > Sorry, something went wrong! Can you please try again? 

Jump back into your Sentry project - you should now be the proud owner of a new issue:

4. Integrate Sentry with GitLab (optional)

If you host you skill’s code on GitLab you can take advantage of GitLab’s first-class Sentry integration to see error details directly in your GitLab project:

From your GitLab project, navigate to Settings > Operations
Expand the Error tracking section
Check the Active checkbox
Assuming you are using Sentry’s hosted solution, enter https://sentry.io/ in the Sentry API URL field
Paste your Sentry auth token into the Auth Token field. To generate an auth token in Sentry:
1. Navigate to your Sentry dashboard
2. Click the ▼ next to your name and select API keys
3. Click Create New Token
4. Leave the default scopes as they are and click Create Token
5. Copy the big string of gibberish
Jump back to GitLab and click Connect and select your Sentry project
Click Save changes

That’s it! Navigate to Operations > Error Tracking. You should see the same set of Sentry errors nicely displayed inside your GitLab project.

Source

See this code in action at https://gitlab.com/nfriend/days-until, or give my Days Until skill a try for yourself. Here’s a quick link to the skill!

Ridiculous Refs

2019-10-19T18:27:01-04:00

How many different special characters can you jam into a Git ref name (i.e. the full name of a tag or a branch) before Git will complain?

As if Git isn't complicated enough without testing its limits.

The answer is… a lot.

Here’s a list of ref names that Git considers valid, ranging from pedestrian to ludicrous:

a/b
A/B
a/b/c/d
a/b.c
0/1/2
!/"#%&'{}+,-.<>;=@]_`{|}
🙂🚀😂🇺🇸💩🇨🇦💯👍❤️/🤦
(╯°□°)/╯︵┻━┻
¯|_(ツ)_/¯

The opposite is a little less flashy, but equally useful. All of these are invalid ref names:

a: At least one / is required
a/.b: Slash-separated components can’t begin with a .
a/b.: No ending with a .
a/.lock: No ending with .lock
a/b..c: Consecutive dots (..) are not allowed
a/~^:?*[\: None of the characters after a/ are allowed
a/b/: No ending with a slash
/a/b: No beginning with a slash
a/b//c: No consecutive slashes
a/@{: The @{ sequence is not allowed

There’s some nuance to this second list - some of these rules can be relaxed in special situations. For the complete specification, check out the documentation for the check-ref-format command.

Wait, I’ve created lots of branches and tags that don’t contain a `/`!

Right! When you create a branch named my-feature, Git actually creates a ref named refs/heads/my-feature.

This is a bit of a tangent, but what Git is actually doing under the hood is creating a new file named my-feature inside your repo’s .git directory at .git/refs/heads/my-feature. You can see this for yourself by opening up .git/refs/heads in a file explorer. Understanding this makes it more obvious why the / character is required.

Why should I care about this?

If all you do with Git is pull, commit, push, and maybe the occasional rebase, you can ignore these edge cases. Just keep using nice, simple, boring names like my-feature-branch or v1.2.

However, if you’re building a tool that interacts with Git refs, you might want to throw a few of the crazier names listed above into your test cases.

In fact, this was my motivation for compiling these lists. I built a new GitLab Issue search feature that involved Git tag names, and I needed to make sure it handled any tag name the user threw at it. I was hoping to find something like the Big List of Naughty Strings for Git refs, but I couldn’t find anything beyond Git’s technical documentation.

Your list is wrong!

I wouldn’t be surprised! Please let me know by opening an issue!

References/Attributions

The “exploding head” icon was created by Anniken & Andreas from the Noun Project.

PDF Gotchas with Headless Chrome

2019-04-15T05:20:55-04:00

Generating PDF reports is one of those features that every enterprise developer will implement at some point in their career. I had my turn on a project with my previous employer. After exploring the available options, I settled on using Chrome’s headless mode to render HTML and save the result as a PDF.

"Headless Chrome" just sounds so.... gruesome.

This approach seems kind of weird and a bit overkill at first, but it has a number of pretty huge advantages:

You can build your PDFs using the most popular layout system in the world (HTML/CSS)
Anything that can be rendered in a webpage can be used in a PDF, including:
- images
- custom fonts
- links
- 3^rd party JavaScript libraries (think visualization libraries like D3.js or graphing libraries like Google Charts)
- etc…
If you’re building a web application, you can reuse components from your application in your PDF
Printing to a PDF is a supported use case of Chrome’s headless mode
Google’s own Puppeteer library gives you full control over the headless instance of Chrome
You can develop your PDF layout in Chrome - with full access to Chrome’s dev tools - instead of continually regenerating the PDF to see changes

It’s not all unicorns and rainbows, though. Below are a few of the gotchas I discovered while building a real PDF using headless Chrome.

Headers and footers can’t use external resources

This is the big one. If you try and place an <img> tag in your header or footer (a pretty common use case for a header or footer):

<img src="/assets/logo.jpg" /> 

…your image won’t show up. This is because Chrome won’t make any requests for external resources that appear in the header or footer templates.

One workaround is to encode the image into the template as a base64’d string:

<img src="data:image/png;base64, iVBORw0KGg..." /> 

I’ve found this site handy for converting an image into an <img>-compatible base64 string.

Headers and footers don’t inherit styles from the rest of the page

Headers and footers are specified at PDF render time by passing HTML strings to the page.pdf() method:

page.pdf({ headerTemplate: '<h1>This is the header!</h1>', footerTemplate: '<h1>This is the footer!</h1>' }); 

These templates are rendered in a separate context than the content of the webpage. Because of this, the CSS styles that apply to the content won’t apply to the header and the footer. Any styles that apply to the content of your PDF that you would like to also apply to your header and footer must be repeated in each of your header and footer templates. And unfortunately, you can’t just reference a common stylesheet using a <link> element - see point #1 above.

Headers and footers require explicit margins to be visible

This one took me a while to figure out. Chrome won’t automatically resize your content to make space for the header and footer templates. You’ll need to make space for your header and footer by specifying a fixed margin at the top and bottom of your page:

page.pdf({ headerTemplate: '<h1>This is the header!</h1>', footerTemplate: '<h1>This is the footer!</h1>', margin: { top: '100px', bottom: '50px' } }); 

Without these margins, the content will be rendered on top of your header and footer, leaving you wondering why your header and footer templates aren’t showing up.

Page breaks can be a pain

CSS provides some rules that determine where a page break should be placed when printing, for example:

@media print { .page { page-break-after: always; } } 

These rules work - but they can be finicky. You may run into problem when trying to page break inside of^[1]:

tables
floating elements
inline-block elements
block elements with borders

I also had issues using page-break-after inside of a flexbox layout.

Some advanced layouts simply aren’t possible

There are a few edge cases - mostly dealing with headers/footers and page wrapping - that you simply can’t control. For example, want to place a special footer only on pages 2, 4, and 7? Not possible. (If it is, let me know how!)

The page needs to finish loading

If the page being screenshotted requires time to load, (for example, if the page has JavaScript that makes an AJAX request for some data), you’ll need to wait for this initialization to complete before triggering the screenshot. If you simply screenshot the page right after the initial load, your PDF will be filled with loading bars and missing data.

I worked around this by setting a global flag in the webpage once all initialization work is finished:

// in the web page async init() { const data = await this.dataService.getData(); const user = await this.userService.getUserProfile(); // ...etc... window.isReadyForPDF = true; } 

Then, using Puppeteer’s page.waitForFunction() method, we can wait for this global variable to bet set:

// on the server await page.waitForFunction('window.isReadyForPDF'); // now we know the page is ready for a screenshot 

The page might require authentication

If the page you’re screenshotting is part of a web application, it’s likely there’s an authentication step that’s required to view the page. This can be a bit of a pain to work around, but fortunately, Puppeteer provides enough control to programmatically log in to the application:

await page.waitForSelector('#username'); await page.waitForSelector('#password'); await page.evaluate(() => { document.querySelector('#username').value = 'my-username'; document.querySelector('#password').value = 'my-password'; document.querySelector('#log-in-button').click(); }); 

There are some downsides to this approach, though:

You now need to maintain a system account for logging in to the application for PDF screenshots
The extra login step adds quite a bit of time to the PDF generation process
Your PDF generation is now dependent on your authentication/authorization system
It just feels kind of wrong

Disclaimer: my PDF generator was written in .NET Core, so I actually used a library called Puppeteer Sharp which aims to replicate the API of the official Puppeteer library (which runs on Node). Some of the code examples above might be slightly off since I translated them from C♯ into JavaScript.

References/Attributions

[1]: https://stackoverflow.com/a/26265549/1063392

Minifigure/Chrome image from https://hackernoon.com/so-many-testing-frameworks-so-little-time-b03c707b8f90

The Next Chapter

2018-12-04T15:30:12-05:00

I’m incredibly excited to announce that I will be joining the GitLab team in just a few days!

I’m a big fan of GitLab - both the product and the company. I will be working as a Senior Frontend Engineer on the Release (Continuous Delivery) team.

If you’re not familiar with GitLab, here’s some info about the company.

ES6 Object Literal Shorthand Is Fun And Kind Of Scary

2018-11-16T14:53:36-05:00

One of the small but important features of ES6 is a shorthand syntax for creating object literals:

const person = { name, age, height, hairColor }; 

I can only assume this syntax sugar was added to the language because the ECMAScript gods were tired of looking at ES5 code like this:

const person = { name: name, age: age, height: height, hairColor: hairColor }; 

This seems like an obvious win - why would you want to write six lines of code when you could just write one?

The kind of scary part

There’s one downside to this syntax that’s not immediately obvious. Let’s say you have some code like this:

getPerson() { const height = 182; // ... 100 more lines of code ... return { name, age, height, hairColor }; } 

After working with this code a bit, you realize that it’s not super clear what unit the height variable uses, so you decide to use your editor’s auto-refactor feature to rename this variable from height to heightInCm:

getPerson() { const heightInCm = 182; // ... 100 more lines of code ... 

But wait! By changing the name of an internal variable, you’ve now changed the public interface of your function!

 // ... 100 more lines of code ... return { name, age, heightInCm, hairColor }; } 

If you’re not using a compile-safe language like TypeScript, you might forget to update the consumers of this function. Or even if you are using TypeScript, you might forget to update an HTML template that references this variable name. This has bitten me a few times in the past.

Like I said, it’s only kind of scary.

Shell-ing With TypeScript

2018-06-17T03:59:12-04:00

Here's a TL;DR for my future self when I inevitably forget the correct shebang to use.

If you’re like me, most of your time writing bash scripts is spent Googling basic things like “how to loop in bash” while grumbling to yourself how easy this would be in a sane language like TypeScript.

As it turns out, you can write shell scripts using TypeScript, and it isn’t even that hard! Here’s how:

1. Install dependencies

If you haven’t already, download and install node.

After node is installed, install both TypeScript and ts-node globally using npm:

npm install typescript ts-node -g

2. Write your shell script in TypeScript

Create a new .ts file with a shebang as its first line that points to ts-node:

#!/usr/bin/env ts-node

Then, add some TypeScript-y stuff:

#!/usr/bin/env ts-node  console.log('Hello from TypeScript!'); 

3. Make your script runnable

After saving your TypeScript file, you’ll need to update its permissions to allow it to be executed:

chmod u+x your-shell-script.ts

4. Run your TypeScript file

You can now run the script as you would any other command-line utility:

> ./your-shell-script.ts

…which should result in a friendly message in your terminal:

> ./my-shell-script Hello from TypeScript!

5. Prove to yourself that it’s working

That seemed a bit too easy - shouldn’t there be an intermediate build step in there somewhere? As a sanity check, update your .ts file with something that shouldn’t compile:

#!/usr/bin/env ts-node  console.log('Hello from TypeScript!'); // TypeScript compiler error: // Type '4' is not assignable to type 'string'. var myStr: string = 4; 

Rerunning your script will now result in something like this:

/Users/nathanfriend/.nvm/versions/node/v7.10.0/lib/node_modules/ts-node/src/index.ts:250 return new TSError(diagnosticText, diagnosticCodes) ^ TSError: ⨯ Unable to compile TypeScript: test.ts(5,5): error TS2322: Type '4' is not assignable to type 'string'. at createTSError (/Users/nathanfriend/.nvm/versions/node/v7.10.0/lib/node_modules/ts-node/src/index.ts:250:12) at getOutput (/Users/nathanfriend/.nvm/versions/node/v7.10.0/lib/node_modules/ts-node/src/index.ts:358:40) at Object.compile (/Users/nathanfriend/.nvm/versions/node/v7.10.0/lib/node_modules/ts-node/src/index.ts:546:11) at Module.m._compile (/Users/nathanfriend/.nvm/versions/node/v7.10.0/lib/node_modules/ts-node/src/index.ts:430:43) at Module._extensions..js (module.js:580:10) at Object.require.extensions.(anonymous function) [as .ts] (/Users/nathanfriend/.nvm/versions/node/v7.10.0/lib/node_modules/ts-node/src/index.ts:433:12) at Module.load (module.js:488:32) at tryModuleLoad (module.js:447:12) at Function.Module._load (module.js:439:3) at Function.Module.runMain (module.js:605:10) 

This is good! You shouldn’t be able to run a TypeScript file if it contains compile-time errors.

So where is that compile step?

There isn’t one! Well, not one that you have to explicitly run, anyway. This is the magic of ts-node - it compiles and runs TypeScript files on the fly much like regular node runs JavaScript files.