Bounded-Error Log

I’m still doing interpretability at Anthropic. This year, among many other research updates, we scaled sparse autoencoders to Claude 3 Sonnet. There’s not a whole lot I have to add. I like to think I improved at various clichéd abstract work skills that I don’t have anything insightful to say about, the most front-of-mind being communication and prioritization. It’s good to, uh, communicate what everybody is prioritizing and communicate all the information everybody needs to prioritize things. Thanks for coming to my TED talk.

In my personal life, I went to more in-person social events than ever before. I took a bunch of improv classes and might actually end up performing soon™. I went to a Jacob Collier concert and a Bear Ghost concert. I scored 141/150 plus two beers on the AMC 12. I attended three separate furry conventions, one of which I believe I have to credit with indirectly motivating me to hit the gym semiregularly for the first time in my life. (Also I went to Tural for summer vacation.) I’m pretty happy with all that, but it also doesn’t really add up to exciting reflections.

If you’ve dabbled in data visualization, there’s a good chance you’ve encountered the colormap viridis, which was created for matplotlib in ~2015.

You may also know about some of the considerations went into the colormap’s design, chiefly perceptual uniformity: equally distant colors on the colormap should “look equally different” to humans.

I won’t discuss color theory much here; the talk introducing viridis (below) is quite good, and there are many other resources online, such as Jamie Wong’s From Hexcodes to Eyeballs or Bartosz Ciechanowski’s Color Spaces.

Instead, I nerd-sniped myself with a different question: How is viridis, specifically, defined?

This seems like it should be easy to answer. Just go into matplotlib and find the source code for viridis, right? Unfortunately, that source code is just a list of 256 RGB triples along which colors are interpolated. This makes sense for efficiency and even portability because (as we’ll see) the formula for producing viridis is incredibly complicated, but it isn’t a very satisfying answer. How did those triples originally come to be?

In theory, the idea here is similar to when I was learning React/Redux and diving into SQL selects. In practice, I think most of D3’s complexity isn’t exactly in a direction that is elucidated by writing down types for everything, so the title is a mere personal snowclone. I’m just writing things out to an arbitrary amount of detail until I understand them and can refer to what I wrote here later.

Background

D3 is “a JavaScript library for visualizing data”. It has a lot of sublibraries that interoperate well but could be used separately — for example, it has utilities for manipulating colors, time, and SVG paths. Of the various concepts, though, I think D3 selections are the most distinctive and fundamental, so they are the focus of this post.

At a high level, D3 selections feel like jQuery. You run some code and it goes into the DOM and adds, deletes, and mutates a bunch of elements. The docs even endorse monkeypatching d3.selection to add custom helpers. However, D3 has data binding and batch operations that make it easy to change the DOM in a way that resembles reconciliation in a framework like React.

Selections

API: Selecting Elements.

A D3 selection holds an array of arrays of nullable¹ DOM elements. The intermediate arrays are called groups. Additionally, each group in a selection is associated with a parent node. During basic D3 usage, you might only ever work with selections with a single group and ignore parent nodes.

When relevant, I will call the index of an element inside its group the “within-group index” and the index of a group among all groups in a selection the “across-group index”.

…it’s still you. Looking at yourself in the window. The second afternoon after you finally get COVID for the first time.

As previously reported, I left Zoom late last year and spent a bit of time unemployed, traveling for some of it but mostly staying home. In the process, I got COVID, though not with a particularly interesting story. Would not recommend.

Then I started work at Anthropic doing interpretability research — moving way back into my comfort zone in a way by returning to my web dev roots to create many of the visualizations we cared about, and way out of it in another by jumping into the deeply theoretical end of research, in a field where my total experience is one college course and one casual reading group. Still, I figured some things out and we published Towards Monosemanticity in early October.

I don’t have much to add to the research results in the paper, though I can share some trivial, mildly entertaining anecdotes about the process:

I want to add a second word in the title, something like “Koans” or “Vignettes”, but I don’t know a word with the right connotations.

I realized recently that I have been walking around for a long time with some confusion and unknown unknowns about how concurrency works in various settings, and decided to write about it until I stopped being confused. This post doesn’t therefore have much of a “point”.

Concurrency and Parallelism

Wikipedia, as of time of writing:

Concurrency is the ability of different parts or units of a program, algorithm, or problem to be executed out-of-order or in partial order, without affecting the outcome.

There are two broad reasons concurrency is useful. One is for performance: if you want your computer to perform as many floating point operations as possible by lunchtime, you want all CPUs/GPUs/etc. to be performing operations simultaneously. Another is that you’re in a problem domain where you simply can’t predict the order of events: you’re writing a user interface, and the user can click on any of multiple buttons in any order; or you’re writing a web server, and any number of clients can request any pages in any order. These reasons are not mutually exclusive.

Passwords. It’s 2023 and we still have to deal with them.

Many people know that, per the canonical xkcd, sequences of randomly chosen words such as

make relatively memorable but hard-to-crack passwords. One popular strategy for randomly choosing words is Arnold Reinhold’s Diceware™, a list of 6⁵ = 7776 “words” that you can randomly sample from by rolling five dice (analog or digital). (I won’t go into topics like how to calculate the entropy of passwords and how long a password you should try to have here, since most Diceware overviews already discuss them at length.)

A few people have iterated on the concept since then: probably most notably, the Electronic Frontier Foundation published their own word list in 2016, with words chosen to be more well-known and memorable, at the cost of taking longer to type. I’m a fast typer and prefer the EFF’s wordlist over the original, and am very grateful to them for creating it, but after generating quite a few passwords with it over the last few years, I began to feel that it still had a lot of room for improvement.

The title is clickbait. I did not design and implement a programming language for the sole or even primary purpose of leaderboarding on Advent of Code. It just turned out that the programming language I was working on fit the task remarkably well.

I can’t name just a single reason I started work on my language, Noulith, back in July 2022, but I think the biggest one was even more absurdly niche: I solve and write a lot of puzzlehunts, and I wanted a better programming language to use to search word lists for words satisfying unusual constraints, such as, “Find all ten-letter words that contain each of the letters A, B, and C exactly once and that have the ninth letter K.”¹ I have a folder of ten-line scripts of this kind, mostly Python, and I thought there was surely a better way to do this. Not necessarily faster — there is obviously no way I could save time on net by optimizing this process. But, for example, I wanted to be able to easily share these programs such that others could run them. I had a positive experience in this with my slightly older golflang Paradoc, which I had compiled into a WASM blob and put online and, just once, experienced the convenience of sharing a short text processing program through a link. (Puzzle: what does this program do?) I also wanted to write and run these programs while booted into a different operating system, using a different computer, or just on my phone.

As I worked on it, I kept accumulating reasons to keep going. There were other contexts where I wanted to quickly code a combinatorial brute force that was annoying to write in other languages; a glib phrasing is that I wanted access to Haskell’s list monad in a sloppier language. I also wanted an excuse to read Crafting Interpreters more thoroughly. But sometimes I think the best characterization for what developing the language “felt like” was that I had been possessed by a supernatural creature — say, the dragon from the Dragon Book. I spent every spare minute thinking about language features and next implementation steps, because I had to.

The first “real program” I wrote in Noulith was to brute force constructions for The Cube, for last year’s Galactic Puzzle Hunt in early August, and it worked unexpectedly well. I wrote a for loop with a 53-clause iteratee and the interpreter executed it smoothly. Eventually I realized that the language could expand into other niches in my life where I wanted a scripting language. For example, I did a few Cryptopals challenges in them. It would take a month or two before it dawned on me that the same compulsion that drove me to create this language would drive me to do Advent of Code in it. That’s just how it has to be.

This post details my thought process behind the design of this language. Some preliminary notes:

My seventh year doing Mystery Hunt with ✈✈✈ Galactic Trendsetters ✈✈✈, and after a hiatus it was in person again! This also makes it my first in-person Mystery Hunt as an alumnus, where I flew in and stayed at a hotel. How time marches on… I appreciated getting to see everybody on Galactic, as well as quite a few internet puzzlers at the location where all the cool people always go, Flour Bakery and Cafe.

Campus hadn’t changed too much. There were more card readers, but also fancy kiosks where ID cards could be printed on demand (via the official 1.2/5★-rated app). I set aside a little time before kickoff to try to locate a working kiosk to print my ID, but the two kiosks I found west of Mass Ave, in W20 and W35, were both out of order; only much later did I print a card in 16. But I am a card-carrying alumnus now. Galactic had two classrooms in 4-2 and lots of masks and tests. One of my teammates brought their dog. It was a fun time.

As typical nowadays, the hunt announcement and kickoff began with a facade theme of a museum. However, the twist was handled a bit differently — kickoff had an additional diegetic level: normally the story is followed by an out-of-character talk about health/safety and policies, but this year that talk, while still in a different universe from the museum, was intertwined with an introduction to MATE, the AI who had ostensibly been writing all the puzzles. Over the course of the hunt, instead of discovering a possibly predictable secret plan or betrayal by MATE, we instead found ourselves on its side because (in the outer fictional diegetic level) teammate had shut off some other “overly creative” AIs and overworked MATE.

One day I’m going to run out of the energy to find barely adequate allusions for the titles and thematic music videos for the openings of these end-of-year posts, and they’ll just be called “2095 in Review” or whatever. Or maybe I’ll just stop making them. But not today.

Good song. Good animation. Incredibly out of place on its YouTube channel, in the most inspiring chaotic good way.

I closed out last year by saying that I wanted to accomplish a “big milestone” this year. I actually had a specific milestone in mind that I did not actually achieve and will not reveal, but I made good progress towards it, and a lot of other things happened, enough that I think I’ll count that as achieved.

The big thing is that I left my job at Zoom to have some time for myself and family… though not before helping to give feedback on a draft internet standard, publish a cryptography research paper (on which I’m the “first author”, strictly due to the vagaries of the English alphabet), and launch end-to-end encrypted email. It was a productive year! I feel like I should have more to say about all this, but it’s hard to think of anything that I didn’t already write about last year and also doesn’t require a blockbuster-length list of prerequisites. However, if you ever want to hear about the difficulties of actually getting end-to-end encryption into production in excruciating detail, invite me to a cocktail party with a lot of whiteboards.

Code golf is the recreational activity¹ of trying to write programs that are as short as possible.² Golfed programs still have to be correct, but brevity is prioritized above every other concern — e.g., robustness, performance, or legibility — which usually leads to really interesting code.

I think code golf is a lot of fun (although I think a lot of things are fun, so it’s one of those hobbies that I get really into roughly one month every year and then completely forget about for the remaining eleven). I wanted to write an introduction because I don’t know of any good general introductions to code golf, particularly ones that try to be language-agnostic and that cover the fascinating world of programming languages designed specifically for code golf, which I’ll call golflangs for short. But more on that later.

Note: If you are the kind of person who prefers to just dive in and try golfing some code without guidance, you should skip to the code golf sites section.

A simple example

Of course, there’s a reason most code golf tutorials focus on a single language: most code golf techniques are language-specific. The Code Golf & Coding Challenges (CGCC) StackExchange community has a list of some golfing tips that apply to most languages, but there are far more tricks in just about any language-specific list, and most of the intrigue lies in knowing the language you’re golfing well. So to provide a taste of the code golf experience, let’s golf a simple problem, Anarchy Golf’s Factorial, in Python.

In this problem, we have to read a series of positive integers from standard input, one per line, and output the factorial of each, also one per line. Here’s a stab at a simple, direct implementation with no golfing at all:³

def factorial(n):
    if n <= 1: return 1
    return n * factorial(n-1)
try:
    while True:
        print(factorial(int(input())))
except EOFError:
    pass