Ryan Greenberg

This is a story that gives you a sense of some of the problems that we had in the early days of Twitter Ads. It’s based on a presentation I gave for Twitter’s “War Stories” series in 2015. I included screenshots and charts from that presentation that have more internal details than I would normally share since this happened 15 years ago and Twitter as a company no longer exists.

It was the spring of 2011 and love was in the air. And not just any type of love: royal love. The entire world was preparing for Prince William’s marriage to Miss Kate Middleton. Buckingham Palace had issued a 91-page press release with details about the motorcade, the wardrobe, the ceremony. On page 56 they wrote, “Twitter has said it may need to bring in extra servers to cope with the anticipated demand on that day.” This is just one more exhibit in the long history of the gap between royalty and reality; we said no such thing. We did, however, bring in one extra server just to be safe.

This was a different time at Twitter. We only showed ads in search results. Ads were only on twitter.com, not in our mobile clients. Promoted trends were our biggest source of ad revenue; they brought in a modest $30 million a year. There was only one worldwide promoted trend each day. But not everything was different. In 2011, we also wanted to make money. So we took the royal wedding and sold #RoyalWedding (Promoted) to SlimFast.

When you bought a promoted trend you were actually buying two things: placement at the top of the trending topics, and exclusive placement in search results for that trend. Exclusive means that you paid a flat rate and didn’t compete in an auction. Normally, the ad server runs an auction every time a users does a search to determine which ads to show. The winner is determined by advertisers’ bids and the predicted click-through rate. (The winner pays the runner-up’s bid for each engagement; it’s a Vickrey auction. It’s also more complicated than I remember now or understood then). To get your ads to show up, you pay more, have more engaging content, or buy exclusive placement and skip the auction. This way when someone clicked your trending topic, your ads were guaranteed to appear at the top of the results.

Continue reading →

There are so many way to get a file from one computer to another: put it on a server, AirDrop, email, Slack, Google Drive, Dropbox, to name a few. Magic Wormhole is one more handy addition to that magic-wormhole is one more addition to that list.

What I like about magic-wormhole is that it’s a CLI and is completely ad hoc. There’s nothing to clean up when you’re done and no leftover artifacts in your email/Drive/Dropbox/etc. If you want to send a bunch of files, you don’t have to create a zip; that happens automatically. You run wormhole send on one machine, wormhole receive [code] on the other, and you’re done. The codes are short and easy to say aloud. Here’s what an exchange looks like:

Computer A

$ wormhole send projects
Building zipfile..
Sending directory (86.9 MB compressed) named 'projects'
Wormhole code is: 44-hemisphere-lemon
█▀▀▀▀▀▀▀█▀▀██▀▀█▀██▀▀▀█▀▀▀▀▀▀▀█
█ █▀▀▀█ █▄▀  █▄█▀▄▄ ▀▄█ █▀▀▀█ █
█ █   █ █▄▄▀▄██ ▀▀▄█ ██ █   █ █
█ ▀▀▀▀▀ █▀█ ▄ ▄▀█▀█ ▄ █ ▀▀▀▀▀ █
█▀█▀▀█▀▀▀▄ ▀▀▀▀█▀   ███▀██▀█▀▀█
█▄█▀ █ ▀▀▄█▄   ▄█  ▄▄▀  ▄▄   ▄█
█ █▀▄▀▀▀▄█▄ ▀██▀██▀▄▄ █████▀▀██
█▄▄█ ▄▀▀▄  ▀██▀▄▄▄ ▄ ▄▄ █▀▀█  █
█▀▀▀█▀ ▀▀▄██ ██ ▄██▀█▄▀▄█ ▄█▀ █
█▀█ ▄▀ ▀██▄█ ▄▀  ▀█ ████▄▀▀ ███
██▀█▀█ ▀▄ █▄▄█  ▀  ▀▄ ▀ ▀▀    █
█▀▀▀▀▀▀▀█ ▀█ ▄█▀█▀ ▄  █▀█ ▄ ▀▄█
█ █▀▀▀█ █▄▄█  ▄▄▀█▀█▄ ▀▀▀ █▀█▄█
█ █   █ █ ▀▄▀▀▄█▄ ▄▀▀█▄█▀▄▄▄█ █
█ ▀▀▀▀▀ █▀ ▀▄▀ ▄█▀█▀  ▀█▄ ██ ██
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
On the other computer, please run:

wormhole receive 44-hemisphere-lemon

Computer B

$ wormhole receive 44-hemisphere-lemon
Receiving directory (86.9 MB) into: 'projects'/
1093 files, 86.7 MB (uncompressed)
ok? (Y/n): Y
Receiving (->tcp:192.168.1.186:54503)..
100%|██████████████████████████████████| 86.9M/86.9M
Unpacking zipfile..
Received files written to: /Users/ryan/Downloads/projects

The code (44-hemisphere-lemon in this example) is short-lived and can only be used once. The docs have more details on how the exchange works. The package is magic-wormhole, with wormhole as an alias for the magic-wormhole command.

Update: a friend pointed me at a Rust implementation of magic-wormhole which gives you a single binary to install. The Homebrew package is magic-wormhole.rs.

At Twitter I worked on the team that ran ads.twitter.com, the advertiser dashboard where people managed their campaigns. This involved selecting tweets to use as ads, setting budgets and timelines, and viewing analytics about how they were performing. One afternoon in April 2011 we got a note from an advertiser—HBO, I think—that they signed in and found themselves looking at another advertiser’s campaigns. That sounded implausible, but they included screenshots that were quite convincing. In the top corner it showed them signed in as a different user. On the dashboard was a table of stuff they were absolutely not supposed to see.

I should start by noting that for this bug and investigation, I did nothing. At the time I was working on ads in users’ timelines and happened to sit next to the people building the advertiser dashboards. I had less than a year of professional experience and no helpful insights to offer. All I did was listen to and be grateful that nobody was asking me to fix it. This is all to say that when I say “we” below, I am being quite generous to myself—Andrew, Avi, Ben, Colin, Greg, and Lennon did all the work. It’s one of my favorite bug memories though because the source of the bug was so unfamiliar to me.

To return to the facts: this is a terrifying bug. Causing site downtime: bad; losing money: also bad; corrupting data: very bad; but showing one user’s data to another, especially when the data is private, concerns money, etc. is arguably the worst. First, we put the site into maintenance mode so nobody could sign in. Then we added piles of logging everywhere, and in particular to the place where we fetched the user in the sign in path. If we fetched a user that didn’t match the request (for example, if it came back with a different user name than the sign in request), we logged it and threw an exception. This gave the user an error and forced them to sign in again, which is much better than signing in as somebody else.

Continue reading →

Joining the Ads Team

I got a job offer from Twitter in the spring of 2010. I was an avid Twitter user and thrilled at the prospect of working at the company. I accepted without even knowing what team I would join. After I signed my recruiter asked, “Would you prefer the analytics or monetization team?” I had just finished a course on data visualization that I loved, so analytics seemed like a good choice. “Analytics,” I replied. A short time later I heard back, “Sorry, that’s not going to be an option.” The reason, I later learned, was that between email 1 and email 2 Twitter had acquired a company that was the new analytics team.

So monetization—ads—it was. When I joined I doubled the size of the ads frontend team. Right before I joined Twitter launched Promoted Tweets in search results (“promoted” was the term Twitter used for ads). The week I joined the team launched Promoted Trends (ads in trends). In the next few months we launched Promoted Accounts (ads in Who to Follow). I had very little to do with this. As a junior engineer, a new grad, and a new hire, I paired with more experienced engineers and helped polish the launch. In my first months I built syndication APIs for partners like Hootsuite, TweetDeck, and Google to show ads. I also fixed bugs and responded to urgent issues, like when one of my favorite executives came to the ads bullpen with a problem. “We can’t,” he said, “have McDonalds pay $80,000 to promote the trend #McRib only to have the top tweet in search results be ‘rat meat.’”

Continue reading →

I redid how images render on my site. While creating a new component for images, I learned about the <figure> and associated <figcaption> elements. Now my images render like this:

<figure>
  <a href="...">
    <img src="..." />
  </a>
  <figcaption>...</figcaption>
</figure>

I’ve never seen these before—are they new? I wondered. This is a recurring theme in my TILs, but no. Not even a little bit. They’re additions to HTML5 that arrived at different points between 2008–2015. There’s probably a meta lesson here about not assuming things I don’t know must be new, but that’s for me to dwell on. Here are a few more that I learned about and incorporated into my site:

• <main> the content part of the page
• <header> and <footer>
• <nav> for site navigation
• <article> for individual posts

I remember intense discussion about the value of semantic markup when I was paying closer attention to HTML. I don’t know how important that is in itself, but practically these tags help screen readers and other tools navigate your site better. I also find it nicer as an author to see more named tags instead of my usual pile of <div>s.

I have too many git branches and like many people I developed a convoluted script to show me the most recent ones. I ran this command many times a day for years:

function branches {
  for k in `git branch | sed "s/^..//"`; do
    echo -e `git log --color -1 --pretty=format:"%Cgreen%ci %Cblue%cr%Creset" "$k"`\\t"$k";
  done | sort
}

This outputs the date as the first column in the output so that sort works. Turns out with modern git it’s totally unnecessary. Just configure git branch to sort by the last commit date:

git config branch.sort committerdate

You can also supply this on a one-off basis as git branch --sort committerdate.

What else can you sort by? The manual says “The keys supported are the same as those in git-for-each-ref(1).” The manpage for git-for-each-ref lists a ton of fields, including some that don’t make sense to me as sort keys. Here are a few that stood out as interesting:

• authordate, committerdate, creatordate, taggerdate
• contents:size: The size in bytes of the commit or tag message.
• push: The name of a local ref which represents the @{push} location for the displayed ref.
• refname: The name of the ref (the part after $GIT_DIR/) (this is the default).
• upstream: The name of a local ref which can be considered “upstream” from the displayed ref.

I have multiple monitors and I use the cmd + tab application switcher often. I find the current behavior of macOS pretty confusing. By default, it only shows the switcher on the most recently used screen. But that’s not the screen I’m looking at when switching to another application!

After getting confused a bunch of times, I found this setting which lets you show the switcher on all screens:

defaults write com.apple.dock appswitcher-all-displays -bool true
killall Dock # you need to restart the Dock for settings to take effect

It’s wonderful.

While researching interactive flags in less I learned about the LESS env var. This lets you set less’s startup flags globally. For example, if you always want ANSI color escape sequences to be interpreted (so you see colors instead of control characters), put LESS=-R in your shell config.

export LESS=-R
less request.log

export LESS=
less request.log

We have piles of shoes in our entryway. We have kids who help the piles grow. We have a dog whose love language is taking shoes, nibbling them, leaving them in unexpected places, and returning for more until our living room contains two different flip flops, a sequined sneaker, a hiking boot, and one adult Croc (I’m guilty of piling too). We have a problem.

I’m a problem solver. I found some very nice, clear, stackable shoeboxes with drop fronts from The Container Store. I also found that they are $20 each and I don’t want to spend $250 on shoe storage (in fairness, they are less during sales). I found a cheaper version on Amazon and after 20 minutes of assembly I had them arranged in our closet. The boxes are large enough that they can accommodate two pairs of kids’ shoes or women’s flats, but it’s a little messy. I’ve been looking for opportunities to design and 3D print something custom and a divider for these shoeboxes seemed low stakes.

My plan was to design an I-beam to sit in the middle of the box, held in place with pressure from the top and bottom. First lesson: the parts of an I-beam are the flange (top and bottom) and the web (the tall part). It took me awhile to prototype the dimensions for the divider. The interior height of the box was hard to measure accurately. I ended up doing more test prints than I would have liked. I figured I would measure, print a sample, and adjust. Instead I printed five samples and made adjustments each time. By the end, I realized I could prototype more quickly with cardboard and used that to figure out the final height.

I used Autodesk Fusion to design this simple part. It’s a very complicated program! The target audience is clearly professionals who know what they’re doing and likely have some training in engineering. Neither of those is me. That’s not to say it’s impossible to use—there are tons of tutorials and videos online—just that I could not open the program and figure out what to do without assistance. Fusion costs $700–$2000 a year but there is a personal use version that is free.

Fusion is a parametric modeling program, which is different from other programs I’ve used like SketchUp. Everything you define in your model is a “parameter” which can be adjusted in the future. For example, an object has such-and-such width and height, extruded by another dimension, offset from another object by this amount, and so on. You can use variables for these values, you can do math like divider_width_mm + border_mm, etc.

To make my I-beam I drew a rectangle with my height and width and extruded the depth. Then I added the flanges, setting the width I wanted to fit into the shoebox. I read that I should add a fillet between the web and the flange. So I planned to do that, except first I needed to figure out what a fillet was.

It’s fun to learn new stuff (including—or maybe even especially?—a complicated program like Fusion). I had heard the term “chamfer” before but didn’t actually know what it was. I had not heard the term fillet, but it’s related and what I needed. A fillet is a rounded transition on an edge. Two pieces that join at a sharp, 90° angle can be fragile, so you add a fillet to strengthen the part. In Fusion, this is easy: you select an edge and select Modify → Fillet.

I decided to add a patterned perforation to the panel so that it looks nicer, uses less filament, and I learn how to do it. In Fusion, you do this by drawing your shape, constraining to a certain distance from the edge, and then turning it into a repeating pattern.

After exporting from Fusion I brought the part into Bambu Studio to slice and print. I ended up printing the structure by laying it down on the bed so it looks like a capital letter I with serifs from the top view. This way it doesn’t need supports to print the upper part of the I (you can’t print into midair). It’s hard to think in 3D!

This was when I realized that the hexagons I picked had a steep angle that, in theory, has the same support problem. Instead of going back to the model and changing to, say, diamonds, I read that I might be able to get away with my 60°-off-vertical overhangs by printing at a smaller layer height, slower speed, and with more ventilation.

It took about 5 hours to print. The edges of the hexagon were a little messy, but it worked! I’m pretty happy with the way this thing looks given that I had no idea how to use Fusion at the beginning of the project.

I love the less pager. You can use it to read, search, tail, and more. As with many versatile programs, however, the man page for less has a bazillion different options:

$ man less

NAME
    less - opposite of more

SYNOPSIS
    less [-[+]aABcCdeEfFgGiIJKLmMnNqQrRsSuUVwWX~]

There’s hardly a letter it doesn’t use!

Of these, my favorites are:

• -S or --chop-long-lines. Truncate lines rather than wrapping them.
• -N or --LINE-NUMBERS. Show line numbers at the start of each line.
• -R or --RAW-CONTROL-CHARS. Show control characters vs. interpret them.

You can supply these as flags when starting less. But you often don’t start less directly; it’s set as your pager and then gets invoked when you run commands that produce lots of output, like man or git. No matter because you can type the startup flags while less is running to toggle their behavior!

If lines are wrapped and you want to truncate them, just type -S. You’ll see a message at bottom describing the change and then you press enter to accept. Want to see what line number something is on? Type -N, enter.

As an aside: the dominant pager that preceded less was more. Despite the names, less has more features. On macOS, more is now less (or more precisely, /usr/bin/more and /usr/bin/less are the same file):

ls -lih /usr/bin/more /usr/bin/less
1152921500312523552 -rwxr-xr-x  2 360K Nov 22  2025 /usr/bin/less
1152921500312523552 -rwxr-xr-x  2 360K Nov 22  2025 /usr/bin/more

Whether you get more’s impoverished capabilities or less’s expansive set is determined by the name you use to invoke it.

I learned a new word today: mojibake!

Some of my old blog posts have random gibberish in them like ñ or ’. I always assumed this was the result of some character encoding mismatch but I didn’t know exactly what was going on. While I was fixing this issue via find/replace technology, I did some reading to learn more.

The short version is a mismatch between UTF-8 and Latin-1 (ISO-8859-1) encodings. As an example, this is how ñ becomes ñ.

First, when I type ñ in my CMS it is represented in UTF-8:

>> "ñ".codepoints
=> [241]
>> "ñ".bytes
=> [195, 177]

In UTF-8, codepoints above 127 are represented with more than one byte. You can see here the codepoint 241 (U+00F1) corresponds to two bytes, 195 and 177.

Then, when these bytes get written to the DB, they are not converted from UTF-8 to Latin-1. We store [195, 177] but believe they are Latin-1. Later, we re-encode these bytes as UTF-8 to convert from not-actually-Latin-1:

bytes = "ñ".bytes               # => [195, 177]
latin1_str = bytes.pack("C*").force_encoding("ISO-8859-1")
puts latin1_str.encode("UTF-8") # => ñ

I was fixing these bugs in old posts and one AI result said, “Oh yeah, this is a classic mojibake problem.” My first reaction was, “I don’t know man—How ‘classic’ can it be if I’m hearing it for the first time after working with strings and encodings for years?” But then again, you have to learn everything sometime. Maybe it is classic!

Mojibake or 文字化け is a Japanese word for exactly this kind of garbled text. Before UTF-8, the dominant character encoding in Japan was Shift JIS, and this kind of character mangling was probably common when text interacted with systems that did not convert encodings properly.

The “moji” in mojibake (文字) means text and, yes, it’s the same moji found in emoji (絵文字)! I knew that emoji originated in Japan but I did not know that the name was actually Japanese. I assumed that emo was related to emotion since the first emoji were all faces showing different emotions. Nope! E (絵) means pictures, moji (文字) means text.

Now that UTF-8 is the dominant text encoding, I’m pretty confident you’ll see all the text here correctly and not as a “classic” mojibake issue.

You can open macOS’s Spotlight with cmd + space. Then you can type in math like (4 + 23) * 92 and it will show you the answer. I’ve been using this feature for awhile but I just learned that it can do a lot more than I realized! This came up because I typed 2**256 and it gave me the answer. Interesting…what else can it do?

I couldn’t find a comprehensive list of supported features. This Stack Overflow answer has a long list; one other commenter says it’s everything the macOS calculator can do. All of these things work:

• Exponents, written as any of 2**3, 3^2, or pow(2, 10)
• Factorials, written as 5! or fact(7)
• Trigonometry, functions like sin, cos, tan, and constants like pi.
• Lots of other math. You can round(), pow(), abs(), sqrt(), ceil(), floor(), and more.

The absolutely best part is that it does unit conversions. Historically this is one of the things I have always typed into Google search. I’m sure I’m going to use this all the time now. Type 95F and it tells you that’s 35°C. Type 170 lbs and it says 77 kg. You can specify the output unit like 4 Tbsp in mL or 10 miles in feet (it does not, however, support unhinged unit conversion like “1 mile in furlongs”).

One really nice UX touch is that Spotlight automatically selects the answer, so you can easily copy it with cmd + C without dragging.

Sometimes when I learn about a tool I think “wow, that would have been useful [when I had some problem in the past]”. That happened with pkill, which is a command you can use to stop a process accurately.

Many years ago I was slogging through debugging a Selenium test failure in a Rails application with my coworker, Ron. Every time we made a change we had to stop the server, restart it, and re-run the test suite. This had a bunch of steps so naturally we wrote a script to automate them. At the end of the script, we killed the Rails server by looking for the process using port 3000 with kill -9 $(lsof -t -i:3000).

• lsof lists open files. In unix/linux, everything is a file including things like network connections (via sockets).
• -t terse output—just the pid
• -i:3000 process using port 3000

It worked! Until one run when, all of a sudden, my entire browser disappeared. It was unexpected, abrupt, and rude. We were using that browser! Ron and I were really baffled … until we realized that there were two processes using the port: the server and the browser (maybe with a keep alive connection?).

This is what the lsof output looks like when you look for processes using a port:

$ lsof -i:3000 -P

COMMAND  PID    TYPE  NODE  NAME
ruby3.2  42727  IPv6  TCP   *:3000                           (LISTEN)
ruby3.2  42727  IPv6  TCP   localhost:3000->localhost:57537  (ESTABLISHED)
Google   6439   IPv6  TCP   localhost:57537->localhost:3000  (ESTABLISHED)

Notice that process ID 42727 is using :3000 in the LISTEN state, while that same process and Chrome are both using it for a connection in the ESTABLISHED state. So we got unlucky on one run, grabbed the browser’s process ID, and unceremonously killed it. At the time, we worked around this by adding another flag, -sTCP:LISTEN. This filters to TCP sockets in the LISTEN state, so we’d always get the server process.

pkill would have been perfect for this! It stops processes by matching the process name. For example, pkill ruby would stop all running ruby processes. That’s a little aggressive if you only want to stop one process. With pkill -f you match the program arguments as well, like ruby bin/rails so pkill -9 -f rails would do it. (You could also argue that I should have the server process itself write its pid to a file to later stop it. You’re probably right!)

One thing I love is programs that have two modes to either (1) take action or (2) preview that action. Many systems expose this as a “dry run”. In the case of pkill, the dry-run equivalent is pgrep, which works the same way, but just outputs the pid instead of sending a kill signal.

pkill has a ton of other options so be sure to look at the tldr or man page.

A poorly understood aspect of male culture is that we express affection by curating content on the Internet to share with the people we love. It’s a bit like being a modern hunter-gatherer. So instead of “Why is Doug showing me this dog video?” you should perhaps think “Wow, of all the content in the world Doug picked this for me. He really cares.”

Circa 2008 to 2023, I spent too much time on Twitter. Was it good for me? No. But did I love every minute of it? Absolutely not. I will say this, however: I curated the hell out of some tweets. Over 10,000 gems. Consider for a moment how many tweets a person has to read in order to find 10,000 good ones. A million? Ten million? That amount of curation takes a tremendous toll, but it’s just something the modern man does for his loved ones.

Thankfully we were able to close that chapter, but the long-term effects on my mind remain. Under certain conditions I am physiologically compelled to recall specific tweets. Poor financial decisions? Please help me budget this my family is dying. My past actions return to me? Me sowing, me reaping. My kids’ latest pile of plastic junk? Marie Kondo in tears. Some random stimulus I haven’t quite figured out? Uh your dragon really should be trained by now.

As these once-bright stars of Twitter fade, I opened the Twitter Hall of Fame to capture the dying light.

• I crawled through the GraphQL API with credentials sucked out of my browser to download 10,064 old likes.
• I put them in a little local UI.
• I took these already curated tweets and ran them back again to find the ones that are (1) text-only, (2) not replies, and (3) made me laugh.
• Then I took the 267 tweets that passed that filter and put them into a little single-page archive.

Yes, you read that right. I took already curated content and re-curated it. Please remember this is a great sign of respect and affection in my culture. Enjoy the tweets.

(Right after I finished everything, I remembered The Verge did the same project. If your thirst for content cannot be sated, I’m sorry, but definitely check it out.)

GUIs are great! But I’m used to typing cd, and sometimes I prefer that to clicking around to get somewhere.

MacOS merges these worlds with cmd + shift + G. In the Finder it’s under Go → Go to Folder.... This opens a dialog where you can type a path to navigate to. It supports tab completion, it understands that ~ is your home directory, and it shows recently used options that you can navigate with your arrow keys.

It also works in the open/save dialog boxes! This is more hidden; as far as I know there is no menu that shows “Go To Folder” so you just have to know the cmd + shift + G shortcut.

Now when I’m not navigating to the Desktop, Documents, or Downloads (which each have their own keyboard shortcuts), I often use cmd + shift + G to get there.

While working on Slack’s unfurl previews I needed to test what happens when the fetcher cannot access a site. People put a lot of different URLs into Slack; some of them are going to be unfetchable for various reasons. Pointing the fetcher at a URL that doesn’t resolve, or one that results in a timeout, is straightforward, but I don’t know enough to set up various bad SSL/TLS scenarios. Surely one of them is going to have a bad config at some point.

That’s when I found https://badssl.com. It’s a project from some security folks that provides tons of misconfigured sites for exactly this purpose. Want to see what happens with an expired cert? Go to https://expired.badssl.com. How about a cert that’s been revoked? That’s https://revoked.badssl.com/.

For example, this is how curl behaves when the certificate doesn’t match the domain name:

$ curl https://wrong.host.badssl.com/
curl: (60) SSL: no alternative certificate subject name matches target host name 'wrong.host.badssl.com'
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

This is what it looks like when you navigate to a site where the certificate has expired:

I love that badssl.com gives you a quick way to check these behaviors interactively. For example, curl https://tls-v1-1.badssl.com:1011 works fine; going to that URL in Chrome presents a warning (because TLS v1.1 is deprecated).

There are more than 70 subdomains in 13 categories so you can trigger mixed content warnings, HTTP form behaviors, HSTS, and more.

diff is a well-known tool to see what lines are different between files; comm is a less-known tool to see the common lines.

I found comm when I was looking for a way to identify lines that are shared (or not) between files. Specifically, I had two lists: one with all the APIs in my codebase, and another with all the tested APIs. From this I wanted to get a list of untested APIs.

As an aside, my go-to in this situation is to drop in a Ruby one-ish-liner:

ruby -e "puts (File.readlines('all') - File.readlines('tested'))"

In Ruby, Array.- (minus) returns a new array after removing the elements from the right-hand side. IMO, this is pretty nice. Ruby has a wide variety of methods for combining arrays (+ to merge, - for difference, & for intersection, | for union). The only caveat is that this is trickier if you haven’t written the lines to a file. That’s pretty common in pipelines and was, in fact, exactly the situation in which I found myself.

Let’s say we have these two lists: mammals.txt and pets.txt:

$ cat mammals.txt
cat
dog
lion
rabbit
whale

$ cat pets.txt
cat
dog
fish
rabbit
snake

We can compare the files with comm:

$ comm mammals.txt pets.txt
		cat
		dog
	fish
lion
		rabbit
	snake
whale

The first time I saw this output I was pretty confused. Is this working? It kind of looks like I just dumped all the lines with weird indentation? comm produces output in three columns:

1. lines that are only in the first file
2. lines that are only in the second file
3. lines that are in both files

Here’s a labeled version:

$ comm mammals.txt pets.txt
just          | just       |
mammals.txt   | pets.txt   | both
--------------+------------+-------
              |            | cat
              |            | dog
              | fish       |
lion          |            |
              |            | rabbit
              | snake      |
whale         |            |

You can use -1 to hide the first column, -2 to hide the second column, and, yes, -3 to hide the third column. In this case, if you want just animals that are mammals and pets: comm -12 mammals.txt pets.txt:

$ comm -12 mammals.txt pets.txt
cat
dog
rabbit

If you want pets that aren’t mammals, that would be comm -13 mammals.txt pets.txt.

To get back to my original use case, if I want a list of untested APIs, you write:

comm -23 all_apis.txt tested_apis.txt

comm works with process substitution so you can also write:

comm -23 <(command to find all APIs) <(command to find tested APIs)

One caveat: the input lines need to be sorted, so you may need to add | sort.

Years ago I was at a coffee shop with my friend Nick. On a lark I ordered something on the menu that I had never seen before: a raspberry shrub. It was delicious! (and not a plant). I took a mental note to learn more about them, and when I got home I immediately did nothing. As time passed occasionally the thought would resurface like an itch. What was that drink called again? A shrub? Right, like a bush. I really should look it up. But the moment would pass. I repeated this process of not taking any action for quite some time until one day I was scrolling and the algorithm brought shrubs back to me. In fact it brought them back so often that I became A Guy Who Has Watched Too Many Videos On The Subject.

A shrub, I learned, is a mixture of a fruit syrup with an acid, usually vinegar, sometimes fruit juice. The combo is part of what makes it so refreshing. Instead of cloying sweetness, you get a more balanced taste. Shrubs are an acidulated beverage, sometimes called a drinking vinegar, that hit their peak popularity in the American colonial era. You can use them to mix cocktails or add them to sparkling water. Once you make a shrub, you typically let it rest in the fridge for a week or two for the flavors to combine and mellow. After that they’ll keep for months (this was historically one reason to make them—food preservation).

Armed with this knowledge I declared it the Summer of Shrub. Why dip your toe in a hobby when you can dive in? I had more shrubs than I could fit into the fridge. I bullied friends and coworkers into making their own. “Stop trying to make shrub happen!” I imagined naysayers naysaying. But none could be found: everyone else just sipped from a glass I foisted upon them and enjoyed a remarkably refreshing drink.

After taking a summer off, I’m back with the reboot: Summer of Shrub 2.0. This is a more demure edition, enthusiasm tamed by wisdom. You’re interested in trying some shrubs of your own? Oh sure, I’d be happy to share some tips.

If you’re getting started, two recipes I recommend are Christina Chaey’s Any-Berry Shrub or Michael Dietsch’s Cold-Processed Shrub at Serious Eats. In typical Serious Eats fashion, Michael’s recipe has background on what shrubs are, the various types, techniques, and more. Michael also wrote the book on shrubs. Naturally, I bought it. A few of my faves are the Strawberry Balsamic Vinegar Shrub and the Blackberry-Lime Shrub.

I really love Rebekah Peppler’s Sweet Bay-Peppercorn Shrub. It’s the backbone of a convincing non-alcoholic French 75. It has a lovely citrus + botanical flavor.

You absolutely do not need additional equipment to make shrubs—you almost certainly have everything you need in your kitchen now. But! I would hate to pass up an opportunity to recommend Crew Supply’s bottles. I keep my shrubs in them and they’re lovely.

As we hit the midpoint of summer, there’s still plenty of time to make shrubs (and make shrub happen).

awk is a whole programming language, but 90% of what I use it for is printing a subset of columns. This is pretty much covered by cut with two exceptions: re-ordering columns and printing the last column. Both cut and awk call columns “fields”.

Here’s some sample tab-separated data (formatted with column -t!):

FirstName  LastName  FavoriteColor  Pet
Elena      Kowalski  yellow         hamster
Marcus     Rivera    green          snake
Nadia      Ferreira  coral          rabbit
Yuki       Tanaka    pink           cat
Zara       Okonkwo   indigo         chinchilla

If you just want a list of favorite colors, you select the third field with cut -f3:

cat data.tsv | cut -f3
FavoriteColor
yellow
green
coral
pink
indigo

But if you want last name then first name, no dice. You can run cut -f2,1 but it will output them in the original order. For this I learned to use awk. You have to remember to specify the field separator with -F, then you put the program to run on each line in braces:

cat data.tsv | awk -F'\t' '{ print $2, $1 }'
LastName FirstName
Kowalski Elena
Rivera Marcus
Ferreira Nadia
Tanaka Yuki
Okonkwo Zara

If you want the last field in the input (in my case I was looking at log lines), you have two options: count the number of fields and use cut (no thanks, I’m too lazy) or use awk. The variable you want is $NF for “number of fields”. Since the fields are one-indexed, this will give you the last field.

cat data.tsv | awk -F'\t' '{ print $NF }'
Pet
hamster
snake
rabbit
cat
chinchilla

I usually have to look up -F'\t' even though I should probably remember “field separator” is -F.

I was filling out a form today and a friend joked, “You should put Number.MAX_SAFE_INTEGER in the budget field.” Woah! I didn’t know we had that constant.

Number.MAX_SAFE_INTEGER was added to JavaScript in 2015 as part of ES6. The constant is new to me, but the value is a number I know by heart: 2**53 - 1.

When I worked at Twitter, we switched from 32-bit IDs generated by MySQL to 64-bit IDs generated by Snowflake. Late in the project—just a few weeks before launch—we hit a small road bump: we could not use the new IDs in the browser.

With the Snowflake IDs, sometimes you would like a tweet and the request would fail, “tweet not found”. You would scroll the timeline and duplicate tweets would appear. These bugs and others were caused by the way JavaScript represents numbers.

JavaScript implents IEEE-754, which uses 64 bits to represent all its numbers—big, small, floating point, integer. Practically this means that you do not get 64 bits to represent integers. Of the 64 bits, 1 bit represents the sign, 11 bits are used to represent the exponent of a number, and the remaining 52 bits are what’s left for the mantissa (what you typically think of as the actual number). This means you can only represent integers faithfully up to 2**53 - 1, Number.MAX_SAFE_INTEGER, 9007199254740991. (This was 2010 and support for bigints didn’t land until 2018-2020.)

Above that limit, the numbers behave in unexpected ways:

> Number.MAX_SAFE_INTEGER
9007199254740991
> Number.MAX_SAFE_INTEGER + 1
9007199254740992
> Number.MAX_SAFE_INTEGER + 2
9007199254740992
> Number.MAX_SAFE_INTEGER + 3
9007199254740994

Tweets in the API came back like this:

{
  "id": 915710611442491393,
  "text": "..."
}

but the browser would parse the id as 915710611442491400. That’s close! But close isn’t usually what you want for IDs.

At the time, we solved the problem by adding "id_str" with a string representation of the ID. We couldn’t change or remove "id" because it was a widely used public API, but that vestigial field was a perennial cactus underfoot. People would do things like use the field, and get unexpected results when the ID was not the ID.

See also the archived source for snowflake and the archived twitpocalypse.com.