Ryan Greenberg

Joining the Ads Team

I got a job offer from Twitter in the spring of 2010. I was an avid Twitter user and thrilled at the prospect of working at the company. I accepted without even knowing what team I would join. After I signed my recruiter asked, “Would you prefer the analytics or monetization team?” I had just finished a course on data visualization that I loved, so analytics seemed like a good choice. “Analytics,” I replied. A short time later I heard back, “Sorry, that’s not going to be an option.” The reason, I later learned, was that between email 1 and email 2 Twitter had acquired a company that was the new analytics team.

So monetization—ads—it was. When I joined I doubled the size of the ads frontend team. Right before I joined Twitter launched Promoted Tweets in search results (“promoted” was the term Twitter used for ads). The week I joined the team launched Promoted Trends (ads in trends). In the next few months we launched Promoted Accounts (ads in Who to Follow). I had very little to do with this. As a junior engineer, a new grad, and a new hire, I paired with more experienced engineers and helped polish the launch. In my first months I built syndication APIs for partners like Hootsuite, TweetDeck, and Google to show ads. I also fixed bugs and responded to urgent issues, like when one of my favorite executives came to the ads bullpen with a problem. “We can’t,” he said, “have McDonalds pay $80,000 to promote the trend #McRib only to have the top tweet in search results be ‘rat meat.’”

Phoenix and the Promoted Pope

At the same time, a group of people were rewriting Twitter’s entire web client. This project was called Phoenix and later #NewTwitter. It was a single-page application that pioneered a bunch of new techniques, but built without key technologies that were still years in the future. It did client-side routing via the URL fragment; history.pushState/popstate weren’t widely supported until 2013. It had its own hand-written class system (twttr.klass); classes came to JS in 2015. We bundled the application using a bunch of regexes in Ruby; AMD, requirejs, and ES modules did not exist. It was rendered client side to be more responsive and take load off the Ruby application; server-side rendering for JavaScript apps wasn’t a thing (node version 0.2.0 had just been released). Phoenix had components, but they were a mashup of business logic, events, and template rendering via Mustache; React and its concepts were still 3 years away.

What that meant for me was a large, complicated, rapidly changing codebase to figure out without being on the team making the change. The ads and web teams didn’t do a great job collaborating with the fast pace of change and the general ambivalence about the new ads team. One morning we got paged for an incident where the Pope was being run as an advertisement. The problem was a recent change to the ads-related code that we didn’t know about. When someone ran a trend as an ad, it appeared first. Someone on the web team had rewritten the trends module and misinterpreted that rule as “the first trend is an ad.” On that day, the Holy See was in the news so we ended up with a Promoted Pope.

After the Promoted Pope, the ads team insisted on implementing support for ads itself in Phoenix. Among other things, I ended up reimplementing ads in search results. A constant source of friction was that ads were tweets, but they didn’t behave like normal tweets. For example, when you get new tweets in search results, they show up at the top of the screen. But ads stayed at the top of the search results even when new results arrived. I had to hack this behavior into the TweetStream class. With various before and after hooks, when we got new tweets, I checked if there was an ad, grabbed it, and put it back at the top of the stream.

To the Timeline!

Google has made a fortune on ads in search. But people didn’t search Twitter anywhere near as often as they search Google. You have to put the ads where people are, and so the plan was to add them to users’ home timelines. This was controversial, not well liked by most of the employees, and it was my job.

Every engineering discussion I had where I needed help or factual information turned into a mini-referendum on whether we should do ads on Twitter.

“Hey can you help me figure out how to cache these ads—”
“Are we really doing this?”

“Is this the right place to add an RPC to the ad server?”
“To add what?”

Home timelines were Twitter’s top API and the team responsible for their reliability was not enthusiastic about adding an RPC for ads to that code. We negotiated a pretty aggressive timeout—something like 50ms—as a condition for inclusion. I put the value into the service config and had them approve it. Not until many weeks later did I realize I had merged 0.5 which gave us a budget of 500ms, 10x what we had agreed on. “I’ll put up a PR to fix it,” I said. “Ryan, no!” my tech lead said. “Once you have the latency you don’t give it back!” (I gave some of it back.)

It seems incredible to modern readers but at that time, with the choices we had made and the stack we were running, rendering a tweet to JSON was expensive enough that we cached it. For ads, we needed a unique impression ID to track user interactions. So I had to write a “fragment post processor” which would retrieve the rendered tweet JSON from memcached and use string replacement to put in a unique impression ID.

It was a frantic and chaotic time. We were trying to create a business that would keep the company afloat while many other people were working to keep the service online. After I wired up the initial version of ads in home timelines I added buttons for broader employee feedback, and put the branch into the deploy queue. Later that day there was an announcement that deploys were frozen. We were scrambling to move to a new data center and, with the exception of changes required to keep the site online, we would not be deploying any branches. My branch sat in the deploy queue for three months.

The sales team was gathering for an onsite in San Francisco. My manager asked me if I’d give them a demo of ads in the home timeline. I immediately said yes because I love presentations. Only later did I realize that I had no idea how to make the presentation any good. Ads in search results were already a thing; the sales team knew what they looked like; I would be showing them the exact same thing but on a slightly different screen. It felt like a 60 second demo.

When the day arrived, I went to the third-floor cafeteria, stood in front of maybe 50 sales reps, and prepared for a lukewarm reception. I picked a tweet and made it into an ad, and marked it for injection into my timeline. “So you know how we put ads into search results?” I said as I showed them our existing product. I switched to another tab and kind of sheepishly said “well you can probably guess what this is going to look like…” The tweet I picked showed up as an ad. The room went absolutely wild in a way that I was completely unprepared for. In hindsight I realize they were seeing it in a different way than I was; this was the feature they could use to sell billions worth of ads.

Disappearing Tweets

After a variety of trials we were ready to launch, pending one small issue: occasionally the user’s entire home timeline would go completely blank. I saw it once myself. My screen went white and I thought “oh boy” and refreshed my page. As much as you want to find and fix bugs before you launch, there are some that you really wish you didn’t see because you just know it’s going to eat your days. Two voices started competing in my head. Is this bug from my code? It could have been caused by anything. The web client is so complex! Was it even that bad? Fixed with a refresh! contrasted with This only happens when your feature is active. It’s really bad. The screen goes completely blank. Don’t ignore this—it’s going to come back and bite you!. So I did what any responsible junior engineer would do: sincerely hoped it would not happen again.

Of course it happened again. But worse: to other people on my team who had the ads feature flag enabled, and only sporadically. Every now and then someone would say, “Hey, my screen when totally white.” After enough casual reports, I knew that I had to figure out why tweets were disappearing when ads were turned on. I believed then, as I do today, that once you have a repro, a bug is basically fixed. If you can deterministically reproduce the problem, then it is merely a test of your stamina and focus to step through the execution and see where your expectations diverge from reality. In this case, it took me a week just to come up with a repro.

I pored over the code and nothing stood out to me. I’m not sure what I was looking for. A stray call to a function named removeAllTweetsFromTimeline? I talked to people who had encountered the bug and asked what they did. “I got new tweets and the screen went white.” Well, when I got new tweets the screen didn’t go white, so what gives? Based on the observation that people hit the bug when they were getting new tweets, I wrote a script to fetch fake new tweets and display them over and over. Finally, I got it: when we fetched new tweets, we showed a blue “Show new tweets” banner. Once you were shown an ad, if you got new tweets and clicked that banner, then repeated that 8 times, your home timeline would disappear.

With that repro, it still took me three more days to figure out the problem (I suspect that a more senior engineer could have figured this out much faster). The repro script was key because I could immediately drop myself into the problem scenario and step through it, rather than waiting for new tweets eight times.

In the end, the problem was code I had written months before, combined with two behaviors of the Phoenix web client. Sometimes the web client made duplicate or overlapping API requests (it’s hard to keep track of everything that’s in-flight in a long-lived client!). When these requests returned we dumped the tweets we got back into the home timeline anyway. This might have resulted in duplicates but as a belt-and-suspenders approach, at rendering time we skipped duplicates. If two API calls returned tweet 123, you would only see one.

The client also had some safeguards to avoid using too much memory. Sometimes people would leave the app open, polling for hundreds or thousands of new tweets. But no matter how many tweets you received, we only showed you 100 new tweets at a time. If you wanted to see more, you could scroll!

Ads in the home timeline were different from search. In search, the ad stayed on top. In the home timeline, it stayed in place. But I had accidentally left the behavior from search—where we moved the ad back to the top of the timeline—in the code. It was in the TweetStream base class. In the home timeline, whenever you got more tweets, we copied the ad to the top and left it in place. But you never saw that bug because we de-duped the copied ad during rendering.

After you saw an ad and fetched new tweets, there were two ads in your timeline, 1 visible, 1 invisible. After more new tweets you suddenly had 4 ads, 1 visible and 3 invisible. People familiar with computers will see where this exponential growth is going: after getting new tweets 8 times you had 128 ads, 1 visible, 127 invisible. Except that we limit ourselves to showing the first 100 tweets, which were all deduped, resulting in a completely blank screen. Once I understood all this, the bug fix was easy: don’t copy the tweet.

The best part was that this was also a bug in search! We actually had a complaint from advertisers that their screen went blank but we had closed it as cannot reproduce. Nobody but advertisers anxious to see people talking about their promoted trend was sitting on search results refreshing 8+ times. I got to re-open that bug and close it as fixed.

My fix landed in June 2011 and we launched Promoted Tweets in the home timeline a few weeks later. I remember being mostly excited at launch, but it was still something of a mixed bag. I loved Twitter and wasn’t particularly fond of ads; but I also was proud of the work I had done; and also hopeful that Twitter could become a sustainable business that would continue to exist; and also that I would get paid.

Many, but not all, of those things happened. The year before I joined Twitter, the company did $0 in ad revenue. By the time I left we were doing $2 billion a year. The team got much bigger and, although you would not know it from this telling, my overall contribution was quite small. By the time we launched ads in home timelines, there was a large team of folks—Anand, Alan, Andrew, Animitra, Avi, Bailey, Ben, Cara, Ed, Eric, Evelyn, Lennon, Luke, Mark, Niels, Ron, Sara, Srini, Tom, Utkarsh, the entire sales team, and many others—but it was exciting to be there at the start.