Sea stories are a foundational part of Naval culture. Whenever military folks get together — whether still on active duty or decades removed — sea stories will start flowing. They’ll usually carry valuable info, e.g. all junior officers on subs better know what a “trim party” is before they cause a breach, are inevitably self-deprecating, get style points for danger or malicious compliance with regulations, and serve as a shibboleth to find common ground across specialties, eras, and commands. Like fish stories, they often grow in the telling or with the addition of camaraderie and alcohol.

They’re also a lot of fun to write. They’ve given me excuses to shake the alumni network to check memories and timelines. I plan to keep doing them.

I also realize that not every reader wants to learn about what happens when a nuclear reactor drops a control rod into the core. Or how to properly flood a hallway to conduct carrier landings. So, I’ve added “Sea Stories” as a tag and a section like “How I…” That way you can skip them when you are more interested in telling me why my thoughts on the future of AI and product development are wrong, but find them if you decide Sea Stories are your cup of tea.

Nothing like sitting in an airport after a redeye to get me pondering my history with computers. While I’ll defend Atari’s honor — and am bemused by the random twists and turns that brought me there) — my Atari period was actually quite brief — 6th through 9th grade.

And then, Big Education forced me to start using an Apple. Today Macs are my primary computers, but it was a long path to get there.

1985-1988: I can play games during study hall?

I switched schools between freshman and sophomore years of high school (I don’t recommend it) but I quickly found an unexpected win. In 1985, we had a computer lab full of Apple ][+ and //e’s. Despite the limited graphics capabilities of that era Apple, the games were captivating. Games quickly replaced my previous study hall activity of devouring Clive Cussler, Isaac Asimov, and Larry Niven books, but there was a problem: you weren’t allowed to play games on the lab computers.

Unless you were the monitor.

OK, if it meant getting to play games, I can work for The Man. There were some responsibilities involved, though I pretty much was Ginger in this comic — just s/Ginger/You can play games/. Moreover, monitor time also introduced me to the incredible world of writing viruses — did you know you could rename a file on disk to include invisible characters, making it nearly impossible to reference the file? This inspired a later “you shouldn’t have left your computer logged in” campaign at USNA, but that’s a story for another day.

So, for a few years it was Apple at school, Atari at home. Pascal replaced Basic. But you could feel the PCs coming — a few weeks of a summer program introduced me to 8088 assembly. Change was coming.

1988-1995: PC iterations with some VAX VMS flings on the side

Like a lot of universities of the time, USNA at the end of the ‘80’s was in the midst of an oddly split personality. All the serious computing was on VAX mainframes, various flavors of VMS. Using Ada because…reasons. Probably the most useful aspect of smashing into Ada repeatedly — and having it used as a teaching language — was to instill a healthy fear of Object Oriented languages and innoculation against the weird 1998-2002 idea that C++ was a good idea.

VMS at USNA ended up being as much about PvP — e.g. “write hack to auto recursively log on so only you could submit projects on time” — as operating systems lessons. It did mean I had a blind spot to Unix, which was picking up steam in a lot of computer science departments at the time. It was helpful at Sanders, where CombatDF, COBLU, and other systems were running on VMS Pascal (despite the whole Ada thing).

1995-2003: PCs and embedded C

Then I moved to California to build Armageddon. We were developing on PCs but targeting our custom MIPS R5000 + 3Dfx arcade hardware. C, obviously. Like most embedded efforts, step one was to get the Green Hills compiler working for our particular configuration. Best lesson here: if you are late paying suppliers, they send you defective chips.

Pentium Pros and Windows NT was pretty bleeding edge. It was the only configuration that could approximate our hardware streaming animation system, by precaching the decompressed texture library in RAM and only using 2 rotations of the sprites, rather than the actual system’s five.

It was also my introduction to Visual Studio. Thanks to my Unix blind spot, I’d dodged the whole VI/Emacs debate. There were a bunch of different C editors floating around for embedded — and later console development — but Visual Studio felt like a product from the future.

After Acclaim, it was more PC + MIPS for Nintendo N64 development, this time the R4000 plus the N64’s custom R4000 graphics/audio ASIC. Slower and trickier than our arcade hardware — and needed some subtle multithreading to manage the split use of the same silicon for audio and graphics — but still of a piece.

The first 3 years of Second Life continued the trend, though now it was just PC development and an even better version of Visual Studio.

2004-2007: A resurgent Apple

Second Life was used by tons of creative people, so as Apple returned to caring about performance and games (sort of) we decided we needed Second Life to build for both platforms. Fun times, since this was pre-Intel transition, it meant dealing with Endianness, but otherwise was pretty smooth. It also meant living and working on Macs for the first time and it was revelatory. Compared to the Wintel machines of the time, the Macs — and already the Macbooks — were just head-and-shoulders ahead as development tools, except for specific cases where Visual Studio and C were deeply entwined.

2008-today: Once you go web, you never go back (except for the mobile transition and playing games)

Starting with EMI, I transitioned from an embedded C developer to a web developer. It was so easy. There was documentation, communities, tooling that transformed how we all thought about speed and productivity. Sure, you had to make significant performance sacrifies, but in consumer products — where you’re always developing something new and it’s a continuous quest for product-market fit — the tradeoff was an easy one to make.

The web also convincingly won this period because suddenly everything was Chrome or Chromium. While there was still some browser variance, we suddenly had a global platform to target that felt like it had been built by grownups.

Unsurprisingly, Macs — with their underlying BSDness — dominated this period for development. When node.js came on the scene, it even became possible to do what we’d been doing in games forever — write C for your entire stack — except with Javascript. Iteration on Javascript runtimes followed.

And, yes, there were exceptions, like the mobile transition, but the web has been the primary platform I’ve thought about and developed on. My primary computer use throughout the period has been on Macs, with two exceptions.

1. While working at Google I tried to use ChromeOS as much as possible. ChromeBooks were occasionally spectacular, often frustrating. Excited to see the convergence that many of us argued for finally happening

2. Games. During COVID I returned to playing online, PC games. While I also like consoles — I’ve spent a staggering amount of time getting to NG7 in Elden Ring — I really prefer playing shooters and FPSs with a mouse and keyboard. Can’t say enough good things about the smart folks at Puget Systems.

What’s coming next

Everything that made the web a robust platform is doubly true for AI + the web, but that’s a post for another day.

Bancroft Hall has 4.8 miles of corridors. It’s the largest dormitory in the world, with all sorts of secrets, quirks, and hidden locations. There’s a rifle range (Clancy’s Patriot Games got the location wrong), squash courts, King Hall, Smoke Hall (where you serve restriction and start fires¹), the Rotunda (the center of the picture above), and Memorial Hall (directly through the main entrance in the picture, then up some stairs).

The Top of Bancroft

Everything beyond this point is dumb. If you happen to have access to this location, don’t do what we did. Duh.

Our room was on 2-4, the 4th floor of the 2nd Wing. This meant we had not just recessed dormers (a.k.a. a private deck) but also easy access to the giant, Versailles-inspired copper gutters. These were an excellent way to move between rooms and explore the Bancroft roofline. An early reconnaissance mission was interrupted when we discovered a hatch above Memorial Hall. We’d seen The Goonies, so in we went.

It turned out to be a secondary machinery room for an elevator, with open access to the shaft. What really caught our attention was the ladder bolted to the inside of the shaft, reachable only when the elevator car was gone. Now fully in Die Hard mode, we had to see what was at the top.

An unlocked hatch. Because who would find this room and ladder, let alone climb it?

We did. A lot. The rooftop became our spot—not as famous as the Chapel dome, but it was ours. The discovery also gave us another prize: access to the elevator itself. It ran down the northeast corner of Memorial, Smoke, and King Halls. It was super old-school, rickety, and absolutely off-limits to Midshipmen. So, of course, we used it all the time, particularly for restriction musters. That plan inevitably backfired when the elevator got stuck, and a fire crew had to rescue us, resulting in even more restriction.

We passed our knowledge on to our Plebes. I wonder if 6th Company still knows about it or if one of the many Bancroft renovations “fixed” it.

Footnotes

1. Your uniform gets inspected at multiple formations a day when you are on restriction. Since our uniforms are Navy Blue so you’re always usings rings of reversed masking tape to pull off the lint. Tradition was to ball up the tape and throw it into the large torchiere lamps in Smoke Hall. Which would eventually get hot enough to start smoking. ↩

When I arrived at Lockheed Sanders in 1994, I was unusual: despite being a junior engineer, I had an active Top Secret/SCI clearance. This made me a cheap resource who could be put on our most secret and pricy projects. I was assigned to Combat DF and inherited the bugs the senior people didn’t want.

Almost immediately I received one that cracked me up. It was, roughly:

If you simultaneously push all the buttons above both workstations, the system will crash and reboot

If you imagine a CombatDF station inside a small compartment on a ship, you’ll immediately start wondering what kind of contortions would be needed to hit all of those buttons simultaneously. And who would think to do it? I knew. I knew immediately.

Bored kids on the midwatch. That’s who.

Weaponized boredom

Of the many things nobody tells you about the military is just how much time you will spend completely and totally bored. Part of why ships run drills all the time is that they fill time. Sure, they also contribute to the US Navy’s unprecendented level of training and damage control capability. Sweat in peace so you don’t bleed in war, and all that. But they also chew up time.

Want to run the “fire in missile upper level” drill on an SSBN? That’s nearly an entire 6 hour watch. 209 watch cycles left to go.

Fire in missile upper level is a great chance to truly become one with your Oxygen Breathing Aparatus. OBAs are a closed cycle rebreather that you use when fighting a fire on board a ship or submarine. USS Lafayette was a SSBN, a ballistic missile submarine, so she had 16 Poseiden submarine-launched ballistic missiles in her tubes. Any fire in a submarine is really bad. A fire in missile upper level had the potential to set off the conventional explosives in the W68 warheads, scattering radioactive and highly toxic materials all over the compartment. To practice this, we’d don OBAs, tape over 95% of the mask to simulate smoke, fight the fire, and then conduct decontamination, which we practiced by cleaning up talcum powder without disturbing it, getting in on your gear, or missing any. We all died. A lot.

But when you’re not conducting drills, there’s a high likelihood you’re bored. Your mind wanders and you start thinking — what would happen if I hit all of these buttons? Can I even hit all of them at once?

Fortunately, we’re a tool-using species, so coffee mugs, shoes, straight edges, and pens all become part of the solution. For me, it was the AN/BQR-21 station — more precisely, the two stations — and trying to flip all of them on both panels. It was possible, though it took two of us. We were a hole in the ocean, we were all bored, nothing on the screens but biologics. It didn’t fully crash, but it did glitch things for a bit on both panels. Nothing broken. If we had known how to submit a bug report, we would have.

It was an instantly recognizable bug, and a hilarious one to debug. I’ll leave as an exercise to the reader what kind of VAX Pascal coding decisions would lead to a limit on active buttons.

The risks of boredom stuck with me

The Navy’s approaches to boredom don’t really translate to the Real World (tm), but here’s the thing: engineering orgs have bored times, too. And I don’t mean “let’s all play Counter-Strike”-time. More the moments when smart but idle developers decide “I might need this data some day, so I’ll just set up a separate logging path to a different database” or “I was reading about Cassandra so I decided to add it our metrics system” or “I know A* was working perfectly, but I thought we could rebuild our world model for better monster pathfinding.”

The issue isn’t bright folks trying something new. Those are often spectacular moments. The boredom version has — in my experience — a more insidious problem, because there’s something about boredom that activates a teenager’s hyperrational thinking that will bypass normal thoughts about reliability, safety, and priorities. Worse, as we all know, once we start on weird side projects, the fun of them becomes an ongoing distraction, and we then rationalize both continued work and adding it to the priority list.

Suddenly, we’re risking the worst code you can ever produce: working code that you don’t need. Because working code that you don’t need is a compounding tax you pay forever.

Mission and A’s and O’s

So, I try to catch moments of boredom early and ensure we have a general back pressure against it. A clear mission and priorities helps everyone default to using time effectively. High transparency — even about hacks and side projects — helps prevent wild excursions into unnecessary areas while increasing the moments for knowledge collision and innovation. A’s and O’s and the 1/n rule are two of favorite approaches to this.

Don’t let boredom get weaponized against you.

Bruce and I joined Facebook in November of 2010, right after The Social Network was released. We joined as two of the dozen-ish Engineering Directors reporting to Shrep. Facebook had acquired our startup — Walletin — to bring in senior engineers who were proficient in javascript and client development. Why? Because Facebook was working on two mobile efforts that relied on high-performance javascript in the browser: a mobile version of Facebook’s platform that could run on mobile phones and the Facebook phone.

Farms and Feed

Although Facebook of 2010 was a largely desktop web experience, with a News Feed dominated by Zynga games, the iPhone was rapidly growing and it was clear to everyone that web usage on desktop was heading the way of the dinosaur. On the one hand, this was great for Facebook — Facebook itself is a far superior mobile product than desktop (sure, I have lots of friends who would take laptops to parties to post, but we’re a pretty nerdy group of friends.) It was obviously time to rethink News Feed, since anyone using it could see that an infinite series of nagging “click on my farm now!” notifications wasn’t good for Facebook.

But, Facebook Platform — and in particular, Facebook Game Platform — was valuable and it wasn’t clear how to make it run on mobile.

Hence, a mobile, HTML5 version of platform, aka Faceweb, Spartan; and, the phone project, aka Buffy, Slayer. Both wanted to bring the same social layer present in Facebook Platform to mobile. Both relied on javascript and web technology and needed skills that were oddly rare at Facebook.

This presented…challenges.

PHP

The brilliance of early Facebook — that nobody outside Facebook seemed to ever really understand — was that it was turtles all the way down.

And by turtles, I mean PHP.

In that era of Facebook, everything was PHP and URL. Want to hack on News Feed? feed.php is just sitting there. Photos? photos.php. Want to ship videos in a hackathon? Commit videos.php and show Mark the demo.

More than that, PHP meant anything you wanted to do was just a web push away (assuming you could get past chuckr’s push karma.) Want to ship? It’s a push. Rollback? Push that endpoint. Fix a bug? Push that endpoint.

PHP + Push meant nobody was moving as quickly or as flexibly as Facebook. There’s lots of writing about the Google+ launch lockdown, but what was less obvious was that every time Tech Crunch leaked something about G+ ahead of the launch, we’d start testing versions of the feature on Facebook within hours or days. There wasn’t a moment of doubt in engineering that Facebook would just run circles around G+.

Ironically, Google was the other suitor for Walletin, and I interviewed repeatedly for the Lead G+ role — which made sense coming off of Second Life. Unfortunately, one conversation with G+‘s leader at the time made it obvious he wasn’t actually conducting interviews and intended to Dick Cheney the process. Story for another day.

The lessons of PHP + Push

For all the (justified) attention “Move Fast and Break Things” gets, Mark’s push to shake up developer thinking was deeply tied to the benefits he had experienced learning to code web pages in PHP. Keep iterating. High information density on pages. Equally high conceptual splits across URLs.

Just keep writing code, just keep pushing.

And it worked spectacularly well. On desktop.

Rocks and snakes

So, Bruce and I landed on the game platform team. We released some cool javascript test code based on the Walletin code that abused browsers and got the Chrome team up in arms. Then in early 2011, Schrep tapped me on the shoulder and asked me to check in on the mobile team, maybe look at how their code was going, get a sense of whether things were on track.

I have an extremely high tolerance for ambiguity (Sundar’s pitch was “come to Google, poke around, find a way to be useful”) so that wasn’t a problem.

Well, it was a problem for some of the more tenured Facebookers. I like to make progress updates public and had a particular Facebooker take strong exception to my update of “continuing to kick over rocks, still finding snakes.” Were I doing it today, I’d try to be kinder — after all, it wasn’t this person’s fault that engineers had put on clown pants before coding FaceWeb or that Apple in that era was determined to keep mobile Safari hobbled. Plus, they’re a fancy CTO now.

Yeah, lots of rocks, lots of snakes. More than that, the reviews of our mobile app were continuing to get worse and worse. Facebook Mobile had gone through a few iterations since the iPhone launch, but it was always a separate team, always sort of chasing Facebook desktop. Of course, people wanted a mobile experience. Unfortunately, Faceweb just made things worse. It was slow, buggy, crashy, and of course still required Apple app store approval for releases.

First step, the safety shutdown

When teams get too stressed, tired, and pressured, they lose sight of how they got wedged in the first place. Soon after Schrep put me in charge of the mobile team, we had one of our typically disastrous releases and the team was scrambling to roll changes back and get an emergency version released through Apple. After we accomplished that, I just sent the team home.

I was super clear — you’re all ok, nobody is in trouble, but go home and get some sleep. Next week, we’re going to talk about the state of the app.

Here’s the thing: the team might have been inexperienced with iOS, but they were a brilliant group of Facebookers, and next week they came in with a very clear view we actually had a cultural challenge.

It was best summed up by a lunch discussion I observed between two Ivy League computer science PhDs. One was explaining to the other how they had completely figured out how to bypass Xcode and do everything with vim and command line tools. After a pause, the second engineer asked how they use the debugger. The first responded immediately: “Debugger? I just use lint!”

Coming from an embedded C background myself — arcade in particular was all assembly and C, often starting with a first step of “build a compiler for this chipset” — I couldn’t imagine a world without debugging and had rarely found value in lint. Conversely, programmers who’d maximized the learnings and advantages of scripting web languages — like PHP — approached things differently. Success on iPhone (and Android) was going to require a more compiled language mindset.

Buffy

A few months before, my unclear mandate from Schrep meant that I had found my way into the most secret project at Facebook: the Facebook Phone. A long-gestating project, the phone had started like doomed projects often do, in a too-clever deal (in this case, ATT, Intel, and Facebook.) As anyone who had worked on high performance and power-constrained computing in that era knew, Intel was really struggling and while they were desperate to get a piece of the new smartphone market, they were nowhere near having a viable (let alone competitive) chip. Shockingly, the project was really struggling.

A bright spot was the OS/UI. Like FaceWeb, it was envisioned as browser based — think mobile ChromeOS — which was a much better idea if you had the Android source code to work with. Schrep wanted it to move faster, so I picked up the phone front end in addition to the rest of mobile. It was a super interesting team — mostly ex-WebOS engineers who built similar concepts at Danger and Palm.

Sitting across both, it seemed like it might be a viable alternative to FaceWeb, though the state of mobile Safari was concerning.

The onsite, offsite

I have a playbook when there’s a big, ornery problem that lots of smart people are working on from different directions. It’s a playbook I learned from my friend Viktor at the unparalleled Rueschlikon conference:

1. Identify the key thought leaders with different views on the topic
2. Have them present their ideas as provocateurs, that is, from a strongly opinionated viewpoint and arguing for their position
3. Treat the setting as a critique, namely that all ideas are to be debated robustly while remembering that we strongly respect each other
4. No decision was to be made in the meeting

The topic was what to do about Facebook Mobile?. We constrained the question to iOS to help simplify the decision. We knew Android mattered and was growing, but we also knew that iOS was our priority in our largest markets. Our most engaged audiences were preferring iOS and the bar for quality there was much higher.

We examined three courses of action:

• Stay the course. Just keep digging into FaceWeb, use HTML5 as our path to a great mobile experience
• Leverage Buffy’s UI work. Take the more modern architecture of we were building for Buffy and use that
• Burn it all down, rebuild on native iOS

Most of the senior Facebook product engineering team was there in the room. Schrep attended as well.

It was an obvious, but not easy decision

It wasn’t even close. As anyone who was there can attest, it was obvious that rebuilding on native was the only viable solution. It really only presented two problems:

1. We only had 3 really great, experienced iOS engineers at Facebook
2. Mark had said — and done interviews — that FaceWeb was the future

Fortunately, career limiting decisions are sort of my thing, so first I pitched Schrep and then he told me to pitch Mark. To Mark’s huge credit, he didn’t fight the overwhelming data that we were going down the wrong path. Instead he simply asked how we were going to do it and how we could derisk such a big change.

It was November 2011. I told him he’d have a native News Feed to play with by March.

Git gud

Next problem: why did Facebook — one of the best places to work on the planet, a product-focused culture, and over 500 product engineers — have only 3 iOS engineers? Fortunately, Facebook had a mature, well-documented hiring process and it was easy to go diving through the reject pile.

Turns out, our process was filtering out iOS developers.

Standard Facebook Interview Question: “Design Facebook News Feed, including the backend”

iOS Developer Answer: “I don’t care about the backend”

Interview Notes: “1) TC does not understand basic Computer Science concepts. 2) TC has a bad attitude. Rejection!”

What was baffling to me was that we somehow were hiring Product Infrastructure Developers (Facebook’s second secret weapon after speed — that’s a post for another day), then called User Interface Engineers, led by Nan, and staffed with folks like Lee and tomo. I asked Nan how he managed to hire people like that given our hiring pipelines.

Turned out, he’d created a separate pipeline. Even better, he graciously offered to adapt his pipeline to iOS engineers. We turned our attention to Apple (and Apple alumni) and were able to rapidly start increasing our talent base.

We also knew we’d need to broadly adapt as an engineering culture. Mike had just joined us and immediately brought two key innovations: regular, date-driven releases and broad native training. Facebook Mobile had historically been a lot like high fashion with seasonal releases. At one point, we even found a meme on a public meme site clearly made by an Apple app store reviewer “Facebook Mobile submitted again, wonder if they’ll pass?” Mike convinced me that fixed-date releases — starting at every 2 months, then getting faster and faster, would be the best path to fix our development process.

I agreed and got Mark to agree. Mark had a brief moment of “but I can just release what I want whenever, right?” but rapidly agreed that fixed dates were worth the small risk of a feature he really wanted getting delayed. To his huge credit, he respected that agreement even when we missed something he cared about.

Mike also brought in Big Nerd Ranch and we let anyone — from developers to PMs to recruiters — take iOS and Android training. We put hundreds and hundreds of engineers through it and rapidly expanded awareness of what it meant to deliver compiled code. We also learned that a lot of our Infrastructure engineers were super excited to work on native product development, a wonderful and unexpected resource.

A different product, too

It was time for another career limiting meeting. Mark had been pushing “mobile first” from long before I joined — the stories of “Facebook missed mobile” were always missing the point. Mark saw mobile as early as anyone. Unfortunately, when an organization doesn’t have the capacity or capability to execute on the CEO’s vision, bad things happen. In the case of Facebook, it meant facing the constant cognitive dissonance of being a very product-focused engineering org but shipping a product that hundreds of millions of people relied on but we all knew was buggy and terrible. Worse, when the CEO keeps saying “mobile first” but nothing changes, people start ignoring the CEO.

Mick and I came up with a new framing: “mobile only.” We looked at the curves and any way you looked at them, it was really clear that within a very short period of time, desktop web would be the legacy product, that the vast majority of usage and growth would be on mobile. I then went to Mark and pitched him on the new framing. I also pointed out that because he was a very involved leader (Eye of Sauron was often invoked for what Mark was paying attention to) his lack of modern mobile instincts were slowing us down. If we really wanted to get great at mobile, he needed to really understand the transition. What it meant for development, shipping, design, etc. So I pitched him on weekly Mobile 101 sessions.

Again, to Mark’s credit, he agreed. He took it seriously, made the time, and like anything he throws himself into, within weeks was deep enough to be asking questions that stumped the rest of us. He rapidly made the kind of change only a CEO can make: any design review that didn’t start with mobile mocks was immediately ended. You don’t have to get thrown out of the fishbowl too many times to adjust your behavior and very quickly everything shifted.

Creating the modern feed

Of course, none of this would have mattered if the native code didn’t move forward. Mick and I rethought how News Feed would behave in a performance mobile environment, the team moved quickly, and as promised in March Mark was playing with a News Feed that loaded instantly, you could infinitely scroll as fast as you could swipe, and was stutter-free.

It was amazing.

At the time, we’d been really struggling with the monetization angle. It’s hard to remember, but at this time, News Feed only had social content in it. Ads existed in the sidebar ads section on desktop and in the top bar on mobile. We’d been debating changing the rules with Mark for a while — early experiments with app install ads, for example, showed that we could connect the ad click to the mobile app installation, which seemed pretty valuable! — but Mark had wanted News Feed to remain purely social.

Using the new iOS app and seeing how quickly you could scroll, he changed his mind and in March of 2012 we walked out of a meeting with permission to explore what would become the News Feed — and really the prototype for every attention-reinforced feed on the planet. (My feelings about the impact of that is a story for another day)

When we knew

Within weeks, we knew what we had. Thanks to the asking-for-forgiveness-not-permission of Michael, we had deployed an experimental, paid mobile News Feed product called “Sponsored People You Might Like” that took off like a rocket. App install ads followed, despite an attempt to kill them by a thankfully rare turf war by another long-tenured Facebooker.

I had picked up Boz as a report as he bounced between jobs and after we built out a new thesis on mobile advertising that led to my third career limiting meeting — when I went in and convinced Sheryl that her ads strategy was about to be out of date and she should bet on Boz instead. She did and that worked out pretty well.

Getting the data out to the edge

Things were going remarkably smoothly. We were pushing towards the IPO and Bret was handing all of mobile over to me because he had other ideas post-IPO. There was really only one big challenge left — with a high performance mobile product, how do we turn all of Facebook into an API?

Nowhere captured the PHP + push mentality quite like the notification beepers on mobile. While debugging why the beepers were wrong, we noticed that the javascript for the beepers was receiving and parsing markup, rather than JSON. That’s weird, we thought. Turns out, markup was being stored in MySQL and sent to the mobile client as data. As Schrep would say, not ideal.

So, I did what any reasonable manager would do in this situation. I turned to Lee, Tomo, Schrock, and Jackson and gave them the requirement. Two weeks later, Schrock demoed what would become GraphQL. Not only was it amazing, because of PHP’s flexibility, they’d already figured out a way to infect basically all of Facebook’s systems with it and turn any URL endpoint into an API.

edit on 15 July 2025: Jackson doesn’t remmeber being at the “how the hell do we do this?” conversation, though he was there at Schrock’s first, mind-blowing demo and an early consumer of GraphQL.

We had our data. We briefly had a bug that doubled the load on all of Facebook’s servers — that wasn’t the best day — but we had our solution!

The rest of the way

We had our “botched” IPO and got to watch the stock go down to $17, but we knew we had mobile solved. We released the new iOS app to rave reviews. Mark gave me a bonus and we talked about next steps for me at Facebook. I made the incredibly stupid decision not to commit to staying for at least four more years because 2012 me was pretty stubborn and unhappy. Dumb.

Mark and I did an all hands after launching the new iOS app and acquiring Instagram that showed just how much we’d transformed the quality and engagement. We showed mobile revenue curves. I tried and failed to get a large roomful of people to stand up to in order to recognize that everyone had been part of this transition, but it was true. It’s hard to capture how many people had been part of this change.

(We knew because along with everything else, Mick had a weekly internal post bragging about everyone who helped.)

Android followed quickly after, too. Before long — as predicted — mobile was the majority of usage, growth, and revenue. Every team at Facebook was directly contributing to mobile and I was excited to be able to eliminate the mobile team. You can’t have a specialized team when the whole company is working on something and that final wind down was the endcap on the transition.

In a little over a year, we’d gone from completely broken mobile products, unhappy users, and basically no revenue to the best and largest mobile products in the world. Not too shabby. Maybe you could argue Satya’s transition to Cloud was a bigger, more important pivot. Or Andy Grove’s DRAM to CPU pivot.

But I think what we did was pretty amazing.

But what about Buffy?

That’s a story for another day.

As every person lucky enough to get old(er) realizes, it’s pretty remarkable how quickly time passes. It was pretty close to exactly 25 years ago that my friend Daniel said he’d just met the most interesting founder and was trying to decide if he was nuts or not. I was at Pacific Coast Power and Light, coming off of Road Rash 64 and simultaneously helping build out the foundations for what would become MX 2002 as well as an ultimately cancelled PS2 FPS title we were calling The Banishers.

My then wife and Dan went to college together and we were catching up after an apartment move. He had signed an NDA, so he suggested I try to interview so that we could discuss it.

That evening, I sent my first email to Philip Rosedale.

Come on up

Philip responded immediately with the kind of energy that would make him such a joy to partner with. He said anyone Dan recommended must be pretty good (truth) and suggested I come on up to San Francisco and see what they’re working on.

A few days later I did.

At the time, Linden Lab was in an alley — Linden Street — in San Francisco, between Dark Garden and a body shop. Three stories of light industrial space with advanced for its time camera pointing at you when you rang the bell.

As you entered, there was a big open garage space to your right — the eventual parking disasters that occurred in that space were many — with a conveyor belt running up to the second floor. Steep steps were right in front of you, lined with both whiteboards and posters that at first looked like lame motivational posters (my first thought was “how lame is that!”) until you realized they were from Despair and were absolutely brilliant.

It could be that the purpose of your life is only to serve as a warning to others.

At the top of the stairs was a wall hiding…something, Tessa at the reception desk (likely arguing with some supplier about not paying shipping), and a scattering of desks. Philip immediately walked over and it all began.

The rig

Meeting Philip for the first time is such a vivid memory. The intensity, the engagement as if you are the only person on Earth he wants to talk to, the obvious, palpable brilliance. But what struck me more than any of that was a moment when his wife called and he had to talk her through moving a motorcycle at their condo. I’ve watched many people be impatient in moments like this (ok, I mean me) — instead he brought calm, care, attention, and detail.

How interesting I thought.

After signing my life away, Philip and Andrew gave me the tour of the rig. Strain gauges for low latency. Fixed positions to reduce dissonance with proprioception. A projected view onto a wraparound screen.

It was awesome! I’ve spent a lot of the last 40 years rendering scenes on computers and until some specialty VR hardware at Stanford and Oculus, this was the most amazing interactive visualization I’d experienced. Philip went on to demo the partially working arm — nobody ever really got that working because the hidden challenge of the rig was always that lacking proprioception, you needed tight-loop visual feedback to control it — but I was already hooked.

Because my mind was racing with the possibilities of the world.

A tiled plane of possibilities

Linden Lab didn’t just have the rig, they had also connected four servers and were experimenting with drawing across them. It didn’t work yet, the graphics were barely there, but it was such a different approach to computing and world building than the FPS and MMO architectures I was familiar with.

What if you could use this to make games?

I’d pretty much lost the thread of my mission at this point — that night I would tell Daniel that while Philip might very well be nuts and I was in.

Guess there should be an interview

Then the interview began. And by interview, I mean the coin problem (oh, back when interviews were fun if wildly inaccurate — remind me to cover the boat problem I asked in return sometime) followed by talking about games, technology, the navy, nuclear power, boson fusion, compression, software development, and how we thought about problem solving.

Hours pass

At the end, Philip asked “so, when do you want to start?”

“When can I start?”

Notice, jpeg, and height fields

Walked out the door. We negotiated the next day and then I told Don I was leaving. He didn’t think that was an ideal choice, but he wished me well. During my two weeks, I bought a new PC (with a brand new GeForce 2, finally leaving 3Dfx behind after building a fun connection to them both on Armageddon and subleasing from them during PCP&L’s early days) and wrote a streaming, jpeg compressed renderer for height fields with interactive height deformation.

I walked in the door on my first real day, showed Philip how quickly and easily we could fly around a changing world.

We were off to the races

That code might still be lurking in the open source viewer. It was a quirky jpeg implementation, because experimentally height fields were so much lower frequency than most images that the quantization I chose was atypical.

It was a really important lesson. Go to jobs that you’re writing code on your time off to bring into work with you.