Paul's blogpaul.lol

Premature Innovation

The future that isn't

The Mechanical Turk was a chess-playing robot unveiled in 1770. A major leap in technology at the time, the robot featured an articulated upper torso controlled by a large box full of cogs and clockwork and was capable of playing a game of chess against a human opponent.

Just one problem: the Mechanical Turk wasn’t a robot or in any way an automaton, it was a puppet controlled by a guy hiding inside its cabinetry. And yet the Turk tricked, or at least baffled, audiences and experts for almost a century.

It would be uncharitable of us to point and laugh at those rubes of centuries prior. They faced a complicated-seeming machine and skilled operators who carried out this magic trick. Perhaps some experts should’ve known better. But perhaps some experts did know better and failed to convince others, or merely enjoyed the joke too much to spoil the secret.

Instead of pointing and laughing, I’d like to see the Mechanical Turk as a caution: what false innovations are currently touring the globe in the 21st century, fooling rubes and experts alike?

Evenly distributed

The future is already here – it's just not evenly distributed.

This is a quote widely attributed to William Gibson. (Just to help us understand how bad we are at knowing things: nobody can figure out exactly when Gibson said this).

There’s a subtle mistake in this quote that I’ve only recently begun to understand. By mixing the concept of time with the benefits of futuristic technological advancement we begin to think of technological advancement, and a wider distribution of existing technological gains, as inevitable. The passage of time is seemingly constant and unstoppable, and so we transfer this property to technology and we’re surprised when life doesn’t work out this way.

“Do things that don’t scale”

A widely cited anecdote in startup circles is how when Airbnb was just getting started, and struggling to gain any traction, its founders went door to door to meet with and learn from early customers. They also would take nice photos on behalf of early listings to improve the perceived quality of the service.

These are Mechanical Turk-style innovations in a sense. They didn’t build personal relationships, in-depth direct CEO feedback, or free photography into some technological service backed by code and CPUs and databases. They just used people. The fact that Airbnb succeeded is largely credited to techniques like this, but nobody pretends that Airbnb succeeded by “scaling” or “distributing” these unscalable techniques.

The ultimate master of doing things that don’t scale is the US government. The Space Shuttle is a vastly unevenly distributed “future” technology: a “reusable space plane.” For instance, the Space Shuttle’s thermal protection system (TPS) was comprised of 31,000 tiles. There’s a lot of literature on the tiles because they sucked so hard, and a failure of the TPS led to the destruction of the Space Shuttle Columbia. But just one interesting unscalable aspect is how hard they were to install:

Each tile used cement that required 16 hours to cure. After the tile was affixed to the cement, a jack held it in place for another 16 hours. In March 1979 it took each worker 40 hours to install one tile; by using young, efficient college students during the summer the pace sped up to 1.8 tiles per worker per week. Thousands of tiles failed stress tests and had to be replaced. By fall NASA realized that the speed of tiling would determine the launch date. The tiles were so problematic that officials would have switched to any other thermal protection method, but none other existed.

No company, not even Airbnb, would build something as wasteful and impractical as the Space Shuttle, because there’s no theoretical profit that could justify a reusable low Earth orbiter this dangerous, complicated, and inefficient.

And yet during the whole Shuttle era — which overlapped my entire childhood — I sat in awe among the Mechanical Turk’s audience. A reusable spaceplane. How futuristic! We’ve come so far as a species!

Opportunity cost

I first started noticing how many innovations surround us that don’t seem to “scale” after listening to a Scott Horton podcast about the destructiveness of military spending.

The accepted narrative I’ve always heard is how so many technological advancements can be traced back to military spending, and how “unfortunate” it is that it takes major conflicts like World War II to spur the sort of unbridled government spending required for major breakthroughs.

The internet, nukes, GPS, and digital cameras are all examples of military-funded inventions from the last century.

So, what’s the counter-narrative? Well, as Tom Woods and Scott Horton point out, we don’t know the true opportunity cost of all that spending. Military inventions are the "seen," but what is the "unseen" in this case? We can’t re-run this experiment to see which of these (or other) innovations would’ve popped up in the free market.

The government, which has no budgetary constraints, bids up the cost of engineers and can easily afford to corner the market on top talent. Instead of solving problems perceived by civilians via product development, some of the best minds of the last century were milking fat government contracts to solve the challenge of killing people overseas, or in the best case winning a propaganda war with Russia by sending people into space.

Many good ideas developed in these efforts have spread widely to civilians — sometimes, like in the case of encryption, much to the chagrin of the government paymasters. Other ideas didn’t help civilians, or didn’t scale, or didn’t work at all… we don’t hear about those as often. But the failure of those ideas didn’t stop paychecks of the engineers nearly as quickly as in the free market where a company has shareholders to placate. The more wasteful the government’s innovation methodology is, the more in effect it bids up the pay for a competent engineer who could theoretically be building profitable innovations in this counter-factual.

There’s no “we” in innovation

Another piece of media that I can’t stop thinking about in this context is Jonathan Blow’s “Preventing the Collapse of Civilization”.

Jonathan Blow is famous for his rants and curmudgeonliness on the topic of modern software quality, and there’s plenty of that to be enjoyed here. But what struck me in this talk is how challenging it is to pass innovations forward in time, and to spread advanced technique to a wide number of people. ("Technology" is just "technique" turtles all the way down)

One of his anecdotes is from Bob Calwell, an Intel engineer, who found problems in a whole range of vendor chips that hadn’t been there previously. The vendor’s excuse:

The first generation of transistor logic was done by the old gray beard guys that really know what they’re doing. The new generation was done by kids who are straight out of school who didn’t know to ask what the change in packaging would do to inductive spikes.

Does the term "inductive spikes" go over your head? Is it too technical to grasp the actual problem? Yeah, for me too. That’s why there’s no “we” in innovation. Some people know how to do some stuff. And most of the stuff they do relies on _other people knowing how to do other stuff. And any break in the chain has the potential of setting back our civilization’s technological level — potentially for centuries or millennia, like in the case of Roman concrete. Everything that's been invented, and then lost, must be reinvented to be useful again.

It always bugs me when a Peter Thiel or Elon Musk type holds up a phone and complains in the vein of: “We know how to build this, but we don't know how to go to the moon anymore.”

Not that they’re completely wrong, but the “we” is misleading. “We” don’t know shit about how to make phones. Some people, including, I’m sure, Thiel and Musk, know more than others, but not enough by a long shot.

I know how to go to the store and buy a phone. Several people at Apple know how to contract certain manufacturers to build one. Those manufacturers hold oodles of received and developed knowledge on how to do what they do. It’s for similar reasons that we don’t go to the moon anymore: “we” never knew how. Some people knew how, and that knowledge began to fade rapidly as soon as Apollo 17 splashed down.

(Ironically, when looking around for a more precise quote on this topic from these guys I'm straw-manning I found this piece about Peter Thiel's thoughts on innovation. Except the site hosting it is down now, so I had to use archive.org to view it)

Abstraction is dependence

The more complicated a piece of knowledge is, the harder it is to propagate forward in time or broadly to many people. And since our brains are finite, even if we do learn many details of an advanced technology, we never have quite enough room for all details of all abstractions. And so we specialize and rely on others.

In technology, this can lead to what I see as a variant of the “good times create weak men, weak men create bad times, bad times create strong men, strong men create good times” meme.

If we are in the upswing of the cycle we should see strong programmers, forged in fire, creating new frameworks. And I see some glimmers of that right now.

Rust (from Mozilla), Jai (from Jonathan Blow), and Zig (from Andrew Kelley) are all examples of new programming languages forged by strong programmers who were tired of suffering in our current mess. All three languages have a strong relationship or reference to the C programming language while rejecting the modern orthodoxy of C++.

But why is there so much C++ code in the world to react against in the first place? This is the bloated, inefficient code that's to blame for Chrome taking a couple of seconds to close a tab. Or Photoshop taking the same amount of time to launch in 2021 as it took to launch in 2001, despite computer speed increasing exponentially. Mozilla built Rust because some problems in Firefox's C++ codebase were becoming intractable. Jonathan Blow built Jai because he figured if he spent a few years building a programming language for game development he'd still net save time over building games with C++ for the rest of his career.

The proliferation of bad programmers, and bloated, inefficient programs, is "sustainable" because we have a government with no real bottom line, money-printing-backed startups doing things that won't scale, and government-protected monopolies with unassailable market positions.

Monopoly profits breed fragility

Aha, I've written myself into a corner where I get to rant about patents.

Patents are a huge honeypot for paying bad programmers to make bad programs. It doesn't matter how efficiently you build your software, or how maintainable it is into the future. Build the product quickly, get it out the door even if it's a buggy mess, and patent "the hell out of it" as Steve Jobs said about the iPhone (and proved by suing the hell out of Samsung).

Patents incentivize being "the first to invent." It's a high time preference activity. You don't have to publish your code in the patent, just a diagram of your idea and a few implementation details. And now, if the patent office falls for your scheme, men with guns will defend your exclusive right to build that product for the next couple of decades. You can clean up the code "later" (you won't), while you're busying harvesting monopoly profits.

The monopoly profits are highly self-perpetuating as well. Once a company has an edge (Google with its search algorithm patent, Amazon with its 1-Click patent, Apple with its swipe to unlock) you can afford to hire more engineers, and, more importantly, more patent attorneys, to build more bullshit and patent that as well. Even if the patent wouldn't hold up in court, it has a chilling effect of discouraging other companies and inventors even trying to build something similar because they'll likely end up in court for a few years if they do. Funding a rival to a well-established tech giant is such a huge risk that it's rarely attempted.

And so knowledge is consolidated inside huge tech companies, and yet inside these companies, this knowledge is spread thinly over thousands of engineers all pumping out gobs of unmaintainable code. Externally we experience the sensation of "innovation" and "the future" because these companies keep delivering astonishing products and incremental improvements on their flagship devices.

But how many billion dollars would you need to re-create the iPhone?

Think of how thick the vertical tech stack of an iPhone is, and how little of that knowledge is widely known or understood outside of Apple. Or is even knowledge legal to apply.

Apple's chip designs are a couple years ahead of the competition. Apple's operating is largely closed source. It's nearly impossible to run an open source operating system like Linux on Apple hardware, even when Apple doesn't explicitly lock it down. Even if you source the same camera module the iPhone uses, the software to process those images represents hundreds of man-years.

And, of course, there are the "innumerable antecedents," as they're called in I, Pencil, that go into the iPhone. These are technologies that Apple doesn't build, but contracts with others to build. Many of which are only affordable with Apple-level margins. And there are plenty of patents along this chain of suppliers — not only of the parts themselves but also of the processes to build the parts.

In the end, the price tag of building an iPhone de novo might not be in the billions, but in the trillions. In fact, "just" to build a modern chip fab, and the dependencies of a chip fab, and the recursive dependencies of the dependencies, might already put you over the trillion dollar mark.

I believe strongly we wouldn't currently sit at such technological heights — we would not be so deep into the future — if it weren't for Apple's monopoly profits via patents. But who is the "we" in this case? Any choice that Apple makes to exclude a user or use case from an ecosystem, in turn, excludes that user from this "future" entirely.

The future is here, but it's not evenly distributed.

And, importantly, it may never be. How much of what Apple builds is just Space Shuttle tiles? Only made possible by a government-enforced monopoly. How much innovation do we currently enjoy that is built on such coercive grounds?

The future is here, but it's owned and permissioned.

Bitcoin in El Salvador

My original inspiration for writing all this is my unsettled feeling about El Salvador's government-forced adoption of Bitcoin.

I was in the crowd in Miami when Jack Mallers got on stage and announced that Bitcoin would be legal tender in El Salvador. I rose to my feet and applauded. I even felt a little choked up about it. I've become extremely dedicated to the cause of Bitcoin in recent years precisely because of Bitcoin's liberating effect. How could I not cheer to see an impoverished country free itself from the constraints of the US dollar and usher in the Bitcoin era?

Unfortunately, the details of the law were a mixed bag. Legal tender? Yes! Forced legal tender? Hmm, that doesn't sound right. A government-built wallet? Government-sponsored ATMs? A $30 airdrop to every citizen? Government offering exchange services? Sounds like you need a whole lot of government to implement freedom money!

And as a developer, I also felt a little bit of responsibility and embarrassment. In casual conversations before June, I think the consensus on whether the Lightning network was ready for mass adoption was a resounding "no." In fact, at the Bitcoin conference in Miami, all the retailers were set up to receive Lightning payments, but the phone service was slammed and I didn't get a single payment through. It sure didn't feel ready to me! Isn't it a little cruel of us to beta test it on an unwitting populace?

Of course, another part of me hates to look a gift horse in the mouth. Isn't this at least better than no Bitcoin law?

Just to make my doubts look even more unfounded, reports I've seen of the actual rollout in El Salvador have been almost universally positive. The government wallet is rough but functional, the same can be said for the ATMs. People got their $30 airdrop. Pizza Hut and Starbucks accept Bitcoin. Crisis averted, yeah?

But I still think El Salvador fucked up.

Because we can't ever know the counterfactual of a government program. The government can pay any amount of money to force an innovation through. Just like the government can enforce a monopoly claim of any dubious quality. The choice is made, the price is of a little concern, and the "unseen" is exactly that: unseen. We won't know if this whole experiment was Space Shuttle tiles until it's too late.

Are we witnessing an autonomous chess-playing robot? Or is there a dude in the box making fools of us all? Is El Salvador "doing things that don't scale," and what happens when the initial investment runs out?

Actually, on the subject of gift horses

I'll admit, at this point in my premature innovation journey I'm suspicious of anything nice.

Nice website? Probably a shitcoin, or at least heavily VC-backed... so it will probably turn into a shitcoin eventually.

Cool application of machine learning? No way you make enough money on playing Go and chess to justify the compute power and man-hours you're dedicating to this. Oh right, you're funded by Google.

Excellent Bitcoin wallet that "just works" and the user "doesn't even need to think about" channel management? There's probably a big gotcha in there as well.

By the way, I use one of those wallets and I love it. And I'm writing this in the Notes app on a new MacBook Pro. So perhaps I should take the log out of my own eye.

And that's where it gets very hard to apply these fears I have about the precarious position of modern innovation. Because modern innovation is so dang wonderful. I even did a whole podcast episode about this with Matt Odell.

I'm worried that the delta between good, broadly distributable, "sustainable" innovation might be as far as the difference between the open source Precursor device and an iPhone. Or maybe, if we're being charitable, the difference between the PinePhone and a Google Pixel.

Open source mobile hardware isn't even in the black and white TV era yet (the Precursor has yet to ship). And open source, DIY computer processor fabrication a few is Ben Eater videos away from not existing at all.

Most open source software is hosted on GitHub.

Basically, everything that's great and accessible and freedom preserving and humanity empowering and widely distributable in technology is also highly dependent on something that is none of those things.

What to do on the subject of premature innovation

I don't want the perfect to be the enemy of good. And I can only control what is within my power. So, with those maxims in mind, my strategy is pretty simple:

How much un-fragile innovation can I build and embrace?

(And maybe with the caveat that I'd like to avoid regressing my personal technology level a full century)

I'd like to use the illustration of modern, unsustainable technology being like the outlines of a picture, and we have to color in the picture with truly democratic technologies to build it out for real. But who even knows if the picture Apple and DARPA drew for us is the right shape?

But here I am straying from practical application once again.

It starts by refusing to be impressed by a dude in a box pretending to be a robot. Step two might involve me not using Apple's Notes app. And who knows where it ends?