A few years ago, a colleague made what felt like a revelatory point at the time: there is a thin line between innovation and experimenting on people. The more time that passes, the more it feels like there is really no line at all. Innovation is experimentation on people. People that invent new things — particularly digital things — have developed a practice of invention, exposure, evaluation, iteration. It is experimentation with no control group, and formalised consent through end user license agreements.
For instance, Apple just announced a new feature for catching child sexual abuse material (CSAM): they will use the processing power of everyone's phone to evaluate images and see if they find a match against a database of other images containing CSAM. If they do, they report back to law enforcement. When Apple (and others) release features like this they just... do it. The effects of this are hard to predict. Maybe it will make life harder for pedos, maybe it will create a chilling effect on communication between couples or teens, maybe it will lead to other hackers more easily gaining access to parts of our devices... maybe maybe maybe.
But for some reason, innovation feels different than experimentation on people. It feels like an opportunity. An opt-in. Why is that? The difference in perception between the controlled experiment and innovation is one of scale and agency.
Stalin is often cited as having said, one death is a tragedy, and a million deaths is a statistic. Beilby Porteus made the same observation but centring those responsible, saying 'One murder made a villain, millions a hero.'
The former sentiment is helpful for thinking about why harm at the systems level — which is particularly rife in technical systems — is hard to understand and galvanise against. The latter, is helpful for understanding why we adore and reward the innovator. We love the arrogance of people that make grand choices about society — even the choices that lead to a lot of harm. That confidence and macro thinking is the stuff of heroes.
Which brings me to anvils.
When you forge a piece of metal in, you know, Game of Thrones or something, you heat it up, and beat the crap out of it. When we see that, most of us think of the power of the blacksmith and the hammer; how malleable the sword is, and how cool it looks. We rarely think about how painful it would be to be the anvil. When we think about innovation and its discontents, we should talk about the anvil. Who is stuck being smacked over and over again? Who is ignored, even though the enterprise would fail without them? And if the anvil is basically everyone, does it make it any less important to consider?