The Open Society and Its Enemies: Summary and Thoughts

Published: 2022-08-15

Tagged: readings essay coordination

In trying to make sense of the world, sometimes the most rewarding questions to pursue are the ones that seem too obvious to ask.

When Russia invaded Ukraine last February, my faith in democracy was shaken. One country decided to start blowing up citizens of another. I expected and waited for an international response that would quickly stop the madness. But no response materialized. What if democracy isn't actually all that good?

So it must have been luck that on my bookshelf I found a copy of Karl Popper's 1945 classic, "The Open Society and Its Enemies." I can't remember how it got there, though I suspect my wife obtained it a few years ago. I dived into the massive tome, hoping it would at least give me some leads to follow up on. However, it not only rekindled my faith in democracy, but also made me look at coordination problems from a new perspective--an experience I would love to share.

So let's start at the beginning, with the titular Open Society.

Open and Closed Societies

Fundamental to Popper's thinking is the dichotomy between open and closed societies.

In open societies, individuals bear the responsibility for their actions. The government does not generally tell its citizens what to do or, more importantly, what to think. People need to figure things out for themselves. That, of course, doesn't mean they can't form groups and follow leaders--it's just that ultimate responsibility rests with each person.

Members of such societies get to choose what job they do, who they hang out with, and what religion they follow. And unless they hurt another person through something like violence or theft, society mostly leaves them alone.

However, this freedom comes with a price:

It is the strain created by the effort which life in an open and partially abstract society continually demands from us-by the endeavour to be rational, to forgo at least some of our emotional social needs, to look after ourselves, and to accept responsibilities.

On a persona level, this "strain" feels like:

There are many people living in a modern society who have no, or extremely few, intimate personal contacts, who live in anonymity and isolation, and consequently in unhappiness.

Closed societies, in contrast, work in a top-down, collective manner. They usually sport some version of The One Truth, whether it be religion or propaganda, and the government punishes noncompliance. This is how pre-modern societies used to work: tribes would often combine man-made with natural law into a form of religion that regulated members' roles and relationships. Thus, breaking the law was seen as a crime against the natural order of things--something that would anger the gods themselves.

A more relevant example, and Popper's main target, are modern totalitarian governments. In these societies, a citizen's only responsibility is to conform to the government-produced Official Truth. Should a citizen seek to expand their responsibility to do their own thing, it would be seen as an attack on society itself. It's the reason why these political systems put dissenters into prison (or disappear them altogether)--they consider thoughtcrime more dangerous than plain old rob-old-ladies crime.

According to Popper, closed societies all share a philosophy: historicism.

Philosophy of Historicism

In a historicist view of the world, societies are influenced by powerful historical trends. Thus, individuals have neither power nor responsibility to effect change. The best they can do is to remove themselves from the path the wheels of history are going.

Being a historicist means being blessed with the ability to predict where history is going. Armed with this knowledge, they can not only assume the safest position, potentially avoiding wars and famines and whatnot, but also help others steer clear of these dangers. Some would go so far as to help remove obstacles from history's path, so that an inevitably better future can arrive sooner.

Popper's book is essentially a critique of this way of thinking. He focuses on two famous historicists--Plato & Marx--and describes how their works have inspired others to build closed societies. These societies, while aiming to create heaven on Earth, instead brought suffering to millions.

Plato as Fundamental Historicist

According to Popper, Plato is the greatest of the early historicists. He not only put down his ideas so that future generations could access them, he did it so eloquently that it inspired some, even thousands of years later, to resume his work on creating utopia.

Plato builds his historicism on the Theory of Forms. This theory says that everything we interact with is merely a poor shadow of a perfect form. What is a perfect form? It's like an abstract template that exists outside of both reality and time. So the computer you're reading this on is merely a flawed instance of some beautiful, amazing idea of a computer that exists in a place which we imperfect humans can't even access.

Now, this is how he applied this theory to society: Since everything has a perfect form, there must exist a perfect state. By definition, this state would be free from strife, suffering, and any discord. Its citizens must be healthy, happy, and rich. This explains why real states, being mere shadows, embody varying degrees of imperfection as indicated by varying amounts of strife, suffering, and discord they have to deal with.

And since these states constantly decay, then the mounting strife, suffering, and discord must ultimately result in each state collapsing into anarchy.

But Plato wasn't content with understanding the past and future. Instead, he did what future historicists would inevitably set out to do: he created blueprints for a perfect state where all would be happy.

Because shadows become worse as time goes on, Plato concluded that earlier ones must be more perfect. So in his search for how to organize a better state, he looked back at earlier, more primitive states for inspiration.

Plato's perfect society would require a strict caste system. At its top would be philosophers, ruling over everything. Beneath them would be warriors, charged with ensuring safety from threats internal and external. Lower yet, there would be merchants, craftsmen, and farmers. At the bottom, of course, would be slaves.

An important feature of this system would be that no one would be allowed to change their station in life. A slave was meant to slave and a farmer was meant to farm. Tradesmen would be allowed to change trades, but they could never become a warrior or philosopher. Plato compared this setup to metallurgy, saying that people were like metals. Mixing metals introduces impurities, creating weaker metals. Thus, members of different castes would be barred from mixing--otherwise, they would produce weak offspring, leading to a weaker state.

Of utmost importance was the need to arrest all change. Only through a policy like this could the process of decay be halted and the state kept safe from anarchy. And of course, the only people capable enough to accomplish this goal were philosophers. That's why Plato put them in charge of his perfect society.

What Plato proposed was in essence a closed society--a society of automatons devoid of responsibility or agency, existing only to execute their rulers' will.

Hegel, the source of all modern historicism

Before Popper undertakes to analyze Marx, he spends two or three chapters to explain how historicism was revived in modern times. According to him, we owe it to an 18th century Germany philosopher, Georg Wilhelm Friedrich Hegel.

Hegel picked up Plato's idea of state-as-organism and combined it with new ideas floating around Europe at the time, the most important of them being Rousseau's concept of "the general will." This concept said that citizens' opinions aggregate into a single "will", which directs society toward the common interest. In Hegel's version, this will is embodied in the person of the state's leader, putting him above and beyond ordinary morality.

A crucial difference that Hegel introduced into Plato's historicism was that shadows, instead of decaying, actually evolve into more perfect versions of themselves. However, this could only happen through a process of self-assertion:

Each must strive to assert and prove himself, and he who has not the nature, the courage, and the general capacity for preserving his independence, must be reduced to servitude.

Most importantly, self-assertion applied not only to individuals, but to states. So the only way for states to evolve toward perfection was to dominate other states--through war. And since states were extensions of their leaders, who in turn were vessels of their peoples' general will, then the noblest pursuit of a citizen was to sacrifice their life for the state.

Despite these differences, Hegel's vision for society is similar to that of Plato: people are merely a means to an end, actually just mere clay meant to be shaped by rulers into something more perfect.

Marx, the greatest historicist

Less than a hundred years after Hegel, another German philosopher went on to create a remix of historicism so potent that we're still dealing with the consequences today. This fellow, named Karl Marx, introduced a seemingly minor change into historicist philosophy that would be nothing less than revolutionary: he shifted the object of historicism from nation to social class.

Marx divided 19th century society into two classes: the bourgeoisie and the proletariat. The first of them was the minority present in all developed nations that owned the means of production and therefore all the power. The latter was the powerless majority that could survive only by selling their labor.

This system--laisazz faire capitalism--dehumanized both groups. The bourgeoisie, nominally living a life free from drudgery, were exploited by the constant need to squeeze the life out of the proletariat. And because capitalism always drove profits down, this group was forever forced to up their exploitation game, lest they fail and have to resort to selling their labor.

In turn, the proletariat was dehumanized by the need to engage in menial, dangerous work for steadily decreasing wages. Because of this, they were prevented from ever exercising and expanding their humanity, which was being ferociously squeezed out of them.

Marx, living in London, was daily exposed to the suffering of the proletariat:

William Wood, 9 years old, was 7 years and 10 months when he began to work... He came to work every day in the week at 6 a.m., and left off about 9 p.m... "Fifteen hours of labour for a child 7 years old!' exclaims an official report of the Children's Employment Commission of 1863. Other children were forced to start work at 4 a.m., or to work throughout the night until 6 a.m., and it was not unusual for children of only six years to be forced to a daily toil of 15 hours.

Popper points out that what animated Marx's work was not so much a hate of capitalism, but rather a deep empathy for fellow human beings, going so far as to state that:

(...) his [Marx's] burning protest against these crimes, which were then tolerated, and sometimes even defended, not only by professional economists but also by churchmen, will secure him forever a place among the liberators of mankind.

From this starting point, Marx went onto construct his historicism thus: since the beginning of time, humans have had to fight against nature to ensure their survival. For most of the time, they had to pursue this goal by arranging societies so that the few enjoyed the fruits of the labor of the many--like in so many kingdoms and empires.

But the industrial revolution changed everything. People found themselves able to produce more than what basic survival required, and so they could finally enjoy leisure. However, some--old aristocrats or new businessmen--found ways of exploiting these changes in a way as to concentrate most of the newfound wealth into their hands. This left those who labored even worse off than before, because their work became mechanical and alienating.

According to Marx, inexorable historical forces would lead to ever-decreasing profits, which would produce ever-decreasing wages, which in turn would lead to ever-increasing suffering of the proletariat. More and more workers would simply die of exhaustion and malnutrition--that is, until the cup of misery overfloweth and the workers declared war against the bourgeoisie. This was a given because while the bourgeoisie depended on the proletariat to exist, the proletariat didn't need the bourgeoisie for anything.

The same historical forces ensured two further things. First, the revolution would be bloody but swift. Marx argued that the bourgeoisie would never, could never agree to giving up control and ownership over the means of production, leaving the proletariat only one option: war. And second, because the destruction of the bourgeoisie and capitalism would be total, the resulting state--communism--would only have one class, creating essentially a classless society.

What would this society look like? There would be no exploitation of anyone by anyone else. Everyone would enjoy the minimum amount of drudgery necessary to survive, with work being fairly distributed among all, ensuring everyone could focus on discovering and expanding their faculties.

Compared to Plato and Hegel, this vision at least sounds pleasant. It's hard to argue with the deeply humanistic roots that forms the foundation of Marx's thought. But we must recognize how thoroughly imbued with historicism it is.

Like other historicists, Marx treats people as motes of dust swept this way and that by the winds of history. The only morally justifiable thing they can do is to move out of the way so that history can run its course. And the faster they do that, the sooner everyone can enjoy the inevitable communist utopia. Aside from depriving regular folks of their dignity and agency, this philosophy rejects any type of political reform--why reform anything when the whole system is fit only for total destruction?

Unfortunately for Marx, reforms proved his historcists theories wrong. In the roughly 150 years since he published his greatest work, Das Kapital, political reforms (often bloody) improved the proletariat's quality of life immensely: health insurance, public education, shorter workday & workweek, etc. What's more, this reformed, regulated capitalism still manages to produce a dizzying surplus of products, enabling workers to enjoy a standard of living above what even the richest could afford even half a century ago.

The inexorable historical forces that lie at the foundation of Marxism turned out to be somewhat less inexorable.

Democracy as Tyranny Prevention System

"The Open Society and Its Enemies" is not just a history of historicism. It's really a vehicle for Popper to champion the ideas that allow open societies to exist.

One such crucial idea has to do with the very definition of democracy:

If we make use of the two labels as suggested, then we can now describe as the principle of a democratic policy, the proposal to create, develop, and protect, political institutions for the avoidance of tyranny. This principle does not imply that we can ever develop institutions of this kind which are faultless or foolproof, or which ensure that the policies adopted by a democratic government will be right or good or wise or even necessarily better or wiser than the policies adopted by a benevolent tyrant. (Since no such assertions are made, the paradox of democracy is avoided.) What may be said, however, to be implied in the adoption of the democratic principle is the conviction that the acceptance of even a bad policy in a democracy (as long as we can work for a peaceful change) is preferable to the submission to a tyranny, however wise or benevolent. Seen in this light, the theory of democracy is not based upon the principle that the majority should rule; rather, the various equalitarian methods of democratic control, such as general elections and representative government, are to be considered as no more than well-tried and, in the presence of a widespread traditional distrust of tyranny, reasonably effective institutional safeguards against tyranny, always open to improvement, and even providing methods for their own improvement.

In other words, democracy is not, as the common and simple definition says, the rule of the people. Rather, it is a set of institutions whose primary objective is to prevent a government from turning into a tyranny.

Looking at democracy this way reveals the crux of Popper's philosophy: the individual is sacred. Because of that, even a badly functioning democracy, since it allocates at least some agency to individuals, is better than a dictatorship, which robs them of any power and responsibility.

Now, the question of why personal agency and responsibility are good is interesting, but so vast that I'll have to leave it for another essay. Instead, let's see why reframing democracy the way Popper did is beneficial.

Popper's definition of democracy enables us to move away from the question of "Who should rule?" to "How is the state to be constituted so that bad rulers can be got rid of without bloodshed, without violence?"

"Who should rule?" has never been truly resolved. Plato thought it should be philosopher kings. Medieval rulers preferred those anointed by God. Hegel: the spirit of the nation as embodied in its benevolent ruler. For Marx, it was the whole of the proletariat. Who's right?

There is no way to answer this question correctly. If we were irrational, we'd probably say something along the lines of "my guy!" because, well, my guy is on my side, so if he ruled, things would be great (sucks for everyone else though). If we were rational, we could probably devise some metrics to optimize for and elect an individual. Except that once we did that, we'd immediately transfer all power to them. So if they turned "bad" in the future, either because our criteria of "good" shifted or they themselves changed, we would be powerless to do anything about it.

Popper describes the second question as "thoroughly practical, almost technical." It invites us to come up with specific solutions, try them out, gather data, and then come up with even better solutions. Iteration is the name of the game, and the criteria for avoiding violence is clear enough to serve as guide, while being broad enough not to block our efforts.

I don't have anything in my notes from "The Open Society and Its Enemies" about what this means in practice, but I found a 1988 The Economist op-ed from Popper himself where he uses the second question to argue that a two-party system is better than a multi-party https://www.economist.com/democracy-in-america/2016/01/31/from-the-archives-the-open-society-and-its-enemies-revisited. I found his line of reasoning quite convincing.

For me, this way of understanding democracy had an electrifying effect.

First, it changed how I think about government. I held a vague belief that government should make its citizens happy. But I now see that if we went down this path, then we would trigger Goodhart's law and end up living in a dystopia like the one from Brave New World or Fahrenheit 451--societies full of happy slaves. What government should do instead is focus on ensuring that citizens are free to pursue happiness however they define it.

Popper spells this out explicitly when he advocates for cautious interventionism, ie. he calls for government to step in when one person harms another. The obvious example of this is having a justice system along with a police force. A less obvious example is creating regulation that prevents common failure modes, such as child labor and other forms of exploitation.

But then he goes farther and says that government should also protect people from wants, meaning, that everyone should have access to food, shelter, education, and medical care. This proposition immediately set off alarm bells in my mind, but then my fears were extinguished when he emphasized the danger of this and suggested a way to reduce it:

But I wish to add here that economic intervention, even the piecemeal methods advocated here, will tend to increase the power of the state. Interventionism is therefore extremely dangerous. This is not a decisive argument against it; state power must always remain a dangerous though necessary evil. But it should be a warning that if we relax our watchfulness, and if we do not strengthen our democratic institutions while giving more power to the state by interventionist planning, then we may lose our freedom. And if freedom is lost, every thing is lost, including 'planning'.

The second effect Popper's definition of democracy had on me was simpler: comparing democratic governments to non-democratic ones is an incorrect, apples-to-oranges way of thinking.

This problem came up for me in the early days of the pandemic, when it looked like countries like China were reacting way more effectively. They were building hospitals, producing PPE, and enforcing strict lockdowns when the US and EU both seemed to be tripping over their legs. Me and my group of friends felt troubled though, because the obvious question to ask was whether it's good to trade individual freedom in return for higher efficiency.

But according to Popper's definition, non-democratic governments are simply bad. It's not even a question of such a government inevitably harming massive numbers of people through misguided policies. Rather, it's about how such a government robs its citizens of responsibility. In a democracy, when harmful policy is deployed, we can't blame voting or the tripartite separation of power--we can only blame the policy as well as those who made it a reality. Then, we can work toward changing that policy. We have skin in the game.

Finally, Popper's definition of democracy made me reconsider how I think of coordination mechanisms. Perhaps this will be obvious to some, but it dawned on me that good mechanisms aren't limited to ones that direct us toward a beneficial outcome--they should also include ones that prevent known failure modes.

Consider, for example, the problem of litter in a park. Directing people toward good outcomes could include putting up posters reminding them that a clean park is a nice park. Alternatively, we could put up posters warning about fines for littering. But we could also set up something to prevent park-goers from failing in the first place: increase the number of trash bins or ban grilling or... well, you get the idea.

Piecemeal engineering

Historicists all seem to call for what Popper labeled "utopian engineering", the practice of planning amazingly good societies that only require changing how humans work on a fundamental level.

Plato's ideal society, based on a caste system, required an empty slate to start with and a group of completely loyal warriors to keep everyone in their assigned roles. Hegel's utopia reminds me of a human anthill, in which every citizen is a disposable, inconsequential part of the whole. Marx's vision seems the most utopian of all because people living in his classless, communist society would somehow be unburdened of emotions such as greed, jealousy, and anger--how else could a classless society remain classless for long?

But Popper criticizes this approach as being inherently flawed and prone to inflicting suffering upon huge swathes of society:

We all feel certain that everybody would be happy in the beautiful, the perfect community of our dreams. And no doubt, there would be heaven on earth if we could all love one another. But, as I have said before (in chapter 9), the attempt to make heaven on earth invariably produces hell. It leads to intolerance. It leads to religious wars, and to the saving of souls through the inquisition.

This feels correct in my experience: I often don't have a clue about what would make me happy. It actually feels like figuring that out is a life-long adventures, because as soon as I achieve some goal, another, more difficult one replaces it--the princess is always in the next castle! I don't think this is the hedonic treadmill though. Rather, it looks like a natural consequence of experience, because the journey to reach each goal changes the traveler, broadening their perspective, making them aware of even more noble goals.

In place of utopian engineering, he offers the alternative of piecemeal engineering:

It is our duty to help those who need our help; but it cannot be our duty to make others happy, since this does depend on us, and since it would only too often mean intruding on the privacy of those towards whom we have such amiable intentions. The political demand for piecemeal (as opposed to Utopian) methods corresponds to the decision that the fight against suffering must be considered a duty, while the right to care for the happiness of others must be considered a privilege confined to the close circle of their friends.

Put another way, piecemeal engineering acknowledges that happiness works differently for different people and so advocates for focusing our efforts on alleviating concrete instances of suffering. This should minimize the harm a government will inflict on its citizens. Popper explains why this last point is true by comparing it to mechanical engineering:

The Utopian engineer will of course claim that mechanical engineers sometimes plan even very complicated machinery as a whole, and that their blueprints may cover, and plan in advance, not only a certain kind of machinery, but even the whole factory which produces this machinery. My reply would be that the mechanical engineer can do all this because he has sufficient experience at his disposal, i.e. theories developed by trial and error. But this means that he can plan because he has made all kinds of mistakes already; or in other words, because he relies on experience which he has gained by applying piecemeal methods. His new machinery is the result of a great many small improvements. He usually has a model first, and only after a great number of piecemeal adjustments to its various parts does he proceed to a stage where he could draw up his final plans for the production. Similarly, his plan for the production of his machine incorporates a great number of experiences, namely, of piecemeal improvements made in older factories. The wholesale or large scale method works only where the piecemeal method has furnished us first with a great number of detailed experiences, and even then only within the realm of these experiences. Few manufacturers would be prepared to proceed to the production of a new engine on the basis of a blueprint alone, even if it were drawn up by the greatest expert, without first making a model and developing' it by little adjustments as far as possible.

I know next to nothing about mechanical engineering, but my experience as a software engineer agrees with Popper's argument: breaking up projects into milestones, milestones into tickets, and tickets into tasks has always helps me in two ways. First, it forces me to think backwards from the end state, which often uncovers dependencies between tasks and, more importantly, gaps in my assumptions. Second, it keeps the blast radius of mistakes smaller, like if a task turns out to take up twice the time, then the change to the project's overall timeline is rather small because the initial estimate for that task was small too.

I don't think Popper states this explicitly, but piecemeal engineering seems like a great fit for a system of government wherein power and responsibility are widely distributed. What I mean is that, in a democracy, you have lots of small, empowered agents executing on lots of small, well-defined projects: coat drives, highway cleanup, selling girl scout cookies, organizing meetups, many kinds of charity, etc.

Sure, from the inside, this looks chaotic and inefficient. One girl scout troop might sell many times more cookies than the other. One LessWrong meetup might be much better organized than another. One city might be run well while another struggles with garbage piling up on its streets. But this limits the blast radius of mistakes to single "cells". More importantly though, this approach allows a lot of people to exercise agency and experiment with novel ways of solving problems--once someone somewhere figures out an effective exercise or martial arts program, then a decade later you can expect to see Crossfit and Brazilian Jiu-Jitsu gyms everywhere.

Strains of Modern Life

In the section on open and closed societies, I brought up Popper's observation that those living in an open society must "forgo at least some of our emotional social needs", which leads to "anonymity and isolation."

This "strain" seems like something we've have been wrestling with for a longer time. The whole Romantic movement of the 18th century looks like a revolt against the explosive growth of open societies. In "Beyond Good and Evil", Nietzsche describes the process of leaving behind the safe, claustrophobic confines of a closed society in order to become an individual capable of confronting their own existence. Marx himself spends considerable amount of time talking about how modern labor alienates the proletariat.

Today, this "strain" is still very much with us, except the players include doomers, ecoterrorists, populists, cyberpunks, rationalists, new age shamans, conspiracy theorists, transhumanists, furries, etc.--basically, everyone working to create meaning for themselves and their group.

It doesn't look like we'll be free of this "strain" any time soon, not even when counting time in human generations. If the trends of the past 20 years continue, each person will become even more specialized; each person will live an even greater part of their life online; and each person will have to deal with an increasingly fragmented and abstract society.

Popper considers this "strain" as the force that drives some of us to embrace historiscist ideologies:

Why do all these social philosophies support the revolt against civilization? And what is the secret of their popularity? Why do they attract and seduce so many intellectuals? I am inclined to think that the reason is that they give expression to a deepfelt dissatisfaction with a world which does not, and cannot, live up to our moral ideals and to our dreams of perfection. The tendency of historicism (and of related views) to support the revolt against civilization may be due to the fact that historicism itself is, largely, a reaction against the strain of our civilization and its demand for personal responsibility

So since we'll collectively experience even more strain, it's safe to assume that historicist ideologies will become more tempting to more people.

And yet, we shouldn't discount the open society. Alongside the great fragmentation, we're also developing increasingly better tools to fight the "strain." Psychotherapy has come a long way both in efficacy as well as availability. We're getting better at building distributed online communities that not only help their members, but also make the world a better place--just look at EA. Some have gone as far as saying that what we're going through today is Enlightenment 2.0, except all the coffeehouses and republics of letters are virtual.

What's amazing about all this is how we've been able to resist our innate programming that steers us toward a closed society. Somehow, we've set our sights on something greater, and have spent thousands of years working toward it. Of course, we can't accept the historicist notion that this is some kind of guarantee that the open society is inevitable. That said, I think it reveals to us the incredible magnitude of a shared yearning for what only an open society can offer us.