The Retrodiction Wall

23 October 2013



Prediction in Science

One of the distinguishing features of science as a system of thought is that it makes testable predictions. The fact that scientific predictions are testable suggests a methodology of testing, and we call the scientific methodology of testing experiment. Hypothesis formation, prediction, experimentation, and resultant modification of the hypothesis (confirmation, disconfirmation, or revision) are all essential elements of the scientific method, which constitutes an escalating spiral of knowledge as the scientific method systematically exposes predictions to experiment and modifies its hypotheses in the light of experimental results, which leads in turn to new predictions.

The escalating spiral of knowledge that science cultivates naturally pushes that knowledge into the future. Sometimes scientific prediction is even formulated in reference to “new facts” or “temporal asymmetries” in order to emphasize that predictions refer to future events that have not yet occurred. In constructing an experiment, we create a few set of facts in the world, and then interpret these facts in the light of our hypothesis. It is this testing of hypotheses by experiment that establishes the concrete relationship of science to the world, and this is also a source of limitation, for experiments are typically designed in order to focus on a single variable and to that end an attempt is made to control for the other variables. (A system of thought that is not limited by the world is not science.)

Alfred North Whitehead captured this artificial feature of scientific experimentation in a clever line that points to the difference between scientific predictions and predictions of a more general character:

“…experiment is nothing else than a mode of cooking the facts for the sake of exemplifying the law. Unfortunately the facts of history, even those of private individual history, are on too large a scale. They surge forward beyond control.”

Alfred North Whitehead, Adventures of Ideas, New York: The Free Press, 1967, Chapter VI, “Foresight,” p. 88

There are limits to prediction, and not only those pointed out by Whitehead. The limits to prediction have been called the prediction wall. Beyond the prediction wall we cannot penetrate.

effective history

The Prediction Wall

John Smart has formulated the idea of a prediction wall in his essay, “Considering the Singularity,” as follows:

With increasing anxiety, many of our best thinkers have seen a looming “Prediction Wall” emerge in recent decades. There is a growing inability of human minds to credibly imagine our onrushing future, a future that must apparently include greater-than-human technological sophistication and intelligence. At the same time, we now admit to living in a present populated by growing numbers of interconnected technological systems that no one human being understands. We have awakened to find ourselves in a world of complex and yet amazingly stable technological systems, erected like vast beehives, systems tended to by large swarms of only partially aware human beings, each of which has only a very limited conceptualization of the new technological environment that we have constructed.

Business leaders face the prediction wall acutely in technologically dependent fields (and what enterprise isn’t technologically dependent these days?), where the ten-year business plans of the 1950’s have been replaced with ten-week (quarterly) plans of the 2000’s, and where planning beyond two years in some fields may often be unwise speculation. But perhaps most astonishingly, we are coming to realize that even our traditional seers, the authors of speculative fiction, have failed us in recent decades. In “Science Fiction Without the Future,” 2001, Judith Berman notes that the vast majority of current efforts in this genre have abandoned both foresighted technological critique and any realistic attempt to portray the hyper-accelerated technological world of fifty years hence. It’s as if many of our best minds are giving up and turning to nostalgia as they see the wall of their own conceptualizing limitations rising before them.

Considering the Singularity: A Coming World of Autonomous Intelligence (A.I.) © 2003 by John Smart (This article may be reproduced for noncommercial purposes if it is copied in its entirety, including this notice.)

I would to suggest that there are at least two prediction walls: synchronic and diachronic. The prediction wall formulated above by John Smart is a diachronic prediction wall: it is the onward-rushing pace of events, one following the other, that eventually defeats our ability to see any recognizable order or structure of the future. The kind of prediction wall to which Whitehead alludes is a synchronic prediction wall, in which it is the outward eddies of events in the complexity of the world’s interactions that make it impossible for us to give a complete account of the consequences of any one action. (Cf. Axes of Historiography)

wyoming dig

Retrodiction and the Historical Sciences

Science does not live by prediction alone. While some philosophers of science have questioned the scientificity of the historical sciences because they could not make testable (and therefore falsifiable) predictions about the future, it is now widely recognized that the historical sciences don’t make predictions, but they do make retrodictions. A retrodiction is a prediction about the past.

The Oxford Dictionary of Philosophy by Simon Blackburn (p. 330) defines retrodiction thusly:

retrodiction The hypothesis that some event happened in the past, as opposed to the prediction that an event will happen in the future. A successful retrodiction could confirm a theory as much as a successful prediction.

I previously wrote about retrodiction in historical sciences, Of What Use is Philosophy of History in Our Time?, The Puppet Always Wins, and Futurism without predictions.

As with predictions, there is also a limit to retrodiction, and this is the retrodiction wall. Beyond the retrodiction wall we cannot penetrate.

I haven’t been thinking about this idea for long enough to fully understand the ramifications of a retrodiction wall, so I’m not yet clear about whether we can distinction diachronic retrodiction and synchronic retrodiction. Or, rather, it would be better to say that the distinction can certainly be made, but that I cannot think of good contrasting examples of the two at the present time.

Albert Einstein Quote mathematics reality

Effective History

We can define a span of accessible history that extends from the retrodiction wall in the past to the prediction wall in the future as what I will call effective history (by analogy with effective computability). Effective history can be defined in a way that is closely parallel to effectively computable functions, because all of effective history can be “reached” from the present by means of finite, recursive historical methods of inquiry.

Effective history is not fixed for all time, but expands and contracts as a function of our knowledge. At present, the retrodiction wall is the Big Bang singularity. If anything preceded the Big Bang singularity we are unable to observe it, because the Big Bang itself effectively obliterates any observable signs of any events prior to itself. (Testable theories have been proposed that suggest the possibility of some observable remnant of events prior to the Big Bang, as in conformal cyclic cosmology, but this must at present be regarded as only an early attempt at such a theory.)

Prior to the advent of scientific historiography as we know it today, the retrodiction wall was fixed at the beginning of the historical period narrowly construed as written history, and at times the retrodiction wall has been quite close to the present. When history experiences one of its periodic dark ages that cuts it off from his historical past, little or nothing may be known of a past that once familiar to everyone in a given society.

The emergence of agrarian-ecclesiastical civilization effectively obliterated human history before itself, in a manner parallel to the Big Bang. We know that there were caves that prehistorical peoples visited generation after generation for time out of mind, over tens of thousands of years — much longer than the entire history of agrarian-ecclesiastical civilization, and yet all of this was forgotten as though it had never happened. This long period of prehistory was entirely lost to human memory, and was not recovered again until scientific historiography discovered it through scientific method and empirical evidence, and not through the preservation of human memory, from which prehistory had been eradicated. And this did not occur until after agrarian-ecclesiastical civilization had lapsed and entirely given way to industrial-technological civilization.

We cannot define the limits of the prediction wall as readily as we can define the limits of the retrodiction wall. Predicting the future in terms of overall history has been more problematic than retrodicting the past, and equally subject to ideological and eschatological distortion. The advent of modern science compartmentalized scientific predictions and made them accurate and dependable — but at the cost of largely severing them from overall history, i.e., human history and the events that shape our lives in meaningful ways. We can make predictions about the carbon cycle and plate tectonics, and we are working hard to be able to make accurate predictions about weather and climate, but, for the most part, our accurate predictions about the future dispositions of the continents do not shape our lives in the near- to mid-term future.

I have previously quoted a famous line from Einstein: “As far as the laws of mathematics refer to reality, they are not certain; and as far as they are certain, they do not refer to reality.” We might paraphrase this Einstein line in regard to the relation of mathematics to the world, and say that as far as scientific laws of nature predict events, these events are irrelevant to human history, and in so far as predicted events are relevant to human beings, scientific laws of nature cannot predict them.


Singularities Past and Future

As the term “singularity” is presently employed — as in the technological singularity — the recognition of a retrodiction wall in the past complementary to the prediction wall in the future provides a literal connection between the historiographical use of “singularity” and the use of the term “singularity” in cosmology and astrophysics.

Theorists of the singularity hypothesis place a “singularity” in the future which constitutes an absolute prediction wall beyond which history is so transformed that nothing beyond it is recognizable to us. This future singularity is not the singularity of astrophysics.

If we recognize the actual Big Bang singularity in the past as the retrodiction wall for cosmology — and hence, by extension, for Big History — then an actual singularity of astrophysics is also at the same time an historical singularity.

. . . . .

I have continued my thoughts on the retrodiction wall in Addendum on the Retrodiction Wall.

. . . . .


. . . . .

Grand Strategy Annex

. . . . .



Bertrand Russell

Bertrand Arthur William Russell (b.1872 – d.1970)

In 1948, shortly after the end of the Second World War and the first use of atomic weapons, Bertrand Russell wrote an essay titled, “The Future of Man”, apparently published in The Atlantic in 1951 (and subsequently collected in Russell’s Unpopular Essays). Russell opened his essay with a sweeping prediction:

Before the end of the present century, unless something quite unforeseeable occurs, one of three possibilities will have been realized. These three are: —

1. The end of human life, perhaps of all life on our planet.

2. A reversion to barbarism after a catastrophic diminution of the population of the globe.

3. A unification of the world under a single government, possessing a monopoly of all the major weapons of war.

I do not pretend to know which of these will happen, or even which is the most likely. What I do contend is that the kind of system to which we have been accustomed cannot possibly continue.

Russell numbered three possibilities for the future, but there is a fourth, which we can call the zeroeth possibility: something quite unforeseeable. Russell left himself an out, but even with the out, I will argue, he got it wrong.

In any case, here are Russell’s four possibilities, which closely correspond to several categories of futurism hotly debated at the present time:

● 0th scenario: unforeseeable developments — this is Russell’s singularity, i.e., the occurrence of an event so discontinuous with previous history that it results in a “prediction wall” that prevents us from seeing or understanding subsequent historical developments.

● 1st scenario: human extinction — following the use of nuclear weapons to end the Second World War, Russell (like Jaspers and other contemporaneous philosophers) was fully aware of anthropogenic existential risks, of which human extinction from nuclear war is a paradigm case, so this is one of Russell’s qualitative risk categories.

● 2nd scenario: global catastrophic failure — Russell identified a two-fold global catastrophic event — drastic diminution of the human population followed by a return to barbarism — which obviously followed from his concern that the next war would be so catastrophic as to end civilization (this is a scenario that also worried Einstein, who famously said that, “I know not with what weapons World War III will be fought, but world War IV will be fought with sticks and stones.”). Whether we consider this a global catastrophic risk, or a form of subsequent ruination, this is another of Russell’s qualitative risk categories.

● 3rd scenario: world government — again like Einstein, Russell was an advocate for world government, and thought it likely the only means by which we could escape our own destruction. In the immediate post-war period, when the US had a nuclear monopoly, Russell actually advocated that the US should use its nuclear monopoly to assert global hegemony and enforce a world government. Later, Russell was to become much more well known for protesting against nuclear weapons, being sharply critical of the Cold War, and writing telegrams to both Khrushchev and Kennedy during the Cuban Missile Crisis.

It seems to me beyond dispute that human life has not come to an end (Russell’s 1st scenario), that human society has not reverted to barbarism after a catastrophic diminution of population (Russell’s 2nd scenario), the world has not been unified under a single government (Russell’s 3rd scenario), and nothing quite unforeseen has happened (Russell’s 0th scenario). It is important to spell this out, being entirely explicit about it, because it is easy to imagine that any or all four of these possibilities might be disputed.

Of the strictly quantifiable predictions, any disputant would really have to tie themselves in knots in order to maintain the human beings have gone extinct or that there has been a catastrophic diminution of population. Only the philosophically desperate would attempt to argue that human life, as we knew it in 1951, has ended forever, or that the seven billion souls alive today somehow do not represent a much larger human population than in 1948. However, I must pause to say this, because there clearly are philosophically desperate disputants who are willing to make claims precisely of this character. But having explicitly acknowledged these strategies of desperation, I will henceforth dismiss them and consider them no further, except in so far as the bear upon the other scenarios.

It could be argued, and it has been argued, that the result of the resolution of the Cold War (which did occur before the end of the century in which Russell was writing) was the installation of US global hegemony as a de facto world government. It has also been argued by conspiracy theorists that there is in fact a world government operating behind the scenes, but not in any public and explicit fashion. It might also be argued that the UN and its associated international agencies (like the International Criminal Court) constitute a nascent world government that will someday coalesce into something more robust and capable of exercising authority. Sometimes these latter theses — government by conspiracy and the UN as world government — are merged together into a single claim.

Even if any or all of these claims are true, none of them have accomplished what was central to Russell’s concern for the future: the abolition of war. Near the end of the same essay Russell wrote:

Owing to the increased productivity of labor, it has become possible to devote a larger percentage of the population to war. If atomic energy were to make production easier, the only effect, as things are, would be to make wars worse, since fewer people would be needed for producing necessaries. Unless we can cope with the problem of abolishing war, there is no reason whatever to rejoice in laborsaving technique, but quite the reverse. On the other hand, if the danger of war were removed, scientific technique could at last be used to promote human happiness. There is no longer any technical reason for the persistence of poverty, even in such densely populated countries as India and China. If war no longer occupied men’s thoughts and energies, we could, within a generation, put an end to all serious poverty throughout the world.

The conspiracy theorists argue that war is part of the plan of subduing the global population, but this isn’t at all the kind of world government that Russell had in mind. When Russell and Einstein wrote about world government in the middle part of the twentieth century, they implicitly had in mind the Weberian conception of sovereignty, i.e., a legal monopoly on violence. Both Russell and Einstein wanted to see a single military power that would beneficently impose its unilateral will upon the world so that we would not see the perpetuation of armed conflict between nation-states.

This did not happen, nor did anything like it happen. On the contrary, the second half of the twentieth century demonstrated the possibility of a state of near-permanent armed conflict as definitive of the world order. In order for this to happen, something did come about, which I have called the devolution of warfare — that is to say, parties to conflicts throughout the world realized that nuclear war could lead to global catastrophic risks, so everyone decided to continue to make war, but to do so without atomic weapons. This way human beings could indulge to the full their love of war and violence without making themselves extinct (and thereby ending the fun for everyone).

This brings us to Russell’s 0th scenario: has the devolution of warfare constituted something quite unforeseeable? Not in my judgment. The devolution of warfare is a negative historical development, involving the suppression or limitation of human agency and capabilities previously demonstrated. The limitation of a demonstrated human capability represents a retrograde development, and I don’t think retrograde developments of this kind rise to the level of constituting a singularity in history.

If anything, the development and use of nuclear weapons constituted an historical singularity, therefore creating a “prediction wall,” so that the deliberate tradition of non-use represents a step back from an historical singularity and a return to predictability. Indeed, what some scholars have called “the return of history” might also be called “the return of predictability” in the sense of being a return to the predictable behavior of nation-states in anarchic competition employing conventional weapons.

It could be argued that what Russell did not see was that at precisely the time he was writing his essay a world order of sorts was being forged, in the post-war agreements on economics at Bretton-Woods and on political matters at Yalta — and, as importantly, if not more importantly, how these explicitly formulated agreements were worked out in practice, sometimes through open warfare, and usually through superpower competition, as in the Berlin Airlift. This de facto world order essentially held throughout the period that Russell considered in his essay — the second half of the twentieth century. Since the actually working out of these agreements in practice was as essential as the agreements themselves, we cannot blame Russell for a lack of prescience in not recognizing in Bretton-Woods and Yalta the foundations of the post-war world. And I don’t think that anything in that war-torn whilst stable post-war world could be said to have fulfilled any of Russell’s predictions.

Now that the post-war world that Russell failed to recognize as it was taking shape has finally become unraveled, we find ourselves once again contemplating the future with great uncertainty, and asking ourselves about the possibilities of radical historical discontinuity (i.e., a singularity), global catastrophic risks, existential risks, and world governance. Dante similarly found himself asking questions of this sort just at the very earliest moment when the scholastic synthesis of the medieval world was beginning to unravel — not only did Dante consider eschatological scenarios that would have constituted a singularity, global catastrophic risks, and existential risks, but also considered world government in his De Monarchia. But Dante was a great poet, and great poets are sensitive souls, and are likely to hear the rumbling on the horizon even when the rest of us are blissfully unaware.

Perhaps whenever the world finds itself at a point of historical transition, grand narratives of transition are contemplated — but in the final analysis (the Hegelian analysis, in which the owl of Minerva takes flight only with the setting of the sun) we usually end up muddling through in the best human tradition, rarely realizing any grand narrative.

. . . . .


. . . . .

Grand Strategy Annex

. . . . .


extreme academe

Today’s Financial Times has an article about the so called Singularity University. The FT gave a rather hyperbolic cover blurb to the story: “Extreme academe: Teraflops and killer robots at the university trying to save the world.” Thus, in the eyes of the Financial Times, this absurd exercise is given the status of a legitimate university. So much for journalistic standards.

The article is written by Simon Daniels, CEO of Moixa Energy, who, he tells us, is one of 40 selected from 1,200 applications. No doubt Daniels views himself as one of those elites who is going to save the world for the rest of us. I’m glad the scholars at the Singularity University enjoy such a high level of self-esteem.

Someone once said, in response to the question as to why corporations hire management consultants, “Because they can.” And why do CEOs and executives go to the Singularity University? Because they can. Despite endemic poverty in the world — and, of course, the poor will always be with us — and despite the recession, there is still a lot of money in the world looking for a place to go. One place for that money to go is into the coffers of the Singularity University. This way Kurzweil and Diamandis get to pal around with even more movers and shakers, and the singularity scholars get to tick off another item on their been there, done that list.

Daniels tells us that the Singularity University is supposed to “educate ‘a cadre of leaders’ about the rapid pace of technology and to address humanity’s grand challenges.” Wonderful. I feel much better already to know that Daniels and his ilk are taking field trips to the National Research Supercomputing Center to see the supercomputer there. I hope they didn’t steal any knobs off the consoles as souvenirs. And, of course, they are engaging in this sacrifice of their valuable time all for my benefit.

I have previously discussed the fallacy of believing that society can educate “leaders” in The Future of Literacy. If anyone thinks that an absurd stunt like the Singularity University is going to educate leaders that will make a difference in the future, then I have a bridge to sell them. And if they have money to burn by attending the Singularity University, I’m sure they can afford a bridge too, and why not?

It is a particular irritant to me when I hear advanced industrialized democracies characterized as being meritocracies. This is simply not true. And it is only in a very loose sense that we can even think of the most advanced industrialized societies of our day as being “democratic.” Sure, we hold mostly clean elections and we can cashier our rulers for misconduct, but we are a long way from anything like a genuine democracy.

Who you are, where you are from, and who you know still has everything to do with how far you get in the world. I wrote about this previously in The Birth Lottery, but it bears repeating because we will understand nothing about how the world works unless we understand that wealth, power, and privilege are almost always kept “in the family” so to speak.

At this point in my rant, someone is sure to point out to me a few cases of self-made men who came up from nearly nothing to be great successes in their industries. Yes, it does happen. But it is rare. Even within the ossified feudal society of the Middle Ages, where social mobility was rare, there were always a few cases of men born into poor or uninfluential circumstances who went on to assume positions of wealth and privilege. Given the fact that our societies today are supposed to be so different from that of the feudal societies of the Middle Ages (and, in fact, they are), it is remarkable how little has changed in terms of social hierarchy.

The exceptions to the rule are exceptions, i.e., outliers. They are played up in the media precisely because they are exceptions: they constitute an amusing human-interest story. The rule remains the rule, and the rule is that the well-connected and the wealthy disproportionately monopolize positions of leadership and influence. These are the kind of people who go (and will go) to the Singularity University, and these are not the people will “save the world” (as though the world needed saving).

We pretend to have democratic institutions, and we pretend to have a meritocracy, but it is mere pretense. The most brilliant individuals might improve their status in the world incrementally, but they will not come into positions of privilege except in the rarest of circumstances. The people who do come into positions of privilege are the merely passably bright members of the already privileged classes. They have the head start, and those coming from behind cannot realistically hope to pass those with the handicap of birth on their side. At best, the truly knowledgeable can hope to become an adviser of those who possess a de facto hereditary claim to power.

What the world really needs is not the Singularity University, but universal (or as near universal as possible) primary education around the world. It is widely disseminated education that reaches millions, and not elite education for forty people at a time, that makes the biggest difference to society in the long run. This is a non-privileged and non-elite approach that involves no CEOs, no field trips to see supercomputers or to the Joint Bioenergy Institute, and no headlines on the Financial Times.

. . . . .

I’ve written about the Singularity University previously in The Singularity Has No Clothes and several subsequent posts.

. . . . .


. . . . .

Grand Strategy Annex

. . . . .

%d bloggers like this: