Tuesday, August 12, 2008

Musing about the epistemology of prediction

Prediction: Foretelling of a future event. Predictions are probabilistic estimates of future occurrences based upon many different estimation methods, including past patterns of occurrence and statistical projections of current data.
Predictions can be made simply as a projection of past trends, under the assumption that the causes of those trends are stable, and the trends will continue. On the one hand, that process saves having to understand the basic causality of the trends, but on the other hand is vulnerable to inaccurate predictions when causal factors change.

An alternate process is to infer causation from history, and to make predictions based on the continuity or modification of the causative factors of the past. There are various problems with this approach. Perhaps most important is that "correlation is not causality", and history does not allow experimental hypothesis testing.

Some factors are ideosyncratic, and sometimes new factors pop up that were not influential in the past.

An example

Francis Fukuyama famously wrote a book titled The End of History at the fall of the Soviet Union. I heard someone say that the Russian incursion into George marked the rebirth of history. Fukuyama apparently believed that it was adherence to Communism as a political ideology that resulted in the division between East and West that formed so much of the matter of history in the late 20th century. The recent comment may suggest that he was wrong.

Russia and China are huge countries, both in terms of geographic area and population, with large supplies of natural resources, and millions of highly educated people. When their centrally planned economic institutions were demonstrated by decades of experience to be less successful than alternatives in the West, both deconstructed those economic institutions. It took longer in Russia than in China, and China started poorer than Russia when Communism fell, but both have now revised economic institutions which are allowing rapid economic growth.

Both countries had not only decades of experience under coercive governments, but long prior histories of coercive central governments. Neither culture responded to the failure of its economic systems by a permanent, radical restructuring of political institutions that created participatory governance systems fully respecting rule of law and individual human rights, as had been hoped by the West.

So the inference that the divide between East and West was a divide between Communism and democratic, market economies may have been wrong, and the divide may have been more due to a divide based on economic competition between different political cultures.

It might have been well to consider some of the broader lessons of history. Regions that had advanced civilizations in the past tended to remain such, in spite of changes in forms of government and economic activity. When you had large adjacent powers, their rivalry tended to remain through changes in regime.

A different example - globalization

Technological knowledge is cumulative. Thus communications and transportation technology have been on a long term path of improvements. Indeed, there seem to be clear indications that steam locomotion technology drove the transportation technology improvement until internal combustion technology. Comparably, there have been generational changes of communication technologies, moving from telegraph, to telephone, from wire to fiber optics and satellite, from electromechanical to vacuum tube to transistor; however, the ultimate impact has been to allow increases in communication speed and reductions in cost.

The improvements in transportation and communications technology have been accompanied by major investments and improvements in transportation and communications infrastructure. As a result, the unit costs of transportation of goods and people have been reduced and the distances over which goods can be transported successfully have increased. So too the improved communications have made it possible to trade in what were once non-tradable services, and to coordinate transactions at a distance more quickly and at more affordable costs.

The economics of comparative advantage informs us that profits are to be made in exchanging goods and services where there are differences in the relative efficiency of their production, and as long as the transaction costs do not overwhelm the advantage of the the trades. Thus profits will be expected to drive wider and wider trading arrangements as transportation and communications infrastructures improve and transaction costs come down.

In fact there was a process of globalization that took place at the end of the 19th and beginning of the 20th century, reversed during World Wars I and II and the Depression, and a process of globalization that took place since World War II. Indeed, rather than two processes of globalization as usually conceived, there may have been only one that was temporarily interrupted for several decades.

We can see globalization then as a trend that is causally dependent on trends in technology and infrastructure, as well as in political factors that determine policies and institutions that militate for or against further globalization.

It is of course difficult to forecast the rate and nature of the improvements of technology and thus the rates and extent in improvement of infrastructure. I would hold, perhaps due to my technological bias, that it is still more difficult to predict political trends, such as those which led to the world wars and the Great Depression. Still, it should be possible to improve over the simple projection of trends by adding technological forecasting and political risk assessment to the mix.

Still another example

The fall of Constantinople to the Turks which interrupted the trade between Europe and Asia led predictably to efforts to find a sea route between Asia and Europe, although one would have to recognize the differences between European and Asian societies that resulted in the Europeans seeking those routes more actively and more successfully than the Asians. Indeed, one might seek to find social and political explanations as to why the Turks preferred to cut the silk route rather than tax the trade along it and benefit economically.

Still, it seems to me that the fact that the Spanish found America and the Portuguese found the route around Africa was fundamentally unpredictable, contingent on personal factors that appear all but random. Moreover, that difference in the exploration patterns affected global patterns in a large number of ways for centuries.

Christopher Columbus sought money for the voyage which ultimately discovered America for years, and sought it from Portugal, France and England as well as Spain. In fact he began his search in with the King of Portugal. When the Spanish monarch finally financed the journey, Spain gained an advantage in the race for American domination and American wealth. Of course, the fact that there was a Spanish Pope who would arbitrate the controversy over priority for domination of newly discovered lands also helped the Spanish. The result, however, of Columbus' success with Queen Isabel was that Spain gained an empire in South and Central America, together with a huge transfer of wealth in the form of silver and gold.

It appears that most people of the time, even expert mariners, thought Columbus was probably wrong, and of course there was no accurate way for them to predict the existence of the American continents.

Even in retrospect it does not seem easy to explain why Spain stepped up first with the resources. This might simply by one of the almost random events which form the contingency of history.

Thinking about predictions from history

Population predictions, such as projections of total population or the urban-rural population distribution seems, at least for a decade or two, to be possible through the projection of recent trends. There are of course refinements, such as use of more or less complex population prediction models, or the use of multiple scenarios to consider the sensitivity of the prediction to important potential variables.

In other situations, such as the prediction of power politics or global economic systems, understanding of underlying trends may improve the results, and even political risk assessment, while perhaps less accurate, may further improve the prediction accuracy.

Still, the third example above suggests that some events are radically determinative of future trends and are themselves quite difficult to predict.

In all cases, the inference of the underlying causal factors from historical evidence will be at best an unproven hypothesis. Those hypotheses are strengthened by theoretical constructs in their support, and by the existence of multiple historical examples of the correlations between factors. Still, correlation is not necessarily causation.

The argument about "the end of history" was intuitive as the result of a century of anti-communist rhetoric, but in retrospect may have been too facile, and one that might have been avoided by a more fundamental examination of the situation,

The impact of Columbus' voyage should lead to some salubrious modesty about our ability to predict the future. Who could or would have predicted the impact of the Columbian Exchange when the existence of the New World had not been suspected.

No comments: