The rise & fall & rise again of statistics

Ajit Balakrishnan on the surprising drivers of mathematical innovation.

Illustration: Dominic Xavier/Rediff.com.

The other day, in an attempt to lift the mood of three-dozen twenty-somethings from an IIM that I was leading through an online mathematical model-building class, I cheerily announced: "Kids, let's celebrate the 200th anniversary of the Gaussian distribution."

I assumed they would all jump and join my cheering. Instead, I was greeted with puzzled looks and silence.

Finally, one picked up the courage to ask me: "Sir, you mean the Gaussian distribution that we use every day as part of our cutting-edge machine learning and artificial intelligence work is that old?"

"Yes," I said. "Johann Carl Friedrich Gauss was born on April 30, 1777, in the city of Brunswick, Germany, came up with the Gaussian distribution in the prime of his life in the 1820s, and died in 1855."

"Sir, one last question, why on earth would anyone, two hundred years or more ago, think of inventing mathematical methods which we today, in 2021, are struggling to master and use?"

I had to humbly tell them that I would answer that question in our next session, and thereby bought myself time to fully understand what makes or drives anyone to create a new mathematical paradigm?

Is it because they are such geniuses that these thoughts occur to them out of the blue? Or do they think things up in the process of searching for a solution to some more real-world problem?

Gauss's life and work is a good starting point to research that question and when I started digging into things, a subtle light began to shine on this mystery.

In the 1820s, Gauss worked as a scientific adviser to the Hannover government in surveying large geographical areas, the purpose being to draw maps staking out the land areas that belonged to Hannover.

The idea of a nation-State arose with the rise of the modern system of states after the Treaty of Westphalia in 1648. From then on, in Europe, being a nation state depended upon being centrally controlled entities with clearly defined geographic boundaries.

It is in this mapping work -- particularly taking into account the curvature of the earth -- that Gauss had to come up with all those techniques that we today call 'Gaussian'.

Similarly, what led ancient Indians to discover the decimal system, linear prediction models, and algebra?

The not so surprising answer to this is that in India, mathematics was the handmaiden, not of cartography, but of astrology.

Mathematically inclined Indians could make a handsome living by applying their skills to predicting movements of the moon, the sun, the planets, and the stars.

The Vedanga Jyotisa, for example, which contains many of these and similar mathematical techniques, is generally believed to have been put together between the 12th and 14th centuries BC.

The efforts to be able to get the prediction of planetary movements right continued from then on.

What drove such passion for mathematical innovation?

As far as I can tell, the driving force was astrology. From very ancient times, in India, if you acquired a reputation for providing the most accurate prediction for the movements of the moon, the stars and outcomes such as eclipses, the astrological predictions you made were also listened to with awe.

We, Indians, of every economic group, then and now, were always prepared to pay good money to get astrological advice even if the links between planetary movements and astrological prediction (who to marry, when to marry, when to lay a foundation stone for your home among other things) remained suspiciously thin. So, in India, for nearly a thousand years, adeptness in mathematics ensured you a prosperous life!

Still another new dawn for mathematics in India was the rise of economic planning in the 1950-1964 period, commonly called the 'Nehru era'. Mathematical models were created, notably by Prasanta Chandra Mahalanobis, and the branch of mathematics called statistics and the 'Mahalanobis model' dominated Indian intellectual thinking.

All this intellectual fervour about statistics led to the conclusion that the future of rapid economic growth for India lay in very heavy investment by the Indian State in manufacturing machinery to produce steel, chemicals, fertiliser, electricity, transport equipment, etc.

The science of statistics then became the handmaiden of economic planning and folks with an advanced statistics degree could demand virtually any position or compensation.

All these came to an end in the 1980s, when the world, and India, lost faith in the elaborate statistical models of the 'planned economy' and swung to so-called 'laissez-faire' or free-market ideas ('free' being seen as the opposite of 'planned') as preached and practised by the likes of Margaret Thatcher in Britain and Ronald Reagan in the United States.

Statistics as a science took a back seat in the corridors of political power but quickly found a place in stock markets, particularly Wall Street.

Using high-powered computers to construct abstruse models, quantitative traders (called 'Quants') grew so powerful that they quickly controlled about a third of all trading on US stock markets.

The traditional art of stock picking, quizzing a company's management about its operations, felt like a relic of the past. The statistician/mathematician tribe was back in control!

And without me having to point out, in the Internet age, mathematics, and statistics have become even more dominating, based on the clear genius of people like Geoffrey E Hinton (intellectual leader of the artificial intelligence movement) deciding what news you get to read/watch, what products merit your attention to buy online, what marriage partner suits you best, what politician you should vote for, and, most of all, how to drive advertisement revenue upwards!

Let's step back for a moment and ask whether through the ages -- from the time of Shankaracharya to Gauss to Mahalanobis to Geoffrey Hinton -- statistics and mathematics have been essentially a tool for the intellectual class jockeying for power.

As the 21st century opens out, should we find a systematic way for mathematics and statistics -- or, for that matter, all science/technology -- to focus on the social good and not merely chase the eminence and financial well-being of a few?

Ajit Balakrishnan (ajitb@rediffmail.com), founder and CEO, Rediff.com, is an Internet entrepreneur and chaired a committee set up by the ministry of human resource development on education and entrepreneurship to provide inputs for the National Education Policy.