Google's Technology is so Expensive it Had to be Created in a Rich Country

Read "Why Google Can’t Stop Tweaking Its Search Engine" by SAUL HANSELL, The New York Times, June 3, 2007. The article is by a reporter given the best access to Google yet. (I quote from the article extensively below.)

Google is the most important node of the World Wide Web! "It is now the most popular Web site in the world, offering its services in 112 languages, indexing tens of billons of Web pages and handling hundreds of millions of queries a day."

A key to Google's success is its system that orders the responses to a query in an attempt to provide the user with the most desired items on the first page of the response. Google has hundreds of engineers, including leading experts in search lured from academia and other firms, working on its search engine — which has many thousands of interlocking equations. "Any of Google’s 10,000 employees can use its 'Buganizer' system to report a search problem, and about 100 times a day they do." Google's closely guarded internal programs, called Debug, "shows how its computers evaluate each query and each Web page".

Google maintaining a giant index of all the world’s Web pages. It "has hundreds of thousands of customized computers scouring the Web to serve that purpose. In its early years, Google built a new index every six to eight weeks. Now it rechecks many pages every few days. And Google does more than simply build an outsized, digital table of contents for the Web. Instead, it actually makes a copy of the entire Internet — every word on every page — that it stores in each of its huge customized data centers so it can comb through the information faster. Google recently developed a new system that can hold far more data and search through it far faster than the company could before."

Google's elaborate system for ranking pages "involves more than 200 types of information, or what Google calls 'signals.' PageRank is but one signal. Some signals are on Web pages — like words, links, images and so on. Some are drawn from the history of how pages have changed over time. Some signals are data patterns uncovered in the trillions of searches that Google has handled over the years."

"Increasingly, Google is using signals that come from its history of what individual users have searched for in the past, in order to offer results that reflect each person’s interests.....Google says it goes out of its way to prevent access to its growing store of individual user preferences and patterns. But the vast breadth and detail of such records is prompting lust among the nosey and fears among privacy advocates."

The article continues:

Once Google corrals its myriad signals, it feeds them into formulas it calls classifiers that try to infer useful information about the type of search, in order to send the user to the most helpful pages. Classifiers can tell, for example, whether someone is searching for a product to buy, or for information about a place, a company or a person. Google recently developed a new classifier to identify names of people who aren’t famous. Another identifies brand names.

These signals and classifiers calculate several key measures of a page’s relevance, including one it calls “topicality” — a measure of how the topic of a page relates to the broad category of the user’s query. A page about President Bush’s speech about Darfur last week at the White House, for example, would rank high in topicality for “Darfur,” less so for “George Bush” and even less for “White House.” Google combines all these measures into a final relevancy score.

Of course, the computing power needed to respond to huge numbers of requests in real time after the complex calculations required for each response is also expensive.

Comment: I have often made the point in this blog that rich nations not only buy more per capita of the ICT things that developing countries buy, (cell phones, radios, televisions, personal computers, etc.), they buy ICT that developing countries can't afford at all. Google has created a search capacity so large and so expensive that it is hard to see how any developing country could have competed. Intel has now begun to transfer its comparably expensive chip manufacturing factories to developing countries, but Intel made the initial investment to create the prototype technology in the United States. So too, Google may transfer its technology to developing nations. Still, the next major innovation that requires huge ICT investment will almost certainly be made in a rich country. JAD