"The Crowd Is Wise (When It’s Focused)"

Robert Bell, foreground, and Chris Volinsky, both of AT&T, are collaborating with other scientists to improve Netflix's personalized film ratings.

Source: STEVE LOHR, The New York Times, July 18, 2009

The article states that:

(A) look at recent cases and new research suggests that open-innovation models succeed only when carefully designed for a particular task and when the incentives are tailored to attract the most effective collaborators.......

The Netflix Prize is a stellar example of crowdsourcing. In October 2006, Netflix, the movie rental company, announced that it would pay $1 million to the contestant who could improve the movie recommendations made by Netflix’s internal software, Cinematch, by at least 10 percent.......

The contest will end next week because a contestant finally surpassed the 10 percent hurdle on June 26, and, according to the rules of the competition, rivals have 30 days from that date to try to beat the leader. The frontrunner is a seven-person team, and its members are statisticians, machine learning experts and computer engineers from the United States, Austria, Canada and Israel. It is led by statisticians at AT&T Research.

Comment: Too often people assume that a good idea is all that is necessary, but it seems that good implementation is usually the key, and good implementation usually involves someone working smart and working hard! JAD

Do Microfinance Projects Reduce Poverty

The Economist this week has an article on the evidence supporting the efficacy of microfinance programs, citing two recent papers:

“The Miracle of Microfinance? Evidence from a Randomised Evaluation” by Abhijit Banerjee, Esther Duflo, Rachel Glennerster and Cynthia Kinnan, May 2009. “Expanding Microenterprise Credit Access: Using Randomized Supply Decisions to Estimate the Impacts in Manila” by Dean Karlan and Jonathan Zinman, July 2009.

The article suggests that there is very little evidence from methodologically sound studies of the effectiveness of these programs. While there are lots of true believers, there are also skeptics as to the utility of the programs.

The two research projects suggest that the microfinance project which they studied did not reduct poverty, although one suggested that in the short term effect was to switch purchases into longer term, more durable materials and The Economist inferred that that might have a long term beneficial impact on poverty.

The Economist article states that

despite growing interest from private investors, 53% of the $11.7 billion that was committed to the microfinance industry in 2008 still came at below-market rates from aid agencies, multilateral banks and other donors.

Of course, if microfinance does not reduce poverty, one should question the allocation of scarce donor funding to microfinance programs. On the other hand, it might be the case that the donor subsidies for loans to poor people are a reasonable approach to humanitarian aid since the loans tend to generate new forms of matching funds. I assume that there is a welfare improvement when people can borrow to make an immediate purchase, paying back the loan later.

The Economist has a touching faith that since microfinance programs trigger investment in microenterprises, they should have an eventual economic benefit. I wonder how much money is lost by incompetent entrepreneurs encouraged by the availability of microfinance to invest in businesses they are incompetent to run well.

What We Don't Know Due to Community Decision Making in Scientific Paradigms

Thomas Bouchard says in an interview in Science magazine:

But we still have whole domains we can't talk about. One of the great dangers in the psychology of individual differences is self-censorship. For example, when I was a student, it was widely accepted that black self-esteem was much lower than white self-esteem, and that was a cause of differences in achievement between the twogroups. Now that's been completely overturned—there is virtually no racial difference in self-esteem. But people had enormous amounts of data [showing this] that they didn't publish because it did not fit the prevailing belief system. How much wasted effort was generated by the flawed self-esteem work as an explanation of the black-white IQ difference? Now a days, I'm sure there are people who are not publishing stuff on sex differences.

Comment: I suspect that he is right in that we fail to have access to important scientific knowledge that its holders fail to publish because it does not fit the prevailing belief system (or that reviewers fail to recommend and journals fail to publish for the same reason). JAD

"An Opportunity We Cannot Waste"

In an editorial in Science magazine Thomas Pickering writes:

The struggles against world poverty are more challenging than ever, given the global financial crisis. At the London G-20 summit in April, leaders of the world's largest economies acknowledged that financial recovery could be sustained only if progress is made in alleviating world poverty. Thus, the path to stable worldwide recovery requires that the issues of economic growth, development, and poverty be seen as linked with the key drivers of food, water, and health, just as climate change is now linked to the key drivers of energy and environment........

The world still looks to the United States for leadership in such work because of our scientific capabilities and our wealth. We are already demonstrating commitment in a few of these areas, notably in the last administration's multibillion dollar program to relieve the impact of HIV/AIDS, principally in Africa. But overall, U.S. foreign assistance programs have been lagging in the food, water, and health areas, with funding for the U.S. Agency for International Development (USAID) declining from $8 billion to $6 billion per year over recent decades. Moreover, the numerous federal agencies that provide science and technology–based international aid suffer from lack of coordination in their efforts. Another National Academies study that I co-chaired concluded that USAID, bolstered by new senior administrators with science and technology expertise, should play a major part in overseeing this much-needed coordination.

Comment: This is an important editorial from a man who has served as U.S. Ambassador to the United Nations and Undersecretary of State.

Re: "1984"

Source: "Amazon Erases Orwell Books From Kindle," BRAD STONE, The New York Times, July 17, 2009.

In George Orwell’s “1984,” government censors erase all traces of news articles embarrassing to Big Brother by sending them down an incineration chute called the “memory hole.”

On Friday, it was “1984” and another Orwell book, “Animal Farm,” that were dropped down the memory hole — by Amazon.com.

Amazon apparently is learning to check for ownership of copyright before selling digital copies.

We are all learning that what we regard as our property may not be our property, simply because we have paid for it. Apparently Amazon not only removed the book from the memories of Kindles owned by its customers, but did so without prior warning, and in the process removed their bookmarks and notes made on the books, which should clearly have been the intellectual property of the authors of those notes. Indeed, I would guess that sufficient annotation, bookmarking and other inputs might constitute a transformation of the original document, and if so might give the Kindle owner/user intellectual property rights to the materials taken from their property by Amazon.

The article also says:

While the copyright on “1984” will not expire until 2044 in the United States, it has already expired in other countries, including Canada, Australia and Russia. Web sites in those countries offer digital copies of the book free to all comers.

The controversy caught my attention in part because I have been involved in a discussion with copyright issues with respect for Zunia, the new portal for international development information which has replaced the Development Gateway "knowledge communities". This whole area of distribution of digital documents is hard to understand, and hard to get on top of.

Interesting tidbit from Mozy -- Where does all the Computer Power Go

To help you further visualize the petabyte: one petabyte is roughly one thousand terabytes, one million gigabytes, or one billion megabytes. That's the space of 10,000 laptops, each with a 100 GB hard drive, on which you could store approximately one of the following:

• 13.3 years of HD-TV video
• 10 million yards of books on a shelf
• 20 million four-drawer filing cabinets filled with text
• 250 million mp3 songs

Mozy currently stores more than 15 petabytes of customer files.

The Hidden Wars

I previously posted on The Hidden War: A Russian Journalist's Account of the Soviet War in Afghanistan by Artyom Borovik. Last night my book club discussed the book.

We are a pretty liberal bunch, but one of the conclusions might be of interest. We all felt that the objectives of the United States in both Iraq and Afghanistan should be stabil societies that did not harbor or support unacceptable numbers of people who would threaten terrorist attacks against the United States, and there was agreement that stability in these societies need not be based on homogeneous nation states with participatory democracy, and indeed that such conditions could hardly be expected in the short term in Iraq, much less Afghanistan. There also was a general perception that the U.S. forces should be drawn down quickly in Iraq and eventually in Afghanistan.

I made the point that we Americans focused too much on the short term military crises and not enough on the long term efforts needed to build a culture of peace. There are a number of failed states in the world today, and if the terrorists are expelled from Afghanistan they can and probably will move to another place where angry people will support them and governments will be powerless to contain them. In the long run we need societies that will not produce nor support terrorists.

Education is probably the strongest weapon in the long term fight against terrorism. Yet as experience with madrassa based promoters of terrorism, schools can be perverted. There has been a huge global effort to promote education over recent decades, and it has been quite effective. Yet it has also failed to achieve its benchmarks and there are still some 75 million kids of primary school age who are not in school. It is the failed and failing states that have the worst record in educating their children.

One of the best instruments that we have for promoting education for all is UNESCO. It has a demonstrated record of promoting education, and is an acceptable source of help in places in which direct U.S. assistance or assistance of former colonial powers would not be accepted. Yet UNESO's education budget is less than fice percent of that of my local school district! And the United States government consistantly militates against increases in that budget.

Here is a startling idea

From the Regents of the University of California at Los Angeles:

(I)t now appears likely the UC system, in this current fiscal crisis, will be ordered by Sacramento to absorb yet another $800-plus million in additional cuts. Its 2009-10 core budget will be reduced by an estimated 20 percent. This will bring the amount of state investment in the University down to $2.4 billion - exactly where it was in real dollars a decade ago.

In the same time frame, by the way, funding for state prisons has more than doubled, from $5 to $11 billion. It’s been reported that, based on current spending trends, California’s prison budget soon will overtake that of the state’s universities and community colleges.

I have degrees from three campuses of the University of California system and it is dear to my heart. (Fortunately I was never eligible for the prison system.) The idea that the state which brought us the movies, much of modern aviation and electronics, and the benefits of Silicon Valley spending more on prisons than on higher education is truly ugly, not to mention frightening, JAD

Research and Development Expenditures as Percentage of GDP

Source: A Global Perspective on Research and Development, UNESCO Institute of Statistics, 2007.

Foreign S&T Graduate Students in the US

According to a new publication of the National Science Foundation:

Graduate S&E enrollment reached 516,199 in 2007 . Of these students, 72% were enrolled full-time, 56% were men, and 71% were U.S. citizens or permanent residents.

Thus there were about 145 thousand foreign graduate students in science and engineering in the United States that year. They help pay for the system, and they staff the laboratories of the graduate universities. Many of them stay when they achieve their degrees, and the external benefits of thousands of scientists and engineers are important to our economy.

"The fewer the competitors, the harder they try"

Source: "Encouraging competitiveness: Psyched out," The Economist, July 9th 2009

The article describes research conducted by two behavioural researchers, Stephen Garcia at the University of Michigan and Avishalom Tor at the University of Haifa in Israel, in which they have demonstrated the likelihood that people put out less competitive effort the larger the group of people against which they are competing.

In their report on the matter in Psychological Science, Dr Garcia and Dr Tor dub their discovery the “n-effect” since “n” represents any numerical value in mathematics. If confirmed, it may mean not only that examination halls should be kept small— or, at least, the same size for all participants so that the playing field is level—but also that other competitive activities should be scaled down for best results.

Comment: I suspect this is a result that it is worth pursuing. For example, in organizations it may be useful to divide competitive classes into relatively small comparison groups to stimulate high performance by the individuals. JAD

The Culture Wars Are Nothing New

When Smithson donated a half million dollars to establish what has become the Smithsonian Institution, it created a major controversy in the Congress. Some opposed accepting the donation as unconstitutional, contrary to states rights. Fortunately John Quincy Adams was in the House of Representatives and fought a successful campaign not only to have the gift accepted but to keep it free from graft and patronage and to dedicate it to a national museum.

The White House reports on Adams' presidency:

Well aware that he would face hostility in Congress, Adams nevertheless proclaimed in his first Annual Message a spectacular national program. He proposed that the Federal Government bring the sections together with a network of highways and canals, and that it develop and conserve the public domain, using funds from the sale of public lands. In 1828, he broke ground for the 185-mile C & 0 Canal.

Adams also urged the United States to take a lead in the development of the arts and sciences through the establishment of a national university, the financing of scientific expeditions, and the erection of an observatory. His critics declared such measures transcended constitutional limitations.

The memories of witnessed events are data, not necessarily facts

There was an interesting segment last night on 60 Minutes dealing with the fragility of memory and the consequent inaccuracies that are found in eye witness testimony in police processes and in court. The program emphasized that police should be trained to understand the perils of planting false memories in witnesses (which can happen inadvertantly) and procedures should be institutionalized to maximize the accuracy of eye witness testimony.

Clearly jurors should be trained to understand that eye witnesses may mistake identifications and helped to interpret the testimony of witnesses reasonably. The program mentioned that the jury process is good at dealing with witnesses that tend to lie under oath, but is not good at dealing with witnesses who are simply wrong in what they remember.

There obviously is information in eye witness testimony. Perhaps judges and jurors should think of that information in terms of "information theory". The trial process is one in which each piece of information changes the probabilities that should be attached to not only the assertion of guilt, but also to a large number of subordinate assertions used in building and refuting a case. A witness who has been subject to processes likely to affect or modify recollection should be considered less informative than one whose memory has been fully protected.

Probably there are some important lessons that should transfer to any area of reports from other people.