Archive for October, 2009

Rethinking Crowdsourcing (beyond Howe)

Monday, October 26th, 2009
crowd

Jeff Howe’s Crowdsourcing Re-Visited
hypios has sometimes been called a crowdsourcing platform. We always felt like this comparison was missing an important point about hypios and crowdsourcing. So we took a closer look at the concept. A major contribution to the discussion surrounding crowdsourcing is Wired editor Jeff Howe’s book Crowdsourcing:  How the Power of the Crowd is Driving the Future of Business (2008). We took his contribution as a starting point for our own thoughts.

Howe’s book seeks to show the internal connection between a dispersed family of phenomena cropping up all over the web.  He urges us to see the profileration of wikis, a threadless.com T-shirt design contest, and the collective search for alien life via spare computer cycles as aspects of a single phenomenon: crowdsourcing.

Now one of Howe’s main examples, in working out the meaning and significance of crowdsourcing, is the (recently-concluded) Netflix Prize, which has spurred a lot of commentary here as elsewhere.  For those who don’t know, in 2006, Netflix offered $1 million to the first solver(s) who could improve their film recommendation algorithm by 10%.   The actual outcome of the Netflix challenge, coming about a year after the book’s publication, invites us to take a more critical stance towards some of Howe’s (widely-accepted) arguments.

What is crowdsourcing?
Crowdsourcing, according to Howe, is an umbrella term for diverse phenomena.  It can involve:

  • Tapping spare time (or procrastination propensity) to rate content.
  • Knowledge-transfer across disciplines or research silos.
  • Innovation through incremental improvement, where the quality of the outcome is indexed to the sheer diversity of the solvers involved. (The Netflix challenge is one example.)

Diversity and crowdsourcing
Howe takes diversity as the key reason crowdsourcing is a superior mode of sourcing. And for him, diversity of solvers’ backgrounds and abilities is more important than anything else.  There are two levels of diversity that Howe sees as relevant to crowdsourcing:

  • Diversity of fields represented.  Here Howe’s reference is primarily the work of Harvard B. Prof Lakhani, who discovered that those most likely to crack a thorny problem were solvers from a field peripheral to the problem’s domain. These peripheral solvers could easily see a solution because, in their field, the problem had already received a standard treatment or solution.  Diversity of solvers was critical for this sort of knowledge transfer.
  • Cases where the solution to the problem does not involve a transfer.  If no solution exists, finding it will require a diverse set of solvers, not just from different fields but also of varying cognitive abilities.  This is probably the boldest of the book’s claims.

Team Mensa vs. Team Brown Socks: Howe’s view
The diverse crowd will almost always beat the team that R&D directors would likely choose themselves.  This is because the latter would always go for the “Mensa team” (the society for people with high IQs)—the team recruited from, say, the top 5% of MIT engineers.  According to Howe, this is because diversity in terms of distinction and ability will yield greater creativity in terms of analytic orientations.  Howe’s amusing image for stocking Team Diversity is randomly rapturing all the profs wearing brown socks from a mid-tier American university faculty lounge.   His Exhibit A for this type of crowdsourcing is the Netflix competition.

What  really happened
The only problem with Howe’s analysis is that it doesn’t describe what actually happened.  From Howe’s argument, it seems like the winning team should be a (relative) band of losers.

Howe’s analysis has suggested that, through collaboration, improvement would be incremental (all about the tweak) and come from the darndest place (e.g. unemployed psychologists), leading to a cinematic triumph of the underdog.  Instead, when you look at the two top teams, it ended up looking a lot more like a battle of the Mensa teams. On the one hand was Pragmatic Chaos, a joint effort of the two previously highest scoring teams from AT&T’s Belkor Labs.  On the other was ENSEMBLE, a mash up of lower-ranked teams. Members of the former write articles theorizing matrix factorization techniques for IEEE, while the latter, judging by its roster, is nothing but a giant geek-agglomeration, stocked full of well-credentialed computer scientists and statiscians.  Not so “brown socks” after all.

Team-building and networking are more important than diversity
So what happened to the diverse crowd? In fact, the way hypios sees it, the right way to think about crowdsourcing is less Howe’s  ‘it takes a village’ than it ‘takes a well-developed network.’  (We’ve already argued for this here and here). Better networking and collaboration tools allow the strongest solvers and teams of solvers to self-identify out of the crowd.  Theories about crowdsourcing have correctly emphasized the fact that a marketplace for ideas is essentially social; however, they tend to over-emphasize the democratizing aspects. Crowdsourcing is an alternative, and superior, method for identifying exactly what Howe calls the MENSA team—highly-qualified and highly-trained specialists—by gleaning them from a much larger crowd.

According to Netflix stats, it took three years, over 44,000 valid submissions (from 180 different countries) and 5,169 different teams to best the CineMatch algorithm by a little more than 10%. Clearly, the most notable trend is that participants chose to collaborate by merging into larger and larger teams. As one PhD solver notes, the contest was all about team “agglomeration” where yesterday’s top losers banded up to tag-team today’s winners.

This capacity for iterative-rounds of team building, rather than the diversity of solvers, is probably the factor to isolate when we think about why various types of crowdsourcing work.  Howe is surely right when he says that crowdsourcing yields a superior, more diverse team.  The team is not superior because of its diversity, though, but because it was assembled in the course of the problem-solving process. This allows the problem itself to determine the competency profile of the team needed to solve it, not some ousider’s ideas about what competencies will be needed.

Crowdsourcing actually identifies the best people
If the Netflix contest actually created a super-mechanism for creating self-selecting Mensa teams, then what Howe is describing looks less like a hierarchy-demolishing, collaborative mode of production and more like the new type of market mechanism that hypios’ platform represents—one that intensifies competition in ideas. Howe himself emphasizes that embracing “crowdsourcing” involves embracing the principles that “the best people are always working for somebody else” and “you don’t know who they are.”  What he doesn’t seem to want to say, however, is that crowdsourcing is in fact a new (and surprisingly direct) means for identifying who and where these best people are.

Debbie Goldgaber

Share this post! add to del.icio.usDigg itStumble It!

Tags: , , , , , , , , , , ,
Posted in Book Review, Controversy, Innovation, Internet Issues, Network & Graph Theory | 3 Comments »

Motivating Creativity: Incentives and Innovation

Thursday, October 22nd, 2009

Dysfunctional Incentives?

Former Al Gore speechwriter turned business-world bestseller, Dan Pink recently gave an excellent 19 minutes talk at TED2009.   (For those of you not familiar with TED, hasten to their website—it offers an extensive video archive of important contemporary thinkers on technology, design and innovation).

In this talk, Pink tells us that most of our ideas about what motivates performance are wrong—or at least not well-adapted to important tasks of our times. His main point: our beliefs about incentives come from attempts to motivate tasks that we no longer do.

The mechanical, rule-following tasks of the 20th century have been replaced by those requiring open-ended, creative thinking.  For creative tasks, contingent incentives (if you do X, you will get Y), have a negative impact on performance.

Correlatively, studies show a negative effect with the addition of a competitive element to a “task environment”.  A contingent-reward structure often involves setting a solver against his or her peers performing similiar tasks (only some of them will get Y for doing X).

Pink highlights two studies which clarify the interdependence of contingent reward and competition.  In one experiment, participants are asked to complete a challenging cognitive task, i.e. to solve it, participants need to operate what can be called a change of a perspective, (“the candle problem”). They are divided into two groups.  One group is told that their work will be used to set a “norm” or performance median.  The other group is told that they will be timed and rewarded according to their performance (relative to their peers).  Not only did the norm-setting group do better at their task than the pay-for-performance group, but the more that was on the line (in terms of reward), the worse participants performed the task.  When the task was slightly changed, implying less creativity and more mechanical behavior (what Pink calls “the candle problem for dummies”) contingent incentives do work.  Study after study appear to corroborate this evidence.

Pink does not spend too much time speculating on why contingent rewards would have this effect.  Instead he uses his time to discuss the lessons we ought to learn from this research: pay incentives don’t work when tasks are cognitively challenging;  people do better in less competitive, more autonomous environments. So whereas payment is the best incentive for mechanical tasks like those performed on mturk, payment alone would not seem the best motivation for more complex tasks like those performed on hypios. There has to be some intrinsic motivation: Solvers have to  feel like solving problems “for their own sake” is rewarding.

I think Pink is largely right.  In fact, the work environment he describes, ROWE (results-only work environment)—which allows for relative work autonomy—is what we’ve adopted here at hypios.  What I’d like to do here is to speculate a little about why solvers might have reacted in the way that they did in the experiments and try to show that we shouldn’t be too quick to draw conclusions about the efficacy of either financial incentives or competition in general.

The Creativity Jam

Pink focuses on what he calls contingent incentives.  Studies that Pink cites show that an incentive tied directly to the outcome of a single (timed) task cause worse performance for cognitive tasks.

Studies have also found that setting specific and difficult goals significantly increases solver performance even when the goals are out of reach.  These same studies have found that, in the context of these specific and difficult goals, adding competition hurts performance when competition comes in the form of measurement against peers in view of reward.  Correlatively, studies have shown that cooperative environments were superior to competive ones for enhancing task performance.

So it appears that competition and contingent reward work together to jam creative thought, at least under conditions where the reward was directly tied to the performance of a specific task.  You might say that together they seem to create a virtual axe over your head, which is counter productive to creative results. Working in an autonomous environment where you set your own specific goals is most conducive to them.  What none of these studies have shown (and they acknowledge this) is what happens when competition and incentives are on the horizon of an activity rather than directly tied to it.

Challenging Pink:  Not all competition is zero-sum

If Pink is right, why do prize competitions—competitions that involve pay for winning performance—like the Netflix prize seem to work so well as creative-solution generators?  Does offering payment for solution (as hypios does) discourage or “de-motivate” solvers?  Can’t a little friendly competition (against a rival institution, say) be an added driver (even if it may not be as important as collaboration with peers)?

Pink is surely right when he says that intrinsic motivation (say, finding a task interesting) is more important than most anything else when it comes to the performance of high-level cognitive tasks.  But it is hard to imagine an intrinsically motivating task withdrawn from a horizon of competition and reward.  What makes it worth doing is exactly what makes it valuable to others.  There will always be rewards (in the form of glory and gold) and competition for any genuinely worthwhile task.  And people often (though by no means exclusively) get clues about what tasks are worthwhile from the array of incentives that surround a given task-environment.  How can we put these sorts of intuitions together with the finding that Pink highlights in his talk?

Just as in economic theory, the price of something is thought of as a signal sending a message to a consumer or producer: financial incentives send messages that something has value for a seeker and even for a society as a whole.  As such they focus attention on certain types of problems, issues, tasks, marking certian ones as valuable.  Indeed, the X Prize foundation, which runs prize competitions with purses exceeding 1o million USD, uses these large sums to counteract what they see as a market failure, to signal attention to a series of goals that they see as undervalorized (by the market) but that they have identified as key for social advanacement.  If there is only one winner, this does not seem to discourage pariticpants nor limit personal investment in pursuit of the prize.

The Ansari X prize, promoting civilian space travel, had dozens of participants investing millions of their own funds in pursuit of the goal.  This is because the competition is not viewed as a zero-sum game.  Competitors gain from the competition in terms of early entry into a promising market, experience and technology exposure.  Here the award money structures the horizon, but is not linked to strategy development nor to the completion of individual tasks.  How a solver gets to the winning solution is set by that solver or team, and not prescribed by a carrot-wielding outsider with a stopwatch.

Now let’s think a little more about competition.  Is collaboration really the only game in town?  I would say (and for reasons that Pink is willing to call “ideological” and “lazy”) collaboration has traditionally gotten short shrift.  But, there’s evidence from studies of  organizational competition that show that organizations that are “neck to neck’” spur innovation.   In the Netflix competition we saw two teams—agglomerations of single players and smaller teams—that were, in the end, competing neck-and-neck.  Neither the competitive element nor the prize money ($1 million) seemed to create a dysfunctional task environment among the leaders.

Let me offer a  reason why.   Netflix created something called a leaderboard, which showed the performance of all pariticpants and thus allowed participants to gauge their performance against their peers in relative “real time.”  In the cited studies (and in many real “task environments”) participants are in the dark about others’ performance, causing them to feel a loss of control and autonomy with respect to their task.

In the Netflix case, the reward was tied to a specific and challenging goal (10% increase in the quality of the algorithm) and contest rules made it so that teams were given time to match or catch up with the winning team.  Basically, this mimics the information that rival firms would get about their competition in a more or less free market.  Now it’s important to not generalize (as I’ve been doing) too quickly between the response of  individuals and teams in a task environment.  This deserves much more study.

Pink points us towards some very important insights into the nature of motivation.  Financial incentives not only cannot replace intrinsic motivation, they can acutally inhibit it.  Intrinsic motivation, which comes from internal values and goals matching external problems and tasks, is decreased when factors of reward and punishment change the task environment and make the participant feel like they are no longer in control of the situation.  With the solver in control, goal, guts, glory and gold all factor into what challenges he or she will take up.  Watch Pink’s video (if you feel motivated to do so) and then let us know your thoughts. We promise, we won’t pay you for it ;-)

Share this post! add to del.icio.usDigg itStumble It!

Tags: , , , , , , , , , ,
Posted in Intellectual Figures, Mechanism Design | 5 Comments »

X-rays, ribosomes, and Dead Sea bacteria: the 2009 Nobel Prize in Chemistry

Thursday, October 15th, 2009
Translation isn’t just how Herta Müller’s books become available to non-Germanophones.  It’s how genetic information is turned into the materials that make life possible.  After the instructions from DNA are transcribed into RNA, ribosomes use sections of this code to assemble the chains of amino acids that fold into proteins.  This year’s Nobel laureates in chemistry all used X-ray crystallography to map the location of each atom in ribosomes, showing how these organelles work at the lowest level.
The project began in the 1970s when Ada Yonath, an Israeli X-ray crystallographer working at the Weizmann Institute, decided to investigate ribosomes using X-ray crystallography.  This process involves crystallizing a specimen, firing X-rays through the crystal, and analyzing the pattern they produce (using the CCD sensors honored by this year’s Nobel Prize in physics).  Yonath’s first challenge, then, was to crystallize ribosomes–a challenge she pursued even when her colleagues “treated [her] like the village fool,” according to a profile in the Jerusalem Post.  She ended up using organelles from the most resilient bacteria she could find, figuring that if they could survive a harsh habitat like hot springs or the Dead Sea, they could stand up to crystallization.
When Yonath finally produced quality crystals in the early 1990s, however, she faced the “phase problem.”  In order to analyze her X-rays, she needed to deduce the phase angle at which the rays hit the sensors, a process that usually involves coating crystals with heavy metals.  The ribosomes could stand up to hot water or intense salinity, but they couldn’t withstand mercury.  Luckily, Thomas Steitz, an American biophysicist and biochemist working at Yale, had the same problem–and decided to look to another field, electron microscopy, for a solution.  By comparing images of ribosomes from electron microscopes with those from the X-ray crystallography, Steitz was able to calculate the phase angle.
Meanwhile, Venkatraman Ramakrishnan, an Indian-American scientist-of-all-trades, was working on the same problem in his US and UK labs.  (He has a PhD in physics, did graduate work in biology and postdoctoral fellowships in chemistry, and spent his Guggenheim fellowship on learning X-ray crystallography.)  Though he tried to model entire ribosomes, he also focused on the problem of ribosomal accuracy.  If the ribosome incorrectly translates information, the proteins it makes will either function incorrectly or not at all, spelling disaster for the cell.  Ramakrishnan discovered that a subunit of the ribosome uses a kind of molecular ruler to double check each protein as it’s manufactured.  If the protein doesn’t measure up, it’s discarded and the ribosome tries again.
This summer, after years of slow, incremental improvements, Yonath, Steitz, and Ramakrishnan finally modeled the location of each atom in the ribosome, publishing their results within weeks of each other.  Their work has implications beyond satisfying scientists’ curiosity about how this organelle works, though.  Over 50% of antibiotics work by blocking bacterial ribosome activity, and these three researchers all produced 3D models of antibiotics attaching to ribosomes in different ways.  If bacteria become resistant to certain antibiotics, scientists may now be able to analyze what’s happening at an atomic level, and then design drugs that will actually work.  Perhaps we won’t face drug-resistant epidemics after all.
We’re sure these discoveries will continue to have an impact, and can’t wait to hear and read the laureates’ lectures in December.  We hope you’ll be inspired by the work of these scientists, and maybe even inspired to make your own contribution–perhaps by solving some problems on hypios.  We can’t promise a medal or a trip to Scandinavia, but we certainly can offer the satisfaction of finding a solution.

Translation isn’t just how Herta Müller’s books become available to non-Germanophones.  It’s how genetic information is turned into the materials that make life possible.  After the instructions from DNA are transcribed into RNA, ribosomes use sections of this code to assemble the chains of amino acids that fold into proteins.  This year’s Nobel laureates in chemistry all used X-ray crystallography to map the location of each atom in ribosomes, showing how these organelles work at an atomic level. (more…)

Tags: , , , , ,
Posted in Innovation, Intellectual Figures | Comments Off

Spurring a communications revolution: the 2009 Nobel Prize in Physics

Wednesday, October 14th, 2009

The 2009 Nobel Prize in Physics goes to three scientists who, working with light, helped start a telecommunications revolution.  It’s split between Charles K. Kao, for his work on fiber optic communications, and the team of Willard S. Boyle and George E. Smith, for their work on digital imaging.  Without Kao’s improvements to communications technology, the data gathered by Boyle and Smith’s device could never have been transmitted around the world. (more…)

Tags: , , , , , , ,
Posted in Innovation, Intellectual Figures | 3 Comments »

From telomeres to cancer vaccines: the 2009 Nobel Prize in Medicine

Tuesday, October 13th, 2009
It’s that time of year again: leaves are falling, the days are getting shorter, and anxious researchers wait by their phones for the call of a lifetime.  That’s right–it’s Nobel Prize season!  This week, hypios takes a look at this year’s winners, who will be awarded their medals, diplomas, and grants in Stockholm and Oslo on December 10th.  We’re sure they’re too busy fielding calls from reporters and working on their Nobel lectures to solve any of our problems–and besides, they’ve already made great contributions to science!  But if you’re still waiting for your call, may we suggest that you spend the next year solving some problems on hypios.
We’re starting off with medicine, the first prize to be announced.  ”Cervical cancer vaccines” like Gardasil and Cervarix have been on the market–and marketed–since 2006, with taglines like “One less!” or “We chose to protect ourselves!”  Millions of women have been vaccinated, and researchers hope to dramatically reduce rates of cervical, vulvar, and anal cancer.  These particular vaccines, however, work by creating immunity against the HPV virus, which can cause certain types of cancer, rather than attacking cancer itself.
Creating a cancer vaccine is an entirely different task–and one that might be possible, thanks to research by this year’s Nobel Prize in Medicine recipients.  Elizabeth Blackburn, Carol Greider, and Jack Szostak are sharing the prize for their work on telomeres and telomerase.  Thanks to their discoveries, researchers have been able to develop and test new cancer treatments, including a cancer vaccine that targets telomerase.  Even if these don’t work or produce unwanted side effects, the possibilities are exciting and “a continued flow of new data is expected,” as this article from the Nobel organization explains.
Chromosome replication is a long-standing problem in biology.  If DNA doesn’t replicate itself exactly when a cell divides, the new cell lacks the necessary information to function, and may not be viable.  In Szostak’s experiments with yeast, cells without telomeres divided poorly and eventually stopped functioning.  Telomeres protect chromosomes from degradation during replication.  They’re small segments of DNA that, positioned on the ends of chromosomes like tiny magnets, attract proteins to form protective caps.  Blackburn, an Australian-born biologist, and Szostak, an American, discovered this after meeting at a conference in 1980 and collaborating on an experiment involving two different organisms.  The fact that telomeres from one organism were effective in another, entirely different species hinted that there was a very basic biological mechanism at work.
In 1984, Blackburn and her Californian then-grad student, Greider, isolated telomerase.  They discovered that this enzyme, which contains both proteins and RNA, actually extends the telomeres so the ends of chromosomes are not snipped off during replication.  The RNA in telomerase serves as a template or platform for the DNA being built.  The more telomerase activity a cell shows, the more genomic stability and viability it has.
Cancer cells have lots of telomerase activity, so their genomes remain intact through multiple divisions.  This means they can divide over and over without degradation–they’ll function just as well after the fifteenth division as on the fifth or the first.  If treatments could target telomerase activity, they could stop tumor growth and treat cancers.  The research of Blackburn, Greider, and Szostak may also be used to develop treatments for inheritable diseases and to investigate stem cell behavior.
Blackburn in particular credits curiosity and an open mind for her success.  As a child in Tasmania, she wallpapered her bedroom with drawings of amino acids, and as a teenager, once used her chemistry set to make explosives, according to a 2009 interview with Clinical Chemistry.  As a doctoral student at Cambridge, she became interested in the molecular side of biology.  There, she learned from biologists like Francis Crick and Max Delbruck, who were former physicists and worked by trying to ‘leap’ to ideas, placing “huge value on the elegant solution.”  This approach inspired Blackburn to move away from the strictly ‘quantitative thinking’ traditionally used to solve problems in biochemistry.  Later, she tried to give her students the same freedom to explore, allowing grad students and post-docs in her lab to work on their own projects rather than doing legwork for her own investigations.  Asked to reflect on her success as a woman in science in the same interview, she said that perhaps men and women think about things differently, but “that’s great because [it's] how you solve problems–you have different minds, different ways of thinking about problems.”
Given our hope that Solvers will collaborate across fields and contribute solutions to problems beyond their specialties, we couldn’t agree more.  Obviously, this attitude has paid off for Blackburn and her colleagues in their scientific investigations.

It’s that time of year again: leaves are falling, the days are getting shorter, and anxious researchers wait by their phones for the call of a lifetime.  That’s right—it’s Nobel Prize season!  This week, hypios takes a look at this year’s winners, who will be awarded their medals, diplomas, and grants in Stockholm and Oslo on December 10th.  We’re sure they’re too busy fielding calls from reporters and working on their Nobel lectures to solve any of our problems—and besides, they’ve already made great contributions to science!  But if you’re still waiting for your call, may we suggest that you spend the next year solving some problems on hypios.

We’re starting off with medicine, the first prize to be announced. (more…)

Tags: , , , , , ,
Posted in Innovation, Intellectual Figures | Comments Off

The Apple and the Bug (Part II) : Fast-track Prototyping

Thursday, October 8th, 2009
apple_bug
Fast-Track Prototyping, the nail in the coffin for the “Apple-Model?”

In the last post, we talked about how the product development life-cycle might look different in the future as a result of certain trends in open innovation. In particular, we discussed a change in the prototyping model that might, someday soon, put a wrench in the Apple-style secrecy machine.

There we mentioned a young company, Bug labs, based in NYC, that makes open source modular components  allowing DIYers to build their own gadgets. The idea, anway, was that they were for DIYers. According to this article in Electronic Design, however, most of its sales are actually going to corporate entities building prototypes or small production runs. “The open nature of BUGbase makes it easy to add a custom module at significantly lower cost and risk than a completely unique prototype” they say.

(more…)

Tags: , , , , ,
Posted in Innovation, Intellectual Property, Open Source Theory, The Economics of Knowledge | Comments Off

The Apple and the Bug: Secret product design vs. open prototyping

Friday, October 2nd, 2009

(First in a two-post series)

apple_bug

Secrecy, Apple-Style

Apple is famous for the secrecy of its product development. It claims not to believe in market research, relying instead on its designers (and executives) to come up with products that will stun and awe the community, beginning with a set of hip early adopters. It has, however, welcomed outside innovation when it comes to its iPhone.

The iPhone Apps store sells thousands of special applications that are developed externally. Not only does Apple pull in tons of revenue (according to Jobs, at least $1 million/day) through the sale of these apps, but some credit the apps and their buzz with iPhone’s competitive advantage. Brad Stone, from the New York Times’ bits blog writes, “The breadth and depth of Apple’s app store is a big reason why the iPhone continues to maintain its lead against up-and-comers like the Palm Pre and the phones running Google’s Android operating system.”

Even so, the process for approval is by all accounts strenuous, even censoring, and there are some who, choosing to bypass it completely, market their apps through third-party sites.  Running these apps requires the iPhone to have been “jailbroken” (downloading a small piece of code).  Apple is vehemently opposed to this, claiming that it strains and breaks the iphone operating system.  Others counter that Apple is just being a control freak and ungrateful to the thousands of app producers that have worked to make the iPhone into a sort of Swiss Army knife of consumer electronic (CE) devices. In short, some outsiders applaud this sort of hacking, claiming that the process of application innovation and end-user specialization should be more open and symbiotic than the Apple-model allows.   (more…)

Tags: , , , , , , , , , , ,
Posted in Controversy, Essay, Innovation, Mechanism Design, Open Source Theory, The Economics of Knowledge | 4 Comments »