Sign In | Register | User Guide contact us | register for a demo |  
  logo  
 
 

Blog


 
 

Study - Distributed Idea Generation Outperforms Team Brainstorming   (11 / 5)
Tags: crowdsourcing brainstorming innovation management
This has significant managerial implications: if the interactive build-up [of team brainstorming] is not leading to better ideas, an organization might be better off relying on asynchronous idea generation by individuals using, for example, web-based idea management systems.

That quote is from a report by three researchers from the INSEAD and Wharton business schools. They published a study, Idea Generation and the Quality of the Best Idea, that analyzes a mainstay of corporate life: the brainstorming session.

Is it effective in generating quality ideas?

To find out, the researchers conducted a field experiment in which they compared two models of generating ideas:

Team structure: Group works together at the same time together in a room to generate ideas.

Hybrid structure: Individuals generate their ideas independently, then meet together in a group.

Their objective was to determine which of those two structures generated more ideas, ideas of higher quality and is better able to discern the quality of ideas. They found in all cases that the hybrid structure outperformed the team structure.


Extreme Value Theory

The success of idea generation in innovation usually depends on the quality of the best opportunity identified. For most innovation challenges, an organization would prefer 99 bad ideas and 1 outstanding idea to 100 merely good ideas. In the world of innovation, the extremes are what matter, not the average or the norm.

The researchers - Karan Girotra, Christian Terwiesch, Karl T. Ulrich - were interested in determining what methods generate the best ideas. They distinguish their approach from previous research which analyzed the quantity or average quality of ideas generated.

They use extreme value theory to understand the factors impacting the quality of ideas. Extreme value theory shows that the maximum value of an idea from a set of ideas is based on:

  1. The sheer volume of ideas generated
  2. Average quality of all ideas generated
  3. The level of variance in the quality of generated ideas

These concepts are put together nicely in this graphic:

extreme value theory - impact on innovation

 

Once you understand this framework for innovation, it becomes a matter of maximizing the values for each component. Watching, of course, for correlative impacts between them.


Field Research Experiment

The three researchers conducted an exhaustive experiment to determine which of the two methods - team structure or hybrid structure - generated the highest quality ideas at the top end of the scale. Here is the summary of their experiment.

Subjects: 44 juniors, seniors and grad students at the University of Pennsylvania

Challenges: They generated 443 ideas around two challenges.

  • You have been retained by a manufacturer of sports and fitness products to identify new product concepts for the student market. The manufacturer is interested in any product that might be sold to students in a sporting goods retailer.
  • You have been retained by a manufacturer of dorm and apartment products to identify new product concepts for the student market. The manufacturer is interested in any product that might be sold to students in a home-products retailer.

Idea generation formats: Subjects were split into four clusters. Half the clusters did the team structure first, half did the hybrid structure first. The clusters then switched structures for the different ideation challenges.

Idea quality: The quality of the ideas was assessed in two ways.

  1. Business value: Panel of 41 Wharton MBA students each assessed the business value of the ideas on a 1 - 10 scale
  2. Purchase intent: Panel of 88 college students (the target market for the ideas) each assessed their own likelihood of buying a given product proposal on a 1 - 10 scale

Experiment format: Subjects conducted idea generation exercises as follows.

  • Team structure: 30 minutes together in a room to generate ideas together. Then 5 minutes of assessing and selecting the best 5 ideas.
  • Hybrid structure: 10 minutes of generating ideas on their own. Then 20 minutes of discussing these and new ideas. Finally, 5 minutes of assessing and selecting the best 5 ideas.


Results: Hybrid Structure Tops Team Brainstorming

The results of the experiment are eye-opening. The researchers analyzed the two approaches on the three components of extreme value theory. They find hybrid is better on the individual components of the theory, and in the ultimate test: quality of the top ideas produced.

Number of ideas generated. Hybrid structure generates three times more ideas than does the team structure. Researchers attribute this result to three dynamics:

  1. Free riding: it's easy enough to ride the idea coattails of the group
  2. Evaluation apprehension: the fear of negative reaction when proposing an idea in front of a group
  3. Production blocking: participants have to wait while one person is speaking, limiting idea generation throughput

Idea quality: The average quality of the hybrid structure ideas was higher than that of the team structure. Specifically, 0.25 points better in business value, 0.35 points better in purchase intent. To put this in perspective, these differences translate into roughly a 30 point differential in percentile rankings. In other words, the difference between the 1st and 30th idea in a pool of 100 ideas.

Researchers attribute the decrease in idea quality for team structures to the same free riding dynamic that reduces the quantity of ideas.

Idea quality variance: The researchers found no discernible difference in idea quality variance between the hybrid and team structures.

What this means is that from extreme value theory, the quantity and average quality of ideas are the key drivers of generating the highest-ranked ideas.

Best ideas: Here's where the rubber meets the road. Which approach had the highest ranked ideas? Hybrid structure, by a landslide.

The researchers looked at the top 5 ideas, by quality scores, that emerged from the two approaches. The hybrid structure ideas were of much higher quality than those generated from the team structure. This finding held for looking at the top 3, 4 and 6 ideas as well.

To recap:

The hybrid structure produced:

  • More ideas
  • Ideas of better quality on average
  • Highest rated ideas


Ability to Select Best Ideas

Perhaps the one down note from the study is the ability of the group to select the best ideas. Remember that in both the team and hybrid structures, the group did a consensus selection of the top ideas. Participants weren't asked to select the top ideas individually.

The researchers found a small advantage in the hybrid structure group's ability to select the top 5 ideas resulting from their ideation exercises. But it wasn't material. Indeed, they note:

The hybrid process may generate better ideas, but that due to the noisy selection process, its relative advantage is much diminished, to the point of becoming statistically insignificant for one of our quality metrics.

"Noisy selection process", indeed.  Ever been in a brainstorming session where you're supposed to rank the ideas at the end? Imagine the dynamics of resolving differences of opinion, time constraints and the extraordinary influence of certain individuals that drowns out other opinions. This is not an optimal way to determine the ideas that define innovation for your organization.

 

What This Means for Companies Seeking Innovation

As we described previously in Crowdsourcing Is the New Collaboration, there are many benefits to taking a new approach to idea generation, peer collaboration and integrating innovation more deeply into an organization's culture. The Spigit innovation management platform is ideal for this approach.

As this study confirms, distributing the idea generation process, as well as the idea selection process, results in higher quality ideas for organizations. This study dovetails well with another study by Professor Ron Burt, that found that employees with access to a wider range of viewpoints and feedback generate higher quality ideas.

Brainstorming does have its benefits in terms of face-to-face interactions. Perhaps the nature of what is brainstormed needs to change. Brainstorming can be valuable for project-oriented tasks and problem-solving. But don't consider it your go-to activity for the best ideas.



				
				

permalink
Spig It!   Scrap It!      Share 3 Comments


The opposite? by shulkin - Feb 26, 2010

Hutch, Good points and thanks for pointing out the studies.

I've thought of idea management, collaborative networks and open innovation as methodologies to introduce chaos into very structured environments to spur change.  But lately I've been thinking it can do the opposite equally well.  Perhaps it can apply structure to already chaotic systems?

The example I can think of is the loan process.  You have bankers in different departments (customer service, underwriters, different banks, different pools of borrowers with different loan requirements).  Lots of people and facts and sources of facts (I'm thinking of how much paper there was for my mortgage).  Couldn't a collaborative system help apply structure to this chaos by providing a system where the lenders can fish the pool for the best deals, borrowers can shop rates, all the paperwork can be accumulated along the process, input from different departments can be amassed.

So sort of the opposite of Amy Shuen and Clayton Christensen's introduction of chaos to structure....structure to chaos.



	                
        
        	
0 Spig  Scrap   1



Re: Study - Distributed Idea Generation Outperforms Team Brainstorming by Hutch - Feb 26, 2010

Neat way you've framed that. Introducing "chaos" into structure. I also like your loan example. I once worked at Bank of America, so I can appreciate the subtleties of that process. Your example somewhat reminds me of another post written here, Goal Setting Stimulates Employee Innovation.  How could a distributed workforce improve the loan process at each major step along the way?



	                
        
        	
0 Spig  Scrap   1



Your conclusions - conducive or instrumental? by sospetto - Mar 10, 2010
I work as a trainer and educator in both for-profit and not-for-profit environments, and while I can ascertain that the findings of Girotra, Terwiesch and Ulrich tend to hold beyond many other contexts than the academic one they looked at, I have more doubts about some of your conclusions. Most importantly, the study did not look at distributed idea generation - the subjects were in the same room and were introduced to the challenges together; and within this setting one group started with a collective brainstorming session rightaway, while another allowed for individual reflections first. The study did not look at idea generation by groups of people not in the same room, and this makes a tremendous difference. Your physical presence in a particular place for a particular purpose at a particular time heavily influences how much energy and creativity you feel like and can invest in thinking about any kind of challenge. Against my experience from practice I would argue that the results of idea generation processes—when facilitated well—are always better when people are face-to-face, as compared to long-distance distributed processes. What the authors of the study call a hybrid approach has been around for a long time and is usually referred to as nominal group technique - time is given for people to write down their ideas anonymously. Such techniques were developed against the context of needing to bypass hierarchies, allowing people to judge the idea of a quality by the idea itself and not by the rank of the person who suggested it. I don't know any colleague in the field who runs a brainstorming session without giving people time to think on their own; and there are many, many more clever methodological twists in which brainstorming sessions can be made more powerful and rewarding. The idea of applying extreme value theory to brainstorming is a good one, and the illustration is great - but the two examples chosen for the research (referred to as team approach and hybrid approach) are light-years behind actual educational practice in the field. Unfortunately. But be that as it may - I just don't see how there is any way in which you can deduce from this particular study that distributed ways of generating ideas are more powerful.

	                
        
        	
0 Spig  Scrap   1




Post Comment

You must be signed in to post a comment.

Sign in or Register

Sign In
Sign Up

Blog Profile


Hutch

Providing updates on the emerging landscape of innovation for organizations around the world.

Add Connection
Watch this post

Tags

innovation (21)
enterprise-20 (12)
crowdsourcing (10)
open-innovation (6)
collaboration (6)
employees (6)
communities (6)
innovation-management (5)
motivation (4)
repurank (4)
spigit09 (3)
social-networks (3)
prediction-markets (3)
gov20 (3)
competition (3)
reputation (3)
customers (3)
gary-hamel (3)
maslow (2)
city-of-manor (2)
incentives (2)
adoption (2)
ovo (2)
ideas (2)
culture (2)
government (2)
wisdom-of-crowds (2)
cisco (2)
feiboston (2)
i-prize (2)
healthcare (2)
social-objects (2)
white-house (2)
idea-management (2)
social-software (2)
wiki (2)
engagement (1)
ideation (1)
pharma (1)
united-nations (1)
anonymity (1)
disruptive (1)
hype-cycle (1)
peace-dot (1)
tag-clouds (1)
activity-streams (1)
customer-summit (1)
hamel (1)
constraints (1)
goals (1)
fun (1)
jeffrey-phillips (1)
scope (1)
workflow (1)
filters (1)
innovation-strategy (1)
reviews (1)
webinar (1)
burt (1)
enterprise-search (1)
use-cases (1)
award (1)
economy (1)
ibm (1)
performance-appraisals (1)
tweets (1)
dachis-group (1)
hr (1)
spigot (1)
360-degree-reviews (1)
creativity (1)
social-profiles (1)
gartner (1)
knowledge-workers (1)
sensors (1)
x-prize (1)
forums (1)
interop (1)
roi (1)
wif09 (1)
challenge (1)
external (1)
influence (1)
raytheon (1)
votes (1)
blogs (1)
pfizer (1)
typos (1)
andrew-mcafee (1)
design (1)
human-resources (1)
partners (1)
summit (1)
activity-specific (1)
oil-spill (1)
spigit (1)
general-mills (1)
linkedin (1)
smarter-cities (1)
frost-and-sullivan (1)
invention (1)
schering-plough (1)
wbf09 (1)
brainstorming (1)
ieee (1)
pharmaceutical (1)
urban-public-safety (1)
attention-signals (1)
e20forum (1)
performance (1)
taylormade (1)
adidas (1)
opportunities (1)
spigit2010 (1)
creative-economy (1)
golf (1)
medplus (1)
game-mechanics (1)
kaiser-permanente (1)
search (1)
world-business-forum (1)
clay-shirky (1)
focus (1)
innovationspigit (1)
robots (1)
catalyst (1)
experts (1)
incremental (1)
process (1)
utilities (1)
best-practices (1)
education-transportation (1)
performance-reviews (1)
twitter (1)
analytics (1)
definition (1)
htc (1)
participation (1)
suggestion-box (1)
accenture (1)
governance (1)
network-constraint (1)
leader (1)
seo (1)
foursquare (1)
intervet (1)
saffo (1)
charlene-li (1)
failure (1)
walmart (1)
bp (1)

Who Else is Viewing


0 Members, 0 guests
No other logged-in member is viewing this page.
 

© 2008 - 2009 Spigit, Inc.