22 posts categorized "science & research methods"

06 October 2016

When nudging fails, defensive baseball stats, and cognitive bias cheat sheet.

What works reading list

1. When nudging fails, what else can be done?
Bravo to @CassSunstein, co-author of the popular book Nudge, for a journal abstract that is understandable and clearly identifies recommended actions. This from his upcoming article Nudges that Fail:

"Why are some nudges ineffective, or at least less effective than choice architects hope and expect? Focusing primarily on default rules, this essay emphasizes two reasons. The first involves strong antecedent preferences on the part of choosers. The second involves successful “counternudges,” which persuade people to choose in a way that confounds the efforts of choice architects. Nudges might also be ineffective, and less effective than expected, for five other reasons. (1) Some nudges produce confusion on the part of the target audience. (2) Some nudges have only short-term effects. (3) Some nudges produce “reactance” (though this appears to be rare) (4) Some nudges are based on an inaccurate (though initially plausible) understanding on the part of choice architects of what kinds of choice architecture will move people in particular contexts. (5) Some nudges produce compensating behavior, resulting in no net effect. When a nudge turns out to be insufficiently effective, choice architects have three potential responses: (1) Do nothing; (2) nudge better (or different); and (3) fortify the effects of the nudge, perhaps through counter-counternudges, perhaps through incentives, mandates, or bans."

This work will appear in a promising new journal, behavioral science & policy, "an international, peer-reviewed journal that features short, accessible articles describing actionable policy applications of behavioral scientific research that serves the public interest. articles submitted to bsp undergo a dual-review process. leading scholars from specific disciplinary areas review articles to assess their scientific rigor; at the same time, experts in relevant policy areas evaluate them for relevance and feasibility of implementation.... bsp is a publication of the behavioral science & policy association and the brookings institution press."

Slice of the week @ PepperSlice.

Author: Cass Sunstein

Analytical method: Behavioral economics

Relationship: Counter-nudges → interfere with → behavioral public policy initiatives

2. There will be defensive baseball stats!
Highly recommended: Bruce Schoenfeld's writeup about Statcast, and how it will support development of meaningful statistics for baseball fielding. Cool insight into the work done by insiders like Daren Willman (@darenw). Finally, it won't just be about the slash line.

3. Cognitive bias cheat sheet.
Buster Benson (@buster) posted a cognitive bias cheat sheet that's worth a look. (Thanks @brentrt.)

4. CATO says Donald Trump is wrong.
Conservative think tank @CatoInstitute shares evidence that immigrants don’t commit more crimes. "No matter how researchers slice the data, the numbers show that immigrants commit fewer crimes than native-born Americans.... What the anti-immigration crowd needs to understand is not only are immigrants less likely to commit crimes than native-born Americans, but they also protect us from crimes in several ways."

5. The What Works reading list.
Don't miss the #WhatWorks Reading List: Good Reads That Can Help Make Evidence-Based Policy-Making The New Normal. The group @Results4America has assembled a thought-provoking list of "resources from current and former government officials, university professors, economists and other thought-leaders committed to making evidence-based policy-making the new normal in government."

Evidence & Insights Calendar

Oct 18, online: How Nonprofits Can Attract Corporate Funding: What Goes On Behind Closed Doors. Presented by the Stanford Social Innovation Review (@SSIReview).

Nov 25, Oxford: Intro to Evidence-Based Medicine presented by CEBM. Note: In 2017 CEBM will offer a course on teaching evidence-based medicine.

Dec 13, San Francisco: The all-new Systems We Love, inspired by the excellent Papers We Love meetup series. Background here.

October 19-22, Copenhagen. ISOQOL 23rd annual conference on quality of life research. Pro tip: The Wall Street Journal says Copenhagen is hot.

November 9-10, Philadelphia: Real-World Evidence & Market Access Summit 2016. "No more scandals! Access for Patients. Value for Pharma."

27 July 2016

Business coaching, manipulating memory for market research, and female VCs.


1. Systematic review: Does business coaching make a difference?
In PLOSOne, Grover and Furnham present findings of their systematic review of coaching impacts within organizations. They found glimmers of hope for positive results from coaching, but also spotted numerous holes in research designs and data quality.

Over the years, outcome measures have included job satisfaction, performance, self-awareness, anxiety, resilience, hope, autonomy, and goal attainment. Some have measured ROI, although this one seems particularly subjective. In terms of organizational impacts, researchers have measured transformational leadership and performance as rated by others. This systematic review included only professional coaches, whether internal or external to the organization. Thanks @Rob_Briner and @IOPractitioners.

2. Memory bias pollutes market research.
David Paull of Dialsmith hosted a series about how flawed recall and memory bias affect market research. (Thanks to @kristinluck.)

All data is not necessarily good data. “We were consistently seeing a 13–20% misattribution rate on surveys due in large part to recall problems. Resultantly, you get this chaos in your data and have to wonder what you can trust.... Rather than just trying to mitigate memory bias, can we actually use it to our advantage to offset issues with our brands?”

The ethics of manipulating memory. “We can actually affect people’s nutrition and the types of foods they prefer eating.... But should we deliberately plant memories in the minds of people so they can live healthier or happier lives, or should we be banning the use of these techniques?”

Mitigating researchers' memory bias. “We’ve been talking about memory biases for respondents, but we, as researchers, are also very prone to memory biases.... There’s a huge opportunity in qual research to apply an impartial technique that can mitigate (researcher) biases too....[I]n the next few years, it’s going to be absolutely required that anytime you do something that is qualitative in nature that the analysis is not totally reliant on humans.”

3. Female VC --> No gender gap for startup funding.
New evidence suggests female entrepreneurs should choose venture capital firms with female partners (SF Business Times). Michigan's Sahil Raina analyzed data to compare the gender gap in successful exits from VC financing between two sets of startups: those initially financed by VCs with only male general partners (GPs), and those initially financed by VCs that include female GPs. “I find a large performance gender gap among startups financed by VCs with only male GPs, but no such gap among startups financed by VCs that include female GPs.”

4. Sharing evidence about student outcomes.
Results for America is launching an Evidence in Education Lab to help states, school districts, and individual schools build and use evidence of 'what works' to improve student outcomes. A handful of states and districts will work closely with RFA to tackle specific data challenges.

Background: The bipartisan Every Student Succeeds Act (ESSA) became law in December 2015. ESSA requires, allows, and encourages the use of evidence-based approaches that can help improve student outcomes. Results for America estimates that ESSA's evidence provisions could help shift more than $2B US of federal education funds in each of the next four years toward evidence-based, results-driven solutions.

Evidence & Insights Calendar:

September 13-14; Palo Alto, California. Nonprofit Management Institute: The Power of Network Leadership to Drive Social Change, hosted by Stanford Social Innovation Review.

September 19-23; Melbourne, Australia. International School on Research Impact Assessment. Founded in 2013 by the Agency of Health Quality and Assessment (AQuAS), RAND Europe, and Alberta Innovates.

20 July 2016

Evidence relativism, innovation as a process, and decision analysis pioneer.


1. Panning for gold in the evidence stream.
Patrick Lester introduces his new SSIR article by saying "With evidence-based policy, we need to acknowledge that some evidence is more valid than others. Pretending all evidence is equal will only preserve the status quo." In Defining Evidence Down, the director of the Social Innovation Research Center responds to analysts skeptical of evidence hierarchies developed to steer funding toward programs that fit the "what works" concept.

Are levels valid? Hierarchies recognize different levels of evidence according to their rigor and certainty. These rankings are well-established in healthcare, and are becoming the standard for evidence evaluation within the Dept of Education and other US government agencies. Critics of this prevailing thinking (Gopal & Schorr, Friends of Evidence) want to ensure decision-makers embrace an inclusive definition of evidence that values qualitative research, case studies, insights from experience, and professional judgment. Lester contends that "Unfortunately, to reject evidence hierarchies is to promote is a form of evidence relativism, where everyone is entitled to his or her own views about what constitutes good evidence in his or her own local or individualized context.

Ideology vs. evidence. "By resisting the notion that some evidence is more valid than others, they are defining evidence down. Such relativism would risk a return to the past, where change has too often been driven by fads, ideology, and politics, and where entrenched interests have often preserved the status quo." Other highlights: "...supporting a broad definition of evidence is not the same thing as saying that all evidence is equally valid." And "...randomized evaluations are not the only rigorous way to examine systems-level change. Researchers can often use quasi-experimental evaluations to examine policy changes...."

2. Can innovation be systematic?
Everyone wants innovation nowadays, but how do you make it happen? @HighVizAbility reviews a method called Systematic Inventive Thinking, an approach to creativity, innovation, and problem solving. The idea is to execute as a process, rather than relying on random ideas. Advocates say SIT doesn't replace unbridled creativity, but instead complements it.

3. Remembering decision analysis pioneer Howard Raiffa.
Howard Raiffa, co-founder of the Harvard Kennedy School of Government and decision analysis pioneer, passed away recently. He was also a Bayesian decision theorist and well-known author on negotiation strategies. Raiffa considered negotiation analysis an opportunity for both sides to get value, describing it as The Science and Art of Collaborative Decision Making.

4. Journal impact factor redux?
In the wake of news that Thomson Reuters sold its formula, Stat says changes may finally be coming to the "hated" journal impact factor. Ivan Oransky (@ivanoransky) and Adam Marcus (@armarcus) explain that some evidence suggests science articles don't receive the high number of citations supposedly predicted by the IF. The American Society of Microbiologists has announced that it will abandon the metric completely. Meanwhile, top editors from Nature — which in the past has taken pride in its IF — have coauthored a paper widely seen as critical of the factor.

Photo credit: Poke of Gold by Mike Beauregard

19 July 2016

Are you causing a ripple? How to assess the impact of research.


People are recognizing the critical need for meta-research, or the 'science of science'. One focus area is understanding whether research produces desired outcomes, and identifying how to ensure that truly happens going forward. Research impact assessment (RIA) is particularly important when holding organizations accountable for their management of public and donor funding. An RIA community of practice is emerging.

Are you causing a ripple? For those wanting to lead an RIA effort, the International School on Research Impact Assessment was developed to "empower participants on how to assess, measure and optimise research impact with a focus on biomedical and health sciences." ISRIA is a partnership between Alberta Innovates, the Agency for Health Quality and Assessment of Catalonia, and RAND Europe. They're presenting their fourth annual program Sept 19-23 in Melbourne, Australia, hosted by the Commonwealth Scientific and Industrial Research Organisation, Australia’s national research agency.

ISRIA participants are typically in program management, evaluation, knowledge translation, or policy roles. They learn a range of frameworks, tools, and approaches for assessing research impact, and how to develop evidence about 'what works’.

Make an impact with your impact assessment. Management strategies are also part of the ISRIA curriculum: Embedding RIA systemically into organizational practice, reaching agreement on effective methods and reporting, understanding the audience for RIAs, and knowing how to effectively communicate results to various stakeholders.

The 2016 program will cover both qualitative and quantitative analytical methods, along with a mixed design. It will include sessions on evaluating economic, environmental and social impacts. The aim is to expose participants to as many options as possible, including new methods, such as altmetrics. (Plus, there's a black tie event on the first evening.)


Photo credit: Raindrops in a Bucket by Smabs Sputzer.


30 June 2016

Brain training isn't smart, physician peer pressure, and #AskforEvidence.


1. Spending $ on brain training isn't so smart.
It seems impossible to listen to NPR without hearing from their sponsor, Lumosity, the brain-training company. The target demo is spot on: NPR will be the first to tell you its listeners are the "nation's best and brightest". And bright people don't want to slow down. Alas, spending hard-earned money on brain training isn't looking like a smart investment. New evidence seems to confirm suspicions that this $1 billion industry is built on hope, sampling bias, and placebo effect. Arstechnica says researchers have concluded that earlier, mildly positive "findings suggest that recruitment methods used in past studies created a self-selected groups of participants who believed the training would improve cognition and thus were susceptible to the placebo effect." The study, Placebo Effects in Cognitive Training, was published in the Proceedings of the National Academy of Sciences.

It's not a new theme: In 2014, 70 cognitive scientists signed a statement saying "The strong consensus of this group is that the scientific literature does not support claims that the use of software-based 'brain games' alters neural functioning in ways that improve general cognitive performance in everyday life, or prevent cognitive slowing and brain disease."


2. Ioannidis speaks out on usefulness of research.
After famously claiming that most published research findings are false, John Ioannidis now tells us Why Most Clinical Research Is Not Useful (PLOS Medicine). So, what are the key features of 'useful' research? The problem needs to be important enough to fix. Prior evidence must be evaluated to place the problem into context. Plus, we should expect pragmatism, patient-centeredness, monetary value, and transparency.


3. To nudge physicians, compare them to peers.
Doctors are overwhelmed with alerts and guidance. So how do you intervene when a physician prescribes antibiotics for a virus, despite boatloads of evidence showing they're ineffective? Comparing a doc's records to peers is one promising strategy. Laura Landro recaps research by Jeffrey Linder (Brigham and Women's, Harvard): "Peer comparison helped reduce prescriptions that weren’t warranted from 20% to 4% as doctors got monthly individual feedback about their own prescribing habits for 18 months.

"Doctors with the lower rates were told they were top performers, while the rest were pointedly told they weren’t, in an email that included the number and proportion of antibiotic prescriptions they wrote compared with the top performers." Linder says “You can imagine a bunch of doctors at Harvard being told ‘You aren’t a top performer.’ We expected and got a lot of pushback, but it was the most effective intervention.” Perhaps this same approach would work outside the medical field.

4. Sports analytics taxonomy.
INFORMS is a professional society focused on Operations Research and Management Science. The June issue of their ORMS Today magazine presents v1.0 of a sports analytics taxonomy (page 40). This work, by Gary Cokins et al., demonstrates how classification techniques can be applied to better understand sports analytics. Naturally this includes analytics for players and managers in the major leagues. But it also includes individual sports, amateur sports, franchise management, and venue management.

5. Who writes the Internet, anyway? #AskforEvidence
Ask for Evidence is a public campaign that helps people request for themselves the evidence behind news stories, marketing claims, and policies. Sponsored by @senseaboutsci, the campaign has new animations on YouTube, Twitter, and Facebook. Definitely worth a like or a retweet.

September 13-14; Palo Alto, California. Nonprofit Management Institute: The Power of Network Leadership to Drive Social Change, hosted by Stanford Social Innovation Review.

September 19-23; Melbourne, Australia. International School on Research Impact Assessment. Founded in 2013 by the Agency of Health Quality and Assessment (AQuAS), RAND Europe, and Alberta Innovates.

05 May 2016

Counting to 10, science about science, and Explainista vs. Randomista.


SPOTLIGHT 1. Take a deep breath, everybody.
Great stuff this week reminding us that a finding doesn't necessarily answer a meaningful question. Let's revive the practice of counting to 10 before posting remarkable, data-driven insights... just in case.

This sums up everything that's right, and wrong, with data. In a recent discussion of some impressive accomplishments in sports analytics, prior success leads to this statement: “The bottom line is, if you have enough data, you can come pretty close to predicting almost anything,” says a data scientist. Hmmm. This sounds like someone who has yet to be punched in the face by reality. Thanks to Mara Averick (@dataandme).

Sitting doesn't typically kill people. On the KDNuggets blog, William Schmarzo remarks on the critical thinking part of the equation. For instance, the kerfuffle over evidence that people who sit most of the day are 54% more likely to die of heart attacks. Contemplating even briefly, though, raises the question about other variables, such as exercise, diet, or age.

Basic stats - Common sense = Dangerous conclusions viewed as fact

P-hacking is resilient. On Data Colada, Uri Simonsohn explains why P-Hacked Hypotheses are Deceivingly Robust. Direct, or conceptual, replications are needed now more than ever.

2. Science about science.
The world needs more meta-research. What's the best way to fund research? How can research impact be optimized, and how can that impact be measured? These are the questions being addressed at the International School on Research Impact Assessment, founded by RAND Europe, King's College London, and others. Registration is open for the autumn session, September 19-23 in Melbourne.

Evidence map by Bernadette Wright

3. Three Ways of Getting to Evidence-Based Policy.
In the Stanford Social Innovation Review, Bernadette Wright (@MeaningflEvdenc) does a nice job of describing three ideologies for gathering evidence to inform policy.

  1. Randomista: Views randomized experiments and quasi-experimental research designs as the only reliable evidence for choosing programs.
  2. Explainista: Believes useful evidence needs to provide trustworthy data and strong explanation. This often means synthesizing existing information from reliable sources.
  3. Mapista: Creates a knowledge map of a policy, program, or issue. Visualizes the understanding developed in each study, where studies agree, and where each adds new understanding.

30 March 2016

$15 minimum wage, evidence-based HR, and manmade earthquakes.


Photo by Fightfor15.org

1. SPOTLIGHT: Will $15 wages destroy California jobs?
California is moving toward a $15/hour minimum wage (slowly, stepping up through 2023). Will employers be forced to eliminate jobs under the added financial pressure? As with all things economic, it depends who you ask. Lots of numbers have been thrown around during the recent push for higher pay. Fightfor15.org says 6.5 million workers are getting raises in California, and that 2/3 of New Yorkers support a similar increase. But small businesses, restaurants in particular, are concerned they'll have to trim menus and staff - they can charge only so much for a sandwich.

Moody's Analytics economist Adam Ozimek says it's not just about food service or home healthcare. Writing on The Dismal Scientist Blog, "[I]n past work I showed that California has 600,000 manufacturing workers who currently make $15 an hour or less. The massive job losses in manufacturing over the last few decades has shown that it is an intensely globally competitive industry where uncompetitive wages are not sustainable." 

It's not all so grim. Ozimek shows that early reports of steep job losses after Seattle's minimum-wage hike have been revised strongly upward. However, finding "the right comparison group is getting complicated."

Yellow Map Chance of Earthquake

2. Manmade events sharply increase earthquake risk.
Holy smokes. New USGS maps show north-central Oklahoma at high earthquake risk. The United States Geological Survey now includes potential ground-shaking hazards from both 'human-induced' and natural earthquakes, substantially changing their risk assessment for several areas. Oklahoma recorded 907 earthquakes last year at magnitude 3 or higher. Disposal of industrial wastewater has emerged as a substantial factor.

3. Evidence-based HR redefines leadership roles.
Applying evidence-based principles to talent management can boost strategic impact, but requires a different approach to leadership. The book Transformative HR: How Great Companies Use Evidence-Based Change for Sustainable Advantage (Jossey-Bass) describes practical uses of evidence to improve people management. John Boudreau and Ravin Jesuthasan suggest principles for evidence-based change, including logic-driven analytics. For instance, establishing appropriate metrics for each sphere of your business, rather than blanket adoption of measures like employee engagement and turnover.

4. Why we're not better at investing.
Gary Belsky does a great job of explaining why we think we're better investors than we are. By now our decision biases have been well-documented by behavioral economists. Plus we really hate to lose - yet we're overconfident, somehow thinking we can compete with Warren Buffet.

16 March 2016

Equity crowdfunding algorithms, decision-making competitions, and statistical wild geese.


1. CircleUp uses algorithm to evaluate consumer startups.
Recently we wrote about #fintech startups who are challenging traditional consumer lending models. CircleUp is doing something similar to connect investors with non-tech consumer startups (food, cosmetics, recreation). It's not yet a robo adviser for automated investing, but they do use machine learning to remove drudgery from the analysis of private companies. @CircleUp's classifier selects emerging startups based on revenue, margins, distribution channels, etc., then makes their findings available to investors. They've also launched a secondary market where shareholders can sell their stakes twice annually. The company has successfully raised Series C funding.

2. Student decision-making competition.
In the 2016 @SABR case competition, college and university students analyzed and presented a baseball operations decision — the type of decision a team’s GM and staff face over the course of a season. Contestants were required to construct and defend a 2016 bullpen from scratch for any National League team, focusing on that team's quality of starting pitching, defense, home ballpark, division opponents, and other factors. The Carnegie Mellon team from the Tepper School of Business won the graduate division.

3. For many, writing is an essential data science skill.
Matt Asay (@mjasay) reminds us data science breaks down into two categories, depending on whether it's intended for human or machine consumption. The human-oriented activity often requires straightforward steps rather than complex digital models; business communication skills are essential. Besides manipulating data, successful professionals must excel at writing paragraphs of explanation or making business recommendations.

Writing for data science

4. Chasing statistical wild geese.
The American Statistical Association has released a statement on p-values: context, process, and purpose. There's been a flurry of discussion. If you find this tl;dr, the bottom line = "P-values don't draw bad conclusions, people do". The ASA's supplemental info section presents alternative points of view - mostly exploring ways to improve research by supplementing p-values, using Bayesian methods, or simply applying them properly. Christie Aschwanden wrote on @FiveThirtyEight that "A common misconception among nonstatisticians is that p-values can tell you the probability that a result occurred by chance. This interpretation is dead wrong, but you see it again and again and again and again. The p-value only tells you something about the probability of seeing your results given a particular hypothetical explanation...." Hence ASA Principle No. 2: “P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.” Nor can a p-value tell you the size of an effect, the strength of the evidence, or the importance of a result. The problem is the way p-values are used, explains Deborah Mayo (@learnfromerror): “failing to adjust them for cherry picking, multiple testing, post-data subgroups and other biasing selection effects”.

Photo credit: Around the campfire by Jason Pratt.

12 January 2016

Game theory for Jeopardy!, evidence for gun control, and causality.

1. Deep knowledge → Wagering strategy → Jeopardy! win Some Jeopardy! contestants struggle with the strategic elements of the show. Rescuing us is Keith Williams (@TheFinalWager), with the definitive primer on Jeopardy! strategy, applying game theory to every episode and introducing "the fascinating world of determining the optimal approach to almost anything".

2. Gun controls → Less violence? → Less tragedy? Does the evidence support new US gun control proposals? In the Pacific Standard, Francie Diep cites several supporting scientific studies.

3. New data sources → Transparent methods → Health evidence Is 'real-world' health evidence closer to the truth than data from more traditional categories? FDA staff explain in What We Mean When We Talk About Data. Thanks to @MandiBPro.

4. Data model → Cause → Effect In Why: A Guide to Finding and Using Causes, Samantha Kleinberg aims to explain why causality is often misunderstood and misused: What is it, why is it so hard to find, and how can we do better at interpreting it? The book excerpt explains that "Understanding when our inferences are likely to be wrong is particularly important for data science, where we’re often confronted with observational data that is large and messy (rather than well-curated for research)."

5. Empirical results → Verification → Scientific understanding Independent verification is essential to scientific progress. But in academia, verifying empirical results is difficult and not rewarded. This is the reason for Curate Science, a tool making it easier for researchers to independently verify each other’s evidence and award credit for doing so. Follow @CurateScience.

Join me at the HEOR writing workshop March 17 in Philadelphia. I'm speaking about communicating data, and leading an interactive session on data visualization. Save $300 before Jan 15.

22 December 2015

Asthma heartbreak, cranky economists, and prediction markets.

1. Childhood stress → Cortisol → Asthma Heartbreaking stories explain likely connections between difficult childhoods and asthma. Children in Detroit suffer a high incidence of attacks - regardless of allergens, air quality, and other factors. Peer-reviewed research shows excess cortisol may be to blame.

2. Prediction → Research heads up → Better evidence Promising technique for meta-research. A prediction market was created to quantify the reproducibility of 44 studies published in prominent psychology journals, and estimate likelihood of hypothesis acceptance at different stages. The market outperformed individual forecasts, as described in PNAS (Proceedings of the National Academy of Sciences.)

3. Fuzzy evidence → Wage debate → Policy fail More fuel for the minimum-wage fire. Depending on who you ask, a high minimum wage either bolsters the security of hourly workers or destroys the jobs they depend on. Recent example: David Neumark's claims about unfavorable evidence.

4. Decision tools → Flexible analysis → Value-based medicine Drug Abacus is an interactive tool for understanding drug pricing. This very interesting project, led by Peter Bach at Memorial Sloan Kettering, compares the price of a drug (US$) with its "worth", based on outcomes, toxicity, and other factors. Hopefully @drugabacus signals the future for health technology assessment and value-based medicine.

5. Cognitive therapy → Depression relief → Fewer side effects A BMJ systematic review and meta-analysis show that depression can be treated with cognitive behavior therapy, possibly with outcomes equivalent to antidepressants. Consistent CBT treatment is a challenge, however. AHRQ reports similar findings from comparative effectiveness research; the CER study illustrates how to employ expert panels to transparently select research questions and parameters.

Subscribe by email