7 posts categorized "public policy"

06 October 2016

When nudging fails, defensive baseball stats, and cognitive bias cheat sheet.

What works reading list


1. When nudging fails, what else can be done?
Bravo to @CassSunstein, co-author of the popular book Nudge, for a journal abstract that is understandable and clearly identifies recommended actions. This from his upcoming article Nudges that Fail:

"Why are some nudges ineffective, or at least less effective than choice architects hope and expect? Focusing primarily on default rules, this essay emphasizes two reasons. The first involves strong antecedent preferences on the part of choosers. The second involves successful “counternudges,” which persuade people to choose in a way that confounds the efforts of choice architects. Nudges might also be ineffective, and less effective than expected, for five other reasons. (1) Some nudges produce confusion on the part of the target audience. (2) Some nudges have only short-term effects. (3) Some nudges produce “reactance” (though this appears to be rare) (4) Some nudges are based on an inaccurate (though initially plausible) understanding on the part of choice architects of what kinds of choice architecture will move people in particular contexts. (5) Some nudges produce compensating behavior, resulting in no net effect. When a nudge turns out to be insufficiently effective, choice architects have three potential responses: (1) Do nothing; (2) nudge better (or different); and (3) fortify the effects of the nudge, perhaps through counter-counternudges, perhaps through incentives, mandates, or bans."

This work will appear in a promising new journal, behavioral science & policy, "an international, peer-reviewed journal that features short, accessible articles describing actionable policy applications of behavioral scientific research that serves the public interest. articles submitted to bsp undergo a dual-review process. leading scholars from specific disciplinary areas review articles to assess their scientific rigor; at the same time, experts in relevant policy areas evaluate them for relevance and feasibility of implementation.... bsp is a publication of the behavioral science & policy association and the brookings institution press."

Slice of the week @ PepperSlice.

Author: Cass Sunstein

Analytical method: Behavioral economics

Relationship: Counter-nudges → interfere with → behavioral public policy initiatives


2. There will be defensive baseball stats!
Highly recommended: Bruce Schoenfeld's writeup about Statcast, and how it will support development of meaningful statistics for baseball fielding. Cool insight into the work done by insiders like Daren Willman (@darenw). Finally, it won't just be about the slash line.


3. Cognitive bias cheat sheet.
Buster Benson (@buster) posted a cognitive bias cheat sheet that's worth a look. (Thanks @brentrt.)


4. CATO says Donald Trump is wrong.
Conservative think tank @CatoInstitute shares evidence that immigrants don’t commit more crimes. "No matter how researchers slice the data, the numbers show that immigrants commit fewer crimes than native-born Americans.... What the anti-immigration crowd needs to understand is not only are immigrants less likely to commit crimes than native-born Americans, but they also protect us from crimes in several ways."


5. The What Works reading list.
Don't miss the #WhatWorks Reading List: Good Reads That Can Help Make Evidence-Based Policy-Making The New Normal. The group @Results4America has assembled a thought-provoking list of "resources from current and former government officials, university professors, economists and other thought-leaders committed to making evidence-based policy-making the new normal in government."


Evidence & Insights Calendar

Oct 18, online: How Nonprofits Can Attract Corporate Funding: What Goes On Behind Closed Doors. Presented by the Stanford Social Innovation Review (@SSIReview).

Nov 25, Oxford: Intro to Evidence-Based Medicine presented by CEBM. Note: In 2017 CEBM will offer a course on teaching evidence-based medicine.

Dec 13, San Francisco: The all-new Systems We Love, inspired by the excellent Papers We Love meetup series. Background here.

October 19-22, Copenhagen. ISOQOL 23rd annual conference on quality of life research. Pro tip: The Wall Street Journal says Copenhagen is hot.

November 9-10, Philadelphia: Real-World Evidence & Market Access Summit 2016. "No more scandals! Access for Patients. Value for Pharma."

22 September 2016

Improving vs. proving, plus bad evidence reporting.

Turtle slow down and learn something

If you view gathering evidence as simply a means of demonstrating outcomes, you’re missing a trick. It’s most valuable when part of a journey of iterative improvement. - Frances Flaxington

1. Immigrants to US don't disrupt employment.
There is little evidence that immigration significantly affects overall employment of native-born US workers. This according to an expert panel's 500-page report. We thought you might like this condensed version from PepperSlice.

Bad presentation alert: The report, The Economic and Fiscal Consequences of Immigration, offers no summary visuals and buries its conclusions deep within dense chapters. Perhaps methodology is the problem, documenting the "evidence-based consensus of an authoring committee of experts". People need concise synthesis and actionable findings: What can policy makers do with this information?

Bad reporting alert: Perhaps unsatisfied with these findings, Julia Preston of the New York Times slipped her own claim into the coverage, saying the report "did not focus on American technology workers [true], many of whom have been displaced from their jobs in recent years by immigrants on temporary visas [unfounded claim]". Rather sloppy reporting, particularly when covering an extensive economic study of immigration impacts.


Immigration

Key evidence: "Empirical research in recent decades suggests that findings remain by and large consistent with those in The New Americans (National Research Council, 1997) in that, when measured over a period of 10 years or more, the impact of immigration on the wages of natives overall is very small." [page 204]

Immigration also contributes to the nation’s economic growth.... Perhaps even more important than the contribution to labor supply is the infusion by high-skilled immigration of human capital that has boosted the nation’s capacity for innovation and technological change. The contribution of immigrants to human and physical capital formation, entrepreneurship, and innovation are essential to long-run sustained economic growth. [page 243]

Author: @theNASEM, the National Academies of Sciences, Engineering, and Medicine.

Relationship: immigration → sustains → economic growth


2. Improving vs. proving.
On @A4UEvidence: "We often assume that generating evidence is a linear progression towards proving whether a service works. In reality the process is often two steps forward, one step back." Ugly Research supports the 'what works' concept, but wholeheartedly agrees that "The fact is that evidence rarely provides a clear-cut truth – that a service works or is cost-beneficial. Rather, evidence can support or challenge the beliefs that we, and others, have and it can point to ways in which a service might be improved."


3. Who should make sure policy is evidence-based and transparent?
Bad PR alert? Is it government's responsibility to make policy transparent and balanced? If so, some are accusing the FDA of not holding up their end on drug and medical device policy. A recent 'close-held embargo' of an FDA announcement made NPR squirm. Scientific American says the deal was this: "NPR, along with a select group of media outlets, would get a briefing about an upcoming announcement by the U.S. Food and Drug Administration a day before anyone else. But in exchange for the scoop, NPR would have to abandon its reportorial independence. The FDA would dictate whom NPR's reporter could and couldn't interview.

"'My editors are uncomfortable with the condition that we cannot seek reaction,' NPR reporter Rob Stein wrote back to the government officials offering the deal. Stein asked for a little bit of leeway to do some independent reporting but was turned down flat. Take the deal or leave it."


Evidence & Insights Calendar

November 9-10, Philadelphia: Real-World Evidence & Market Access Summit 2016. "No more scandals! Access for Patients. Value for Pharma."

29 Oct-2 Nov, Vienna, Austria: ISPOR 19th Annual European Congress. Plenary: "What Synergies Could Be Created Between Regulatory and Health Technology Assessments?"

October 3-6, National Harbor, Maryland. AMCP Nexus 2016. Special topic: "Behavioral Economics - What Does it All Mean?"


Photo credit: Turtle on Flickr.

25 August 2016

Social determinants of health, nonfinancial performance metrics, and satisficers.

Dear reader: Insights Weekly is starting a new chapter. Our spotlight topics are now accompanied by a 'newsletter' version of a PepperSlice, the capsule form of evidence-based analysis we've created at PepperSlice.com. Let me know what you think, and thanks for your continued readership. - Tracy Altman, Ugly Research

1. Is social services spending associated with better health outcomes? Yes.
Socialhealth-pepperslice-thumbnail Evidence has revealed a significant association between healthcare outcomes and the ratio of social service to healthcare spending in various OECD countries. Now a new study, published in Health Affairs, finds this same pattern within the US. The health differences were substantial. For instance, a 20% change in the median social-to-health spending ratio was associated with 85,000 fewer adults with obesity and more than 950,000 adults with mental illness. Elizabeth Bradley and Lauren Taylor explain on the RWJF Culture of Health blog.

This is great, but we wonder: Where/what is the cause-effect relationship?

The Evidence. Peer-reviewed: Variation In Health Outcomes: The Role Of Spending On Social Services, Public Health, And Health Care, 2000-09.

Data: Collected using longitudinal state-level spending data and analyzed with repeated measures multivariable modeling, retrospective.

Relationship: Social : medical spending → associated → better health outcomes

From the authors: "Reorienting attention and resources from the health care sector to upstream social factors is critical, but it’s also an uphill battle. More research is needed to characterize how the health effects of social determinants like education and poverty act over longer time horizons. Stakeholders need to use information about data on health—not just health care—to make resource allocation decisions."

#: statistical_modeling social_determinants population_health social_services health_policy

2. Are nonfinancial metrics good leading indicators of financial performance? Maybe.
Nonfinancial-metrics During the '90s and early 00's we heard a lot about Kaplan and Norton's Balanced Scorecard. A key concept was the use of nonfinancial management metrics such as customer satisfaction, employee engagement, and openness to innovation. This was thought to encourage actions that increased a company’s long-term value, rather than maximizing short-term financials.

The idea has taken hold, and nonfinancial metrics are often used in designing performance management systems and executive compensation plans. But not everyone is a fan: Some argue this can actually be harmful; for instance, execs might prioritize customer sat over other performance areas. Recent findings in the MIT Sloan Management Review look at whether these metrics truly are leading indicators of financial performance.

The Evidence. Business journal: Are Nonfinancial Metrics Good Leading Indicators of Future Financial Performance?

Data: Collected from American Customer Satisfaction Index, ExecuComp, and Compustat and analyzed with econometrics: panel data analysis.

Relationship: Nonfinancial metrics → predict → Financial performance

From the authors: "We found that there were notable variations in the lead indicator strength of customer satisfaction in a sample of companies drawn from different industries. For instance, for a chemical company in our sample, customer satisfaction’s lead indicator strength was negative; this finding is consistent with prior research suggesting that in many industries, the expense required to increase customer satisfaction can’t be justified. By contrast, for a telecommunications company we studied, customer satisfaction was a strong leading indicator; this finding is consistent with evidence showing that in many service industries, customer satisfaction reduces customer churn and price sensitivity. For a professional service firm in our sample, the lead indicator strength of customer satisfaction was weak; this is consistent with evidence showing that for such services, measures such as trust provide a clearer indication of the economic benefits than customer satisfaction.... Knowledge of whether a nonfinancial metric such as customer satisfaction is a strongly positive, weakly positive, or negative lead indicator of future financial performance can help companies avoid the pitfalls of using a nonfinancial metric to incentivize the wrong behavior."

#: customer_satisfaction nonfinancial balanced_scorecard CEO_compensation performance_management

3. Reliable evidence about p values.
Daniël Lakens (@lakens) puts it very well, saying "One of the most robust findings in psychology replicates again: Psychologists misinterpret p-values." This from Frontiers in Psychology.

4. Satisficers are happier.
Fast Company's article sounds at first just like clickbait, but they have a point. You can change how you see things, and reset your expectations. The Surprising Scientific Link Between Happiness And Decision Making.

Evidence & Insights Calendar:

September 19-21; Boston. FierceBiotech Drug Development Forum. Evaluate challenges, trends, and innovation in drug discovery and R&D. Covering the entire drug development process, from basic research through clinical trials.

September 13-14; Palo Alto, California. Nonprofit Management Institute: The Power of Network Leadership to Drive Social Change, hosted by Stanford Social Innovation Review.

September 20-22; Newark, New Jersey. Advanced Pharma Analytics. How to harness real-world evidence to optimize decision-making and improve patient-centric strategies.


Photo credit: Fat cat by brokinhrt2 on Flickr.

13 July 2016

Academic clickbait, FCC doesn't use economics, and tobacco surcharges don't work.

Brady

1. Academics use crazy tricks for clickbait.
Turn to @TheWinnower for an insightful analysis of academic article titles, and how their authors sometimes mimic techniques used for clickbait. Positively framed titles (those stating a specific finding) fare better than vague ones: For example, 'smoking causes lung cancer' vs. 'the relationship between smoking and lung cancer'. Nice use of altmetrics to perform the analysis.

2. FCC doesn't use cost-benefit analysis.
Critics claim Federal Communications Commission policymaking has swerved away from econometric evidence and economic theory. Federal agencies including the EPA must submit cost-benefit analyses to support new regulations, but the FCC is exempted, "free to embrace populism as its guiding principle". @CALinnovates has published a new paper, The Curious Absence of Economic Analysis at the Federal Communications Commission: An Agency In Search of a Mission. Former FCC Chief Economist Gerald Faulhaber, PhD and Hal Singer, PhD review the agency’s "proud history at the cutting edge of industrial economics and its recent divergence from policymaking grounded in facts and analysis".

3. No bias in US police shootings?
There's plenty of evidence showing bias in US police use of force, but not in shootings, says one researcher. But Data Colada, among others, describes "an interesting empirical challenge for interpreting the shares of Whites vs Blacks shot by police while being arrested is that biased officers, those overestimating the threat posted by a Black civilian, will arrest less dangerous Blacks on average. They will arrest those posing a real threat, but also some not posing a real threat, resulting in lower average threat among those arrested by biased officers."

4. Tobacco surcharges don't work.
The Affordable Care Act imposes tobacco surcharges for smokers. But findings suggest the ACA has not led more people to stop smoking.

5. CEOs lose faith in forecasts.
Some CEOs say big-data predictions are failing. “The so-called experts and global economists are proven as often to be wrong as right these days,” claims a WSJ piece In Uncertain Times, CEOs Lose Faith in Forecasts. One consultant advises people to "rely less on forecasts and instead road-test ideas with customers and make fast adjustments when needed. He urges them to supplement big-data predictions with close observation of their customers."

6. Is fMRI evidence flawed?
Motherboard's Why Two Decades of Brain Research Could Be Seriously Flawed recaps research by Anders Eklund. Cost is one reason, he argues: fMRI scans are notoriously expensive. "That makes it hard for researchers to perform large-scale studies with lots of patients". Eklund has written elsewhere about this (Can parametric statistical methods be trusted for fMRI based group studies?), and the issue is being noticed by Neuroskeptic and Science-Based Medicine ("It’s tempting to think that the new idea or technology is going to revolutionize science or medicine, but history has taught us to be cautious. For instance, antioxidants, it turns out, are not going to cure a long list of diseases").

Evidence & Insights Calendar:

August 24-25; San Francisco. Sports Analytics Innovation Summit.

September 13-14; Palo Alto, California. Nonprofit Management Institute: The Power of Network Leadership to Drive Social Change, hosted by Stanford Social Innovation Review.

September 19-23; Melbourne, Australia. International School on Research Impact Assessment. Founded in 2013 by the Agency of Health Quality and Assessment (AQuAS), RAND Europe, and Alberta Innovates.

28 April 2016

Bitcoin for learning, market share meaninglessness, and fighting poverty with evidence.

College diploma

1. Bitcoin tech records people's learning.
Ten years from now, what if you could evaluate a job candidate by reviewing their learning ledger, a blockchain-administered record of their learning transactions - from courses they took, books they read, or work projects they completed? And what if you could see their work product (papers etc.) rather than just their transcript and grades? Would that be more relevant and useful than knowing what college degree they had?

This is the idea behind Learning is Earning 2026, a future system that would reward any kind of learning. The EduBlocks Ledger would use the same blockchain technology that runs Bitcoin. Anyone could award these blocks to anyone else. As explained by Marketplace Morning Report, the Institute for the Future is developing the EduBlocks concept.

 

Market share MIT-Sloan

2. Is market share a valuable metric?
Only in certain cases is market share an important metric for figuring out how to make more profits. Neil T. Bendle and Charan K. Bagga explain in the MIT Sloan Management Review that Popular marketing metrics, including market share, are regularly misunderstood and misused.

Well-known research in the 1970s suggested a link between market share and ROI. But now most evidence shows it's a correlational relationship, not causal.

 

Adolescent crime

3. Evidence-based ways to close gaps in crime, poverty, education.
The Laura and John Arnold Foundation launched a $15 million Moving the Needle Competition, which will fund state and local governments and nonprofits implementing highly effective ways to address poverty, education, and crime. The competition is recognized as a key evidence-based initiative in White House communications about My Brother’s Keeper, a federal effort to address persistent opportunity gaps.

Around 250 communities have responded to the My Brother’s Keeper Community Challenge with $600+ million in private sector and philanthropic grants, plus $1 billion in low-interest financing. Efforts include registering 90% of Detroit's 4-year-olds in preschool, private-sector “MBK STEM + Entrepreneurship” commitments, and a Summit on Preventing Youth Violence.

Here's hoping these initiatives are evaluated rigorously, and the ones demonstrating evidence of good or promising outcomes are continued.

 

Eddie Izzard

4. Everyday health evidence.
Evidence for Everyday Health Choices is a new series by @UKCochraneCentr, offering quick rundowns of the systematic reviews on a pertinent topic. @SarahChapman30 leads the effort. Nice recent example inspired by Eddie Izzard: Evidence on stretching and other techniques to improve marathon performance and recovery: Running marathons Izzard enough: what can help? [Photo credit: Evidence for Everyday Health Choices.]

5. Short Science = Understandable Science.
Short Science allows people to publish summaries of research papers; they're voted on and ranked until the best/most accessible summary has been identified. The goal is to make seminal ideas in science accessible to the people who want to understand them. Anyone can write a summary of any paper in the Short Science database. Thanks to Carl Anderson (@LeapingLlamas).

30 March 2016

$15 minimum wage, evidence-based HR, and manmade earthquakes.

Fightfor15.org

Photo by Fightfor15.org

1. SPOTLIGHT: Will $15 wages destroy California jobs?
California is moving toward a $15/hour minimum wage (slowly, stepping up through 2023). Will employers be forced to eliminate jobs under the added financial pressure? As with all things economic, it depends who you ask. Lots of numbers have been thrown around during the recent push for higher pay. Fightfor15.org says 6.5 million workers are getting raises in California, and that 2/3 of New Yorkers support a similar increase. But small businesses, restaurants in particular, are concerned they'll have to trim menus and staff - they can charge only so much for a sandwich.

Moody's Analytics economist Adam Ozimek says it's not just about food service or home healthcare. Writing on The Dismal Scientist Blog, "[I]n past work I showed that California has 600,000 manufacturing workers who currently make $15 an hour or less. The massive job losses in manufacturing over the last few decades has shown that it is an intensely globally competitive industry where uncompetitive wages are not sustainable." 

It's not all so grim. Ozimek shows that early reports of steep job losses after Seattle's minimum-wage hike have been revised strongly upward. However, finding "the right comparison group is getting complicated."


Yellow Map Chance of Earthquake

2. Manmade events sharply increase earthquake risk.
Holy smokes. New USGS maps show north-central Oklahoma at high earthquake risk. The United States Geological Survey now includes potential ground-shaking hazards from both 'human-induced' and natural earthquakes, substantially changing their risk assessment for several areas. Oklahoma recorded 907 earthquakes last year at magnitude 3 or higher. Disposal of industrial wastewater has emerged as a substantial factor.

3. Evidence-based HR redefines leadership roles.
Applying evidence-based principles to talent management can boost strategic impact, but requires a different approach to leadership. The book Transformative HR: How Great Companies Use Evidence-Based Change for Sustainable Advantage (Jossey-Bass) describes practical uses of evidence to improve people management. John Boudreau and Ravin Jesuthasan suggest principles for evidence-based change, including logic-driven analytics. For instance, establishing appropriate metrics for each sphere of your business, rather than blanket adoption of measures like employee engagement and turnover.

4. Why we're not better at investing.
Gary Belsky does a great job of explaining why we think we're better investors than we are. By now our decision biases have been well-documented by behavioral economists. Plus we really hate to lose - yet we're overconfident, somehow thinking we can compete with Warren Buffet.

23 March 2016

Rapid is the new black, how to ask for money, and should research articles be free?

Digitalhealthnetwork

1. #rapidisthenewblack

The need for speed is paramount, so it's crucial that we test ideas and synthesize evidence quickly without losing necessary rigor. Examples of people working hard to get it right:

  • The Digital Health Breakthrough Network is a very cool idea, supported by an A-list team. They (@AskDHBN) seek New York City-based startups who want to test technology in rigorous pilot studies. The goal is rapid validation of early-stage startups with real end users. Apply here.
  • The UK's fantastic Alliance for Useful Evidence (@A4UEvidence) asks Rapid Evidence Assessments: A bright idea or a false dawn? "Research synthesis will be at the heart of the government’s new What Works centres" - equally true in the US. The idea is "seductive: the rigour of a systematic review, but one that is cheaper and quicker to complete." Much depends on whether the review maps easily onto an existing field of study.
  • Jon Brassey of the Trip database is exploring methods for rapid reviews of health evidence. See Rapid-Reviews.info or @rapidreviews_i.
  • Miles McNall and Pennie G. Foster-Fishman of Michigan State (ouch, still can't get over that bracket-busting March Madness loss) present methods and case studies for rapid evaluations and assessments. In the American Journal of Evaluation, they caution that the central issue is balancing speed and trustworthiness.

2. The science of asking for donations: Unit asking method.
How much would you give to help one person in need? How much would you give to help 20 people? This is the concept behind the unit asking method, a way to make philanthropic fund-raising more successful.

3. Should all research papers be free? 
Good stuff from the New York Times on the conflict between scholarly journal paywalls and Sci-Hub.

4. Now your spreadsheet can tell you what's going on.
Savvy generates a narrative for business intelligence charts in Qlik or Excel.

Subscribe by email