2016 Federal Index
Innovation
Did the agency have staff, policies, and processes in place that encouraged innovation to improve the impact of its programs in FY17? (Examples: Prizes and challenges; behavioral science trials; innovation labs/accelerators; performance partnership pilots; demonstration projects or waivers with strong evaluation requirements)
Score
8
8
Administration for Children and Families (HHS)
- ACF’s Behavioral Innovations to Advance Self-Sufficiency (BIAS) project is the first major effort to apply a behavioral economics lens to programs that serve poor families in the U.S. Since its inception in 2010, the project has conducted 15 rapid-cycle randomized tests of behavioral innovations in seven states with nearly 100,000 sample members.
- ACF’s Behavioral Interventions for Child Support Services (BICS) demonstration project is applying behavioral insights to child support contexts, developing promising behavioral interventions, and building a culture of regular, rapid-cycle evaluation and critical inquiry within the child support community.
- ACF has actively participated in the HHS IDEA Lab, an entity within HHS launched in 2013, to invest in internal innovation, leverage external innovation, and build collaborative communities to tackle cross-cutting issues of strategic importance. Recent projects include the ACF Administration for Native Americans’ Application Toolkit and DataQuest: Making ACF Native Data Visible and Useful, the ACF Office of Family Assistance’s Understanding Temporary Assistance for Needy Families Through Data Vizualization, and the ACF Office of Head Start’s Partnership Alignment Information Response System.
Score
7
7
Corporation for National and Community Service
- CNCS awarded 7 AmeriCorps Evidence-Based Planning grants (an investment of $500,000 in evaluation funds) in FY17. These one-year planning grants were awarded to encourage the identification of a new evidence-based program or practice and develop a strategy for integrating national service into the effective model. In addition, awardees will develop an evaluation plan to assess the innovation should it be funded in future grant competitions. Research & Evaluation staff will conduct a process evaluation of these grantees to systematically assess the successes and challenges of implementing these grants.
- CNCS awarded another round of Commission Investment Fund grants in FY17. Overall, .3% of these grants (totaling $5,391,750 in AmeriCorps funding) will focus on building the capacity of State Commissions and their grantees to collect and use performance and evaluation data. Research & Evaluation staff are completing a process evaluation of these grantees that will identify the successes and challenges of implementing these grants. These findings will be made public in FY17 and used to inform the second cohort of funded grantees.
- R & E developed and pilot-tested an organizational capacity assessment tool with the goal of providing grantees one instrument to track data across CNCS programs. Encouraged by findings from the SIF National Assessment (which demonstrated the effectiveness of the initiative for improving organizational capacity in the areas of evidence and evaluation), CNCS seeks to assess its impact on organizations more systematically. The instrument will be available for OIRA clearance in FY17.
Score
9
9
Millennium Challenge Corporation
- MCC recently launched a new Millennium Efficiency Challenge to encourage innovation specifically in the compact development phase. The challenge is designed to tap into the extensive knowledge of MCC’s staff to identify efficiencies and innovative solutions that can shorten the compact development timeline while maintaining MCC’s rigorous quality standards and investment criteria. Winning ideas will be integrated into the compact development process starting in 2018 and could have lasting impact on MCC for years to come.
- In September 2014, MCC’s Monitoring and Evaluation division launched the agency’s first Open Data Challenge, a call-to-action to Masters and PhD students working in economics, public policy, international development, or other related fields who were interested in exploring how to use publicly available MCC-financed primary data for policy-relevant analysis. The Challenge was intended to facilitate broader use of MCC’s US-taxpayer funded data. Due to the success of the first Open Data Challenge, a second Open Data Challenge was launched in February 2016 in order to encourage innovative ideas and maximize the use of data that MCC finances for its independent evaluations. For the second Open Data Challenge, MCC announced $1000 prizes for the best papers from students around the world whose work contributed significantly to agency learning and innovation. By August 31, 2017, MCC will announce up to six selected papers and begin planning for public presentation of papers at MCC.
- In 2014, MCC launched an internal “Solutions Lab” that was designed to encourage innovation by engaging staff to come up with creative solutions to some of the biggest challenges MCC faces. To further encourage staff who pursue innovative ideas throughout the compact lifecycle, MCC launched the annual MCC Innovation Award. (MCC promotes agency-wide participation in its Solutions Lab through an internal intranet portal. MCC’s new Innovation Award is a part of the Agency’s Annual Awards Ceremony held each summer.) The Innovation Award recognizes individuals who demonstrate “exemplary” leadership integrating innovation in project design, project implementation, and/or systems functionality and efficiency. Selections for the Innovation Award are based on a demonstrated ability to lead and implement innovative strategies from project conception that foster sustained learning and collaboration and add value to MCC and/or country partnerships. Award nominees must meet at least one of the following criteria:
- Integrating innovation in projects which rigorously assess and scale up pilot projects or new technologies tested by other development institutions, universities, research institutions businesses or non-profit organizations;
- Implementing projects or initiatives which enhance MCC’s ability to effectively carry out its mission, to include introduction and adoption of new technologies in MCA countries, implementation of transformational IT projects, and/or improvement to needed system controls for enhanced functionality or compliance; or
- Leading new types of collaborative partnerships to leverage MCC resources and bolster the development impact and sustainability of projects in MCA countries.
- The PEPFAR/MCC Partnership has an innovation challenge. The Data Collaboratives for Local Impact Innovation Challenge identifies, supports, and involves country-based youth, developers, programmers, and solution providers through targeted challenge competitions that address specific programmatic challenges with data — for example, engaging with communities to lower their risk of contracting HIV/AIDS. In December 2016, the Data Lab also held a mapathon that focused on Tanzanian districts where vulnerable adolescent girls and young women are susceptible to HIV infection. By improving maps, volunteers contributed directly to valuable datasets that can inform the understanding of HIV/AIDS prevalence in priority districts, as well as health facility and antiretroviral drug accessibility. DCLI also hosted a mapathon in Washington in July 2017, bringing together more than 100 people from the US government, the Data Lab, and the private sector to map over 2,000 kilometers of roads and nearly 4,000 buildings in Côte d’Ivoire and Togo, two of MCC’s partner countries. The information is now public and can be used by governments, donors, and investors to make better data-driven decisions when planning for critical services like health clinics, schools, and utilities.
- MCC is conducting an “Innovation Grant Program” in Zambia in order to encourage local innovation in pro-poor service delivery in the water sector through grants to community-based organizations, civil society, and/or private sector entities.
- MCC recently launched an evaluation of an innovative infrastructure grant facility in Cabo Verde to understand whether funding opportunities could incentivize utilities to reform and improve the sustainability of infrastructure investments. This evaluation represents MCC’s commitment to innovate new approaches and use evidence to test whether those new approaches produce results.
- MCC regularly engages in implementing pilot projects as part of its overall Compact programs. A few examples include: 1) in Morocco, an innovative pay for results (PFR) mechanism to replicate or expand proven programs that provide integrated support including short-term (one to six months) job readiness skills training, technical training, job matching, follow-up to ensure longevity, and other services; 2) a “call-for-ideas” in Benin in 2015 that extended an invitation to interested companies and organizations from around the world to submit information regarding potential projects that would expand access to renewable off-grid electrical power in Benin; and 3) a regulatory strengthening project in Sierra Leone that includes funding for a results-based financing system designed to strengthen the regulator’s role, incentivize performance by the utilities, and enhance accountability.
- MCC has signed a five-year, $450m grant with the Kingdom of Morocco, called the Morocco Employability and Land Compact. The focus of the Compact is on making improvements toward land productivity and employability to create new economic opportunities, improve workforce skills, and strengthen the business environment. The Labor Market Impact Evaluation Lab is a key component of the compact aimed at improving labor market outcomes through the use of rigorous quantitative research. The Lab will finance rigorous impact evaluations and other rigorous empirical studies, as well as policy-research engagements, to build the capacity of the Moroccan government to commission/generate such studies. This is the first time MCC has pursued a country-led Lab focused on impact evaluations and policy research.
Score
8
8
Substance Abuse and Mental Health Services Administration
- The SAMHSA Knowledge Network, a collection of technical assistance and training resources provided by the agency, provides behavioral health professionals with education and collaboration opportunities, and ample tools and technical assistance resources that promote innovation in practice and program improvement. Located within the Knowledge Network are groups such as the Center for Financing Reform and Innovation, which works with states and territories, local policy makers, providers, consumers, and other stakeholders to promote innovative financing and delivery system reforms.
- In addition, SAMHSA participates in collaborations with other HHS agencies to promote innovative uses of data, technology and innovation across HHS to create a more effective government and improve the health of the nation, via the HHS IDEA Lab. SAMHSA has co-developed and submitted several innovative data utilization project proposals to the Ignite Accelerator of the HHS IDEA Lab, such as a project to monitor and prevent opioid overdoses by linking heroin users to resources and information.
- The agency is currently exploring the use of tiered-evidence frameworks in its award decision-making to actively encourage innovation at the grantee/program level. In addition, pursuant to the 21st Century Cures Act, SAMHSA is establishing the National Mental Health and Substance Use Policy Laboratory (Policy Lab) (Sec. 7001, p.501), by restructuring the current Office of Policy, Planning, and Innovation (OPPI). The new Policy Lab will review programs and activities operated by the agency to identify programs and activities that are duplicative, identify programs and activities that are not evidence-based or effective, and formulate recommendations for coordinating, elimination, or improving such programs (Sec 7001, pp.502-503).
- To further promote innovation, per the Cures Act, SAMHSA’s Assistant Secretary may coordinate with the Policy Lab to award grants to states, local governments, tribes and tribal organizations, and other eligible organizations to develop evidence-based interventions. These grants can help support the evaluation of models and interventions that show promise, or the expansion, replication, or scaling up of interventions that have been established as evidence-based (Sec. 7001, pp. 503-504).
Score
9
9
U.S. Agency for International Development
- USAID established the U.S. Global Development Lab (the Lab) in 2014 to increase the application of science, technology, innovation, and partnerships to extend the Agency’s development impact in helping to end extreme poverty. The Lab does this by working closely with colleagues across the Agency and by bringing together a diverse set of partners to discover, test, and scale breakthrough innovations to solve development challenges faster, cheaper and more sustainably. The Lab is the home for the Monitoring, Evaluation, Research and Learning Innovations program (MERLIN) to source, co-design, implement and test solutions that innovate on traditional approaches to monitoring, evaluation, research and learning.
- In the past six years, through the Global Development Lab, USAID and its partners have launched eight Grand Challenges for Development: Saving Lives at Birth (2011), All Children Reading (2011), Powering Agriculture: An Energy Grand Challenge for Development (2012), Making All Voices Count (2012), Securing Water for Food (2013), Fighting Ebola (2015), Combating Zika and Future Threats (2016), Scaling Off-Grid Energy (2016), and Ensuring Effective Health Supply Chains (2017). GCDs are robust partnerships that leverage each partner’s strengths to engage new solvers through incentive prizes, challenge grant funding, and crowdsourcing to capture learnings, acceleration support services, and generate awareness to identify the most promising solutions, test them, and scale those that are proven to work.
- Development Innovation Ventures (DIV) is USAID’s tiered, evidence-based innovation fund. It awards grant financing to innovative solutions to development challenges on the basis of rigorous evidence of impact, cost-effectiveness, and potential to scale via the public and/or private sectors. It awards funding across three stages, ranging from under $100,000 for piloting early-stage innovations to up to $15 million for scaling evidence-backed innovations. The DIV model is designed to find breakthrough solutions, minimize risk and maximize impact through staged financing, rigorously test impacts and cost-effectiveness, and scale proven solutions. To date, DIV-supported innovations have impacted 12.6 million beneficiaries and mobilized external financing of $446 million, a nearly 5:1 leverage ratio on DIV’s approximately $90 million in grants awarded. Of the over 90 DIV grants that have been completed, over 90% of grantees collected evidence of impact and more than 60% of innovations had attributable, causal positive impact on development outcomes. (In July 2017, USAID announced it would no longer be accepting applications for new DIV awards, until further notice.)
- USAID also supports innovation through the external Global Innovation Fund (GIF), a private fund co-founded by USAID and based on the DIV model. Like DIV, GIF invests in social innovations to improve the lives of millions of people in the developing world, but, as a private fund, GIF is also able to provide debt and equity financing.
- USAID is working with the Australian Department of Foreign Assistance and Trade (Australian Aid), the Korea International Cooperation Agency (KOICA), and the Bill & Melinda Gates Foundation to fund and promote the Global Innovation Exchange (the Exchange), a free, online platform that connects social entrepreneurs with the funding and other resources they need to be impactful. Already home to nearly $123.8 million funding opportunities, more than 5,500 innovations, and more than 18,500 users, the Exchange is rapidly expanding, reaching nearly every corner of the world.
- The Innovation Design and Advisory Team (iDesign) helps advance USAID’s culture of innovation and intrapreneurship through testing, application and mainstreaming of innovative design and problem-solving processes. iDesign offers customized advisory services and training programs to help Agency offices determine when, how, and with whom to innovate in programs to achieve more cost-effective impact. It applies the latest models for open and collaborative program design, human-centered design and systems thinking, pay for results tools, venture fund models and lean and adaptive management principles.
- USAID’s Applied Innovation team works with programs and implementing partners, including contractors and grantees, to capture learning and accelerate innovations supported by USAID. The Applied Innovation team is working to expand innovation adoption across USAID’s programming, and test the theory that innovations can enhance development impact, save time and resources, and improve programmatic efficiencies.
- USAID’s Higher Education Solutions Network (HESN) program is a partnership with seven competitively awarded universities working with partners worldwide. Leveraging nearly equal investments from each institution, the universities have established eight Development Labs that have built a global research network of 685 partners from 69 countries. Through HESN, USAID has been able to harness the ingenuity of students and faculty to create or test over 300 innovations, which have helped USAID missions reach their development goals and improved the lives of 2.3 million beneficiaries.
- USAID’s Research and Development (R&D) Hub for Monitoring and Evaluation (M&E) helps agency staff discover when emerging M&E approaches may be appropriate for their particular learning needs, working within the space of complexity-aware M&E, context monitoring, monitoring without indicators, and M&E for adaptive management. While champions in USAID and implementing partners have been experimenting with emerging approaches, evidence-based, practical resources on how to apply these approaches have not been systematically created and shared. The R&D Hub for M&E in LER plays the role of connector by linking champions and conducting research on and documenting emerging M&E approaches that have been helpful in various circumstances. The R&D Hub collects data on the use of innovative approaches to M&E within the agency; the US Global Development Lab’s MERLIN is one example of a partner in learning.
- USAID’s Research and Development (R&D) Hub for Monitoring and Evaluation (M&E) helps agency staff determine best fit for emerging M&E approaches to specific contexts and programs.
Score
8
8
U.S. Department of Education
- The Investing in Innovation program (i3) is ED’s signature innovation program for K–12 public education. i3 grants are focused on validating and scaling evidence-based practices, and to encourage innovative approaches to persistent challenges. These “Development” grants are the most prevalent type of i3 grant, comprising 115 out of the 172 i3 grants, and 10 of the 15 new i3 grants made in FY16. i3’s successor program is the Education Innovation and Research (EIR) program, which published notices for the three tiers of grants under that program in December 2016 and will make awards in September 2017. One unique aspect of the EIR notices for the Mid-phase and Expansion tiers was the inclusion of a field-initated innovation priority that allowed applicants to propose a project of their choosing, as long as it met the evidence requirement for that competition.
- ED is participating in the Performance Partnership Pilots for Disconnected Youth initiative. These pilots give state, local, and tribal governments an opportunity to test innovative new strategies to improve outcomes for low-income disconnected youth ages 14 to 24, including youth who are in foster care, homeless, young parents, involved in the justice system, unemployed, or who have dropped out or are at risk of dropping out of school.
- ED is continuing to promote the use of data in innovative ways by engaging with developers. This includes launching a new Developer Hub and GitHub platform, which provides developers with needed information and resources, and the creation of new APIs. Additionally, ED continues to be an active participant on the Opportunity Project, which encourages the use of federal data for social good by providing a process for developers, data enthusiasts, policy leaders, and communities to co-create innovative tech solutions that expand opportunity.
- The White House Social and Behavioral Sciences Team conducted several behavioral science studies related to ED’s work, including looking at the impact of text message reminders for students about key tasks related to college matriculation, such as completing financial aid paperwork, and about notices to student borrowers about income-driven repayment plans.
- ED is currently implementing the Experimental Sites Initiative to assess the effects of statutory and regulatory flexibility for participating institutions disbursing Title IV student aid.
- The IES Research Grants Program supports the development and iterative testing of new, innovative approaches to improving education outcomes. IES makes research grants with a goal structure. “Goal 2: Development and Innovation” supports the development of new education curricula; instructional approaches; professional development; technology; and practices, programs, and policies that are implemented at the student-, classroom-, school-, district-, state-, or federal-level to improve student education outcomes.
- On behalf of ED, IES also administers the Small Business Innovation Research (SBIR) program, which competes funding to small business that propose developing commercially viable education technology projects that are designed to support classroom teaching and student learning. Projects must go through an iterative development process and conduct research to determine promise of effectiveness.
- ED has funded a number of tools in FY16 and FY17 to support innovation and rigorous evaluation in the field, including:
- RCT-YES is a free software tool that uses cutting-edge statistical methods to help users easily analyze data and report results from experimental and quasi-experimental impact studies of education programs.
- Downloadable programs help users build logic models and create ongoing plans for monitoring, measuring, and tracking outcomes over time to measure program effectiveness.
- A guide for researchers on how to conduct descriptive analysis in education to help identify and describe trends in populations, create new measures, or describe samples in studies aimed at identifying causal effects.
- The Ed Tech Rapid Cycle Evaluation Coach, a free online tool that helps users plan, conduct, and report findings from experimental and quasi-experimental impact studies of education technology products. The tool is optimized for non-technical users and employs unique statistical methods that allow findings to be presented.
- ED is implementing a number of Pay for Success projects in FY17:
- Career and Technical Education (CTE): $2 million to support the development of PFS projects to implement new or scale up existing high-quality CTE opportunities.
- English Language Acquisition: $293,000 contract to conduct a feasibility study that will identify at least two promising school sites that are using evidence-based interventions for early learning dual language models where a PFS project could take shape to help scale the interventions to reach more students those who need them.
- Early Learning: $3 million for Preschool Pay for Success feasibility pilots to support innovative funding strategies to expand preschool and improve educational outcomes for 3- and 4- year-olds. These grants will allow states, school districts and other local government agencies to explore whether Pay for Success is a viable financing mechanism for expanding and improving preschool in their communities.
- Technical Assistance: The Office of Special Education Programs is collaborating with early childhood technical assistance centers to educate and build capacity among state coordinators in IDEA Part C and Part B to explore using PFS to expand or improve special education services for young children. In addition, we have conducted a Pay for Success webinar series for the Comprehensive Centers.
Score
8
8
U.S. Dept. of Housing & Urban Development
- HUD’s Policy Development and Research (PD&R) office is conducting a number of evaluated, random-assignment program demonstrations to test new program models, which can be found in PD&R’s biennial report: the Family Self-Sufficiency Demonstration, Pre-Purchase Homeownership Counseling Demonstration, Support and Services at Home (SASH) Demonstration for elderly households, Supportive Services Demonstration for health services in elderly housing, Rent Reform Demonstration, and the Small Area Fair Market Rent Demonstration. The latter demonstrations are in early or middle stages; interim results and long-term follow-up results generally will be reported through HUD USER during the next 2–4 years.
- PD&R also is collaborating with the General Services Administration and U.S. Department of Education to link tenant data with records of students and individuals submitting Free Applications for Federal Student Aid. Through this partnership, three phases of low-cost behaviorally informed experiments are being conducted to increase access of HUD tenants to higher education, through mailed outreach communications (see p.12), mailed plus electronic communications, and HUD-funded education navigators.
- PD&R houses the Office of International and Philanthropic Innovation, and administers five types of Secretary’s Awards to encourage excellence: Public-Philanthropic Partnerships, Opportunity and Empowerment, Healthy Homes, Historic Preservation, and Housing and Community Design. The competitions are judged by juries of professionals, and bring visibility to the nation’s most compelling solutions for addressing housing and community development challenges.
- PD&R sponsors an Innovation in Affordable Housing Competition to engage multidisciplinary teams of graduate students in addressing a specific housing problem developed by an actual public housing agency. The competition increases the nation’s future human capacity to address the affordable housing crisis by exposing future designers, administrators, and policymakers to real-world challenges of a specific legal and community context, with their proposals to be evaluated by an expert jury.
- In FY16, HUD’s National Disaster Resilience Competition provided funding for resilient housing and infrastructure projects to states and communities that suffered major disasters. Collaborative teams were assisted in extensively researching and developing their proposals by nine Resilience Academies developed by the Rockefeller Foundation in partnership with HUD. The in-depth, juried process helped ensure that the $1 billion of resources available for these communities will result in more resilient housing and infrastructure and bridge the gap between social and physical vulnerabilities.
- HUD promotes innovation among staff members through social media sites, and in PD&R, through staff-led Knowledge Collaboratives that focus on selected policy or technical fields to share knowledge and work together on in-house research projects (for examples of in-house work, see footnote 8 in Roadmap Update).
Score
8
8
U.S. Department of Labor
- DOL is participating in the Performance Partnership Pilots (P3) for innovative service delivery for disconnected youth which includes not only waivers and blending and braiding of federal funds, but gives bonus points in application reviews for proposing “high tier” evaluations. DOL is the lead agency for the evaluation of P3.
- DOL recently completed a number of behavioral science evaluations—three in unemployment insurance, two in OSHA, one in OFCCP, and one in EBSA for pension contributions (see the CEO website), and is in the planning phase for a number of others. For more DOL’s work in this area, see DOL’s Behavioral Insights page.
- DOL is using Job Corps’ demonstration authority to test and evaluate innovative and promising models that could improve outcomes for youth. In 2016 DOL awarded a contract for a Job Corps pilot program, the Cascades Job Corps College and Career Academy. The pilot will test alternative strategies for the operation of Job Corps for younger participants (ages 16 and 21). Past research on the program showed that while Job Corps increased the education and earnings of program participants, it was more beneficial for youth over age 20 than for its younger participants. This pilot uses DOL’s demonstration authority to test innovative and promising strategies (which include using a career pathway program approach of integrated academic and technical training, workforce preparation skills and support services) to better meet the needs of this population. CEO is sponsoring a rigorous impact evaluation to examine the effectiveness of the pilot. See the study overview here.
- DOL has two pilot projects testing use of a Pay for Success (PFS) financing model. In 2013, DOL awarded 2 grants of approximately $12 million to two states, Massachusetts and New York, to test a PFS model where private for-profit and philanthropic investors pay the up-front costs of delivering an intervention designed to achieve specific outcomes within a given timeframe. In return for accepting the risk of funding the project, the investors may expect a return if the project is successful; however, repayment by the government is contingent on the validated achievement of results. Both pilots are employing a random assignment methodology to measure results, which are due in 2017. DOL is sponsoring a process study to document project implementation and provide information on the PFS approach for policymakers and program administrators. The first report from this study was released in 2016, documenting the development of pilots and first year of implementation. A second report will document the pilots’ longer-term operational experiences, including the extent to which the pilots achieved their performance milestones.
- DOL has invested more than $90 million through the ApprenticeshipUSA initiative – a national campaign bringing together a broad range of stakeholders, including employers, labor, and States, education and workforce partners, to expand and diversify Registered Apprenticeship in the United States. This includes more than $60 million for State-led strategies to grow and diversify apprenticeship, and State Accelerator Grants to help integrate apprenticeship into education and workforce systems; engage industry and other partners to expand apprenticeship to new sectors and new populations at scale; conduct outreach and work with employers to start new programs; promote greater inclusion and diversity in apprenticeship; and develop statewide and regional strategies aimed at building state capacity to support new apprenticeship programs. All of these grants include funding for data collection; additionally, ETA and CEO are conducting an evaluation of American Apprenticeship Initiative.
- CEO received grant making authority in 2016. In January of 2017, we awarded 9 research grants aim at supporting university-based research of workforce policies and programs. The goal is to build capacity and drive innovation among academic researchers to answer questions that will provide insight into labor policies and programs.