2017 Federal Index
Resources
Did the agency invest at least 1% of program funds in evaluations in FY17? (Examples: Impact studies; implementation studies; rapid cycle evaluations; evaluation technical assistance, and capacity-building)
Score
7
7
Administration for Children and Families (HHS)
- In FY17, ACF plans to spend $162 million on evaluations, representing 0.3% of ACF’s $56.5 billion budget in FY17 (in addition to investments in evaluations by ACF grantees). This spending includes a $30 million increase in welfare research funds appropriated by Congress. The amount of ACF’s spending on evaluation is largely determined by Congress. For example, in FY 2017, Congress designated 0.33% of the Temporary Assistance for Needy Families Block Grant for research, evaluation, and technical assistance.
Score
7
7
Corporation for National and Community Service
- CNCS plans to spend a total of $6.3 million, representing .6% of CNCS’s $1 billion budget in FY17, on evaluation and evaluation capacity building activities (R&E evaluation and program funds combined), including:
- $540,375 of FY17 evaluation funds will be awarded in grants to universities that focus on the economic and social outcomes of the AmeriCorps program (continuation dollars);
- $1,617,525 of FY17 AmeriCorps funds have been awarded in Commission Investment Fund grants for building commission capacity to support grantees (formula grantees) in the areas of evaluation and performance measurement;
- $3.14 million of FY17 Senior Corps funds will be spent on evaluation and evidence-building activities (representing 1.6 percent of Senior Corps’ $202.11 million FY17 budget); and
- $1,000,797.70 of FY17 evaluation funds have been awarded for AmeriCorps grantee evaluation capacity building.
Score
10
10
Millennium Challenge Corporation
- MCC’s FY17 investment in monitoring and evaluation to date is $15.1 million, which amounts to 5% of Compact spending for FY17 ($302.5 million). This investment exceeds the FY16 spending in which MCC invested over $14.4 million in M&E, roughly 3.4% of Compact spending for FY16 ($428.2 million).
Score
9
9
Substance Abuse and Mental Health Services Administration
- SAMHSA will have spent $31.5 million on sixteen evaluations by the end of FY2017. This represents .8 percent of SAMHSA’s $3.78 billion budget appropriated for FY2016.
Score
10
10
U.S. Agency for International Development
- In FY 2016, USAID missions and offices reported completing 145 evaluations with resources totaling approximately $49 million and managing another 323 ongoing evaluations, many that span more than one year, with total ongoing budgets estimated to reach $201 million. Overall spending on evaluations completed or ongoing in FY16 ($250 million) represents about 1.1% of USAID’s $23 billion FY16 program budget.
- This amount does not include the budget for the Office of Learning, Evaluation, and Research which primarily focuses on monitoring, evaluation, and learning capacity building and technical assistance ($16 million FY 2016) or the investment in the Demographic and Health Surveys (DHS) ($189 million total in FY13-FY18) or surveys funded by other sector programs that often make up some of the underlying data used in many evaluations.
Score
7
7
U.S. Department of Education
- Overall spending on evaluation ($144 million in FY16) and evaluation technical assistance and capacity-building ($58 million in FY16) represents 0.4% of ED’s $45.6 billion discretionary budget (without Pell Grants) in FY16. (FY17 estimates are forthcoming.) This total includes impact studies, implementation studies, rapid cycle evaluations, evaluation technical assistance, and capacity-building funded through IES and OII.
- ED spent about $144 million on program evaluations in FY16, including funding evaluation projects under IES’ research grant programs. It is important to note that the timing of evaluation projects and the type of research projects proposed by the field results in year-to-year fluctuations in this amount, and does not reflect a change in ED’s commitment to evaluation.
- It is worth noting that much of ED’s investment in evaluation is directed toward supporting states and school districts so that they can conduct studies of their own education policies and programs. For example, IES runs annual grant competitions to support researcher-practitioner partnerships between state and local education officials and research institutions, including a new program for low-cost, short duration evaluations. IES also provides large grants to states to develop longitudinal data systems for purposes of tracking students through the educational system and conducting state-sponsored evaluations. (To date 47 states have received IES funding for this purpose.) Finally, the Regional Education Laboratories provide extensive technical assistance on evaluation and support research alliances that conduct implementation and impact studies on education policies and programs in ten geographic regions of the U.S., covering all states, territories, and the District of Columbia.
- OII emphasizes evaluation and the building of evidence in a number of its grant programs through requirements that grantees conduct or participate in evaluations. This emphasis on evaluation continues even for programs supported by an existing evidence base, as the evaluation design for these programs looks at impact with new settings, different populations, and project-specific implementation. In many instances, during grant competitions, this evaluation criterion is reviewed and scored by evaluation experts familiar with the What Works Clearinghouse standards, ensuring proposed evaluation plans are of the appropriate design and rigor. Additionally, a number of OII programs dedicate resources to evaluation technical assistance and program level evaluations. The Investing in Innovation (i3) and the Supporting Effective Educator Development programs utilize technical assistance contractors that support grantees in the design and implementation of their project-level evaluations, as well as the development of evaluation resources that are now being shared publicly on the What Works Clearinghouse and program websites.
- IES and OII grantees are expected to make the results of their evaluations public through Education Resources Information Clearinghouse (ERIC) and other grant-funded dissemination activities. In addition, all impact evaluations funded by IES and OII are reviewed by the What Works Clearinghouse (WWC), which plays a major role in summarizing and disseminating findings from the most rigorous studies to ED and the broader field.
- In addition, some programs encourage their grantees to conduct project-level evaluations. One of the key lessons from the Investing in Innovation program (i3) has been that high-quality technical assistance for grantees on project-level evaluations is critical to producing credible information on project outcomes. In FY16 i3 invested more than $4 million of its appropriation in evaluation technical assistance, and the Regional Educational Laboratories continued to provide States and districts with technical assistance on evaluations and data use – virtually no other competitive grant programs at ED have the authority or means to fund such a robust vehicle for technical assistance. ED is working to identify and align its technical assistance efforts on the use of evidence in education at the state and local level.
- ED has an opportunity to significantly increase its annual investment in program evaluation through the reauthorized ESEA pooled evaluation authority, which makes available up to $40 million annually from ESEA programs funded by Congress that can be used to evaluate any ESEA program included in the biennial evaluation plan prepared by IES.
Score
6
6
U.S. Dept. of Housing & Urban Development
- In FY17, HUD plans to spend $89 million on evaluations, representing 0.19% of HUD’s $48 billion discretionary budget in FY17.
- For FY17, Congress appropriated $89 million for the Office of Policy Development and Research’s (PD&R’s) Research & Technology account, including $50 million for core research activities; $16 million for research, evaluations, and demonstrations; and $23 million for technical assistance. The total represents an FY17 investment in evaluations and evidence amounting to 0.19 percent of HUD’s $48 billion gross discretionary budget authority for FY17. The funding for core research is used primarily for the American Housing Survey, other surveys, and data acquisition that indirectly support evaluation of HUD’s mission activities in domains such as affordable housing and housing finance.
- PD&R’s FY17 appropriation of $24 million for Salaries and Expenses also supports evidence in the form of PD&R’s in-house research and evaluation program, economic analyses, data linkage initiatives, and management of housing surveys and contract research and evaluation.
Score
8
8
U.S. Department of Labor
- In FY 17, DOL’s CEO will directly oversee an estimated $40 million in evaluation funding. This represents approximately 0.3% of DOL’s FY17 discretionary budget of $12.8 billion. However in addition to this, many DOL evaluations are supported by funds appropriated to DOL programs and/or are carried out by other offices within DOL. For example, in addition to CEO, most agencies and program offices conduct and support evaluation activities with their dollars. Further, several DOL agencies also have separate evaluation appropriations. DOL studies funded through individual agencies and program offices also coordinate with DOL’s CEO. DOL’s Chief Evaluation Office directly funds evaluations and also combines CEO funds with agency funds to jointly sponsor some evaluations.
- In many areas where DOL is undertaking evaluation activities, the evaluation budget far exceeds 1% of the budget for the program (e.g., the budgets for the evaluations of a number of recent grant programs are between 3% and 5% of the programmatic budget).
- The Administration’s FY14-FY17 budget requests recommended allowing the U.S. Secretary of Labor to set aside up to 1% of all operating agencies’ budgets for evaluations, coordinated by CEO. In FYs 2012-2015, Congress authorized the Secretary to set aside up to 0.5% of these funds for evaluations, in addition to the separate evaluation funds that exist in many DOL agencies. In FY16 and FY17, Congress authorized DOL to set aside up to 0.75% of operating agency budgets for use by CEO for evaluations.
- The development of States’ capacity to conduct evaluations is a long-range and iterative process. Significant resources are dedicated to aiding these efforts through various forms of technical assistance and guidance. A primary goal initially will be to enhance capacity by building knowledge among State staff regarding various methodologies, approaches for enlisting expertise, and the potential role of evaluations and research in meeting State goals and priorities. During FY17, ETA hosted three National Convenings in Washington, DC, Dallas TX, and San Diego, CA. ETA included sessions on Research and Evaluation under WIOA in coordination with the performance accountability sessions. More technical assistance is planned.
- In order to support the workforce system in being prepared to meet evaluation and reporting requirements under their WIOA grants, DOL has invested in evaluation and evaluation capacity-building activities, including:
- DOL has launched a Workforce Innovation and Opportunity Act implementation study to document and describe how critical state activities under WIOA are being implemented, particularly the core programs in titles I and III, and identify possibly areas on which further technical assistance, guidance, or policies might be needed in order to help states implement the law. The WIOA state implementation study will build on both past and ongoing studies, including: 1) the ongoing random assignment evaluation of the adult and dislocated worker programs under the Workforce Investment Act, 2) studies involving data analytics on workforce program services, 3) an ongoing institutional analysis of AJCs, and 4) an ongoing study on AJC customer experiences.
- In response to the WIOA’s requirement to measure effectiveness in serving employers as a primary indicator of performance, DOL is piloting three approaches designed to gauge three critical workforce needs of the business community: (1) Retention with the same employer; (2) Repeat business customers; and (3) Employer Penetration Rate. States must select two of the three approaches to pilot, and may develop an additional State-specific approach. DOL will evaluate State experiences with the various approaches and plan to identify a standardized indicator that the Departments anticipate will be implemented no later than the beginning of Program Year 2019.