Did the agency have a senior staff member(s) with the authority, staff, and budget to evaluate its major programs and inform policy decisions affecting them in FY17? (Example: Chief Evaluation Officer)

• HUD’s Office of Policy Development & Research (PD&R) informs HUD’s policy development and implementation by conducting, supporting, and sharing research, surveys, demonstrations, program evaluations, and best practices. PD&R achieves this mission through three interrelated core functions: (1) collecting and analyzing national housing market data (including with the Census Bureau); (2) conducting research, program evaluations, and demonstrations; and (3) providing policy advice and analytic support to the HUD Secretary and program offices. PD&R is led by an Assistant Secretary who oversees six offices, about 149 staff including a team of field economists that work in HUD’s 10 regional offices across the country, and a budget of $113 million in FY17. The Assistant Secretary ensures that evidence informs policy development through frequent personal engagement with other principal staff, the Secretary, and external policy officials; HUDstat performance review meetings (see Question #4 below for a description); speeches to policy audiences, sponsorship of public research briefings, and policy implications memoranda. The Assistant Secretary also regularly engages with each HUD program office to ensure that metrics, evaluations, and evidence inform program design, budgeting, and implementation.
• Periodic PD&R meetings with program offices enable PD&R to share knowledge about evaluation progress and program offices to share knowledge about emerging needs for research, evaluation, and demonstrations to advance program policy.

Did the agency have an evaluation policy, evaluation plan, and research/learning agenda(s) and did it publicly release the findings of all completed evaluations in FY17?

• HUD’s Office of Policy Development and Research (PD&R) has published an evaluation policy that establishes core principles and practices of PD&R’s evaluation and research activities. The six core principles are rigor, relevance, transparency, independence, ethics, and technical innovation.
• PD&R’s evaluation policy guides HUD’s research planning efforts, known as research roadmapping. Key features of research roadmapping include reaching out to internal and external stakeholders through a participatory approach; making research planning systematic, iterative, and transparent; driving a learning agenda by focusing on research questions that are timely, forward-looking, policy-relevant, and leverage HUD’s comparative advantages and partnership opportunities; and aligning research with HUD’s strategic goals and areas of special focus. HUD also employs its role as convener to help establish frameworks for evidence, metrics, and future research.
• HUD’s original “Research Roadmap FY14-FY18” and “Research Roadmap: 2017 Update” constitute the core of HUD’s learning agenda. The roadmaps are strategic, five-year plans for priority program evaluations and research to be pursued given a sufficiently robust level of funding. PD&R also integrated its evaluation plan into HUD’s FY14-FY18 Strategic Plan (see pp. 57-63) to strengthen the alignment between evaluation and performance management. During FY16, PD&R used similar principles and methods to refresh the Roadmap to address emerging research topics. PD&R’s fiscal year budget requests include annual research plans drawn from the Roadmap. Actual research activities are substantially determined by Congressional funding and guidance.
• The Research Roadmap serves as a long-term evaluation plan and the core of HUD’s learning agenda. HUD also develops annual evaluation plans, consisting of a list of specific research priorities, as requested by Congress.
• PD&R’s policy (p.87950) is to publish and disseminate all evaluations that meet standards of methodological rigor in a timely fashion. Additionally, PD&R includes language in research and evaluation contracts that allows researchers to independently publish results, even without HUD approval, after not more than 6 months. PD&R has occasionally declined to publish reports that fell short of standards for methodological rigor. Completed evaluations and research are summarized in HUD’s Annual Performance Report (see pp.123–131) at the end of each fiscal year, and reports are posted on PD&R’s website, HUDUSER.gov.

Did the agency invest at least 1% of program funds in evaluations in FY17? (Examples: Impact studies; implementation studies; rapid cycle evaluations; evaluation technical assistance, and capacity-building)

• In FY17, HUD plans to spend $89 million on evaluations, representing 0.19% of HUD’s $48 billion discretionary budget in FY17.
• For FY17, Congress appropriated $89 million for the Office of Policy Development and Research’s (PD&R’s) Research & Technology account, including $50 million for core research activities; $16 million for research, evaluations, and demonstrations; and $23 million for technical assistance. The total represents an FY17 investment in evaluations and evidence amounting to 0.19 percent of HUD’s $48 billion gross discretionary budget authority for FY17. The funding for core research is used primarily for the American Housing Survey, other surveys, and data acquisition that indirectly support evaluation of HUD’s mission activities in domains such as affordable housing and housing finance.
• PD&R’s FY17 appropriation of $24 million for Salaries and Expenses also supports evidence in the form of PD&R’s in-house research and evaluation program, economic analyses, data linkage initiatives, and management of housing surveys and contract research and evaluation.

Did the agency implement a performance management system with clear and prioritized outcome-focused goals and aligned program objectives and measures, and did it frequently collect, analyze, and use data and evidence to improve outcomes, return on investment, and other dimensions of performance in FY17? (Example: Performance stat systems)

• HUD conducts regular data-driven performance reviews—“HUDStat” meetings—that focus on quarterly progress toward achieving each of HUD’s priority goals. The HUD Secretary and senior leadership from throughout the agency, and sometimes from partner agencies, attend these meetings to address challenges, review metrics, improve internal and external collaboration, and increase performance. A new strategic framework is being developed in FY17 as provided by OMB Circular A-11 (see section 200.23). HUD documents alignment between strategic goals and supporting objectives and metrics in the Annual Performance Plan and Annual Performance Report, and identifies the staff assigned lead responsibility for each objective.

Did the agency collect, analyze, share, and use high-quality administrative and survey data – consistent with strong privacy protections – to improve (or help other entities improve) federal, state, and local programs in FY17? (Examples: Model data-sharing agreements or data-licensing agreements; data tagging and documentation; data standardization; open data policies)

• The HUDUSER.gov web portal continues to provide researchers, practitioners, and the public with PD&R datasets including the American Housing Survey, HUD median family income limits and Fair Market Rents, and Picture of Subsidized Households tabulations of administrative tenant records at multiple geographic levels. HUD sponsors custom tabulations of American Community Survey data that make standard adjustments of household incomes and units for household size to enable researchers and practitioners to analyze state and local housing needs. HUD provides researchers with microdata from experimental program demonstrations and research initiatives on topics such as housing discrimination, the HUD-insured multifamily housing stock, and the public housing population. To help users identify which data are useful to them, reference guides identify datasets and characterize their relevance and usefulness for research in designated categories. HUD partnered with the Census Bureau to enhance public access to the American Housing Survey with a new custom table creator and infographics to summarize results.
• HUD’s Office of Policy Development and Research (PD&R) has authority to enter into cooperative agreements with research organizations, including both funded Research Partnerships and unfunded Data License Agreements, to support innovative research projects that leverage HUD’s data assets and inform HUD’s policies and programs. A dedicated subject-matter expert is available to answer questions for those seeking a data license. Data licensing protocols ensure that confidential information is protected.
• PD&R partnered with the National Center for Health Statistics at the Centers for Disease Control to link HUD administrative data for assisted renters with respondents to two national health surveys and made the linked data available to researchers to begin building a picture of tenant health issues. Data access through the research data center ensures that confidential information is protected.
• A PD&R partnership with the Department of Education to link administrative records for assisted renters and student aid applications is supporting three low-cost random-assignment experiments that test the impact of various nudges and informational supports on application completion and college attendance.
• HUD is involved in a wide array of data-sharing agreements described under Data Infrastructure in the Roadmap Update (see pp.52–56). Notably, HUD and the Census Bureau have entered an interagency agreement for the Bureau’s Center for Administrative Records Research and Applications (CARRA) to link data from HUD’s tenant databases and randomized control trials with the Bureau’s survey data collection and other administrative data collected under its Title 13 authority. These RCT datasets are the first intervention data added to Federal Statistical RDCs by any federal agency, and strict protocols ensure that confidential information is protected.

Did the agency use a common evidence framework, guidelines, or standards to inform its research and funding decisions and did it disseminate and promote the use of evidence-based interventions through a user-friendly tool in FY17? (Example: What Works Clearinghouses)

• HUD’s Policy Development and Research (PD&R) office provides evidence of “what works” primarily through HUD USER, a portal and web store for program evaluations, case studies, and policy analysis and research; the Regulatory Barriers Clearinghouse; and through initiatives such as Innovation of the Day, Sustainable Construction Methods in Indian Country, and the Consumer’s Guide to Energy-Efficient and Healthy Homes. This content is designed to provide current policy information, elevate effective practices, and synthesize data and other evidence in accessible formats. Through these resources, researchers and practitioners can see the full breadth of work on a given topic (e.g., rigorous established evidence, case studies of what’s worked in the field, and new innovations currently being explored) to inform their work.

Did the agency have staff, policies, and processes in place that encouraged innovation to improve the impact of its programs in FY17? (Examples: Prizes and challenges; behavioral science trials; innovation labs/accelerators; performance partnership pilots; demonstration projects or waivers with strong evaluation requirements)

• HUD’s Policy Development and Research (PD&R) office is conducting a number of evaluated, random-assignment program demonstrations to test new program models, which can be found in PD&R’s biennial report: the Family Self-Sufficiency Demonstration, Pre-Purchase Homeownership Counseling Demonstration, Support and Services at Home (SASH) Demonstration for elderly households, Supportive Services Demonstration for health services in elderly housing, Rent Reform Demonstration, and the Small Area Fair Market Rent Demonstration. The latter demonstrations are in early or middle stages; interim results and long-term follow-up results generally will be reported through HUD USER during the next 2–4 years.
• PD&R also is collaborating with the General Services Administration and U.S. Department of Education to link tenant data with records of students and individuals submitting Free Applications for Federal Student Aid. Through this partnership, three phases of low-cost behaviorally informed experiments are being conducted to increase access of HUD tenants to higher education, through mailed outreach communications (see p.12), mailed plus electronic communications, and HUD-funded education navigators.
• PD&R houses the Office of International and Philanthropic Innovation, and administers five types of Secretary’s Awards to encourage excellence: Public-Philanthropic Partnerships, Opportunity and Empowerment, Healthy Homes, Historic Preservation, and Housing and Community Design. The competitions are judged by juries of professionals, and bring visibility to the nation’s most compelling solutions for addressing housing and community development challenges.
• PD&R sponsors an Innovation in Affordable Housing Competition to engage multidisciplinary teams of graduate students in addressing a specific housing problem developed by an actual public housing agency. The competition increases the nation’s future human capacity to address the affordable housing crisis by exposing future designers, administrators, and policymakers to real-world challenges of a specific legal and community context, with their proposals to be evaluated by an expert jury.
• In FY16, HUD’s National Disaster Resilience Competition provided funding for resilient housing and infrastructure projects to states and communities that suffered major disasters. Collaborative teams were assisted in extensively researching and developing their proposals by nine Resilience Academies developed by the Rockefeller Foundation in partnership with HUD. The in-depth, juried process helped ensure that the $1 billion of resources available for these communities will result in more resilient housing and infrastructure and bridge the gap between social and physical vulnerabilities.
• HUD promotes innovation among staff members through social media sites, and in PD&R, through staff-led Knowledge Collaboratives that focus on selected policy or technical fields to share knowledge and work together on in-house research projects (for examples of in-house work, see footnote 8 in Roadmap Update).

Did the agency use evidence of effectiveness when allocating funds from its 5 largest competitive grant programs in FY17? (Examples: Tiered-evidence frameworks; evidence-based funding set-asides; priority preference points or other preference scoring; Pay for Success provisions)

• In FY17 HUD’s major competitive grant programs are: 1) Continuum of Care Homeless Assistance ($2.0 billion); 2) Disaster Assistance/National Disaster Resilience Competition ($300 million); 3) Choice Neighborhoods Grants program ($125 million); 4) Service Coordinators program ($75 million); 3) Family Self-Sufficiency Program ($75 million); 4) Indian Community Development Block Grant Program ($60 million); and 5) Housing Counseling Assistance ($50 million).
• Decisions regarding the design, funding, and implementation of all HUD competitive grant programs are evidence-based, as specified in funding criteria in HUD’s FY17 Notice of Funding Availability (NOFA). The “Achieving Results and Program Evaluation” factor (see p.13), provides funding priority for applicants that demonstrate effective use of evidence in identifying or selecting the proposed practices, strategies, or programs proposed in the application, and requires all grantees to cooperate in HUD-funded research and evaluation studies (see p. 14). Another factor, “Past Performance,” provides: “In evaluating applications for funding HUD will take into account an applicant’s past performance in managing funds, including, but not limited to….meeting performance targets as established in the grant agreement….” (see p. 13). The “Achieving Results and Program Evaluation” factor and “Past Performance” factor are two of five factors considered that total 100 points, and 2 additional preference points are available for evidence that activities will support other HUD initiatives.
• Competitive grants in the Continuum of Care program account for most HUD grant resources in FY17, and serve homeless populations by providing permanent supportive housing and rapid rehousing services. The FY16 NOFA allocated $1.6 billion using a 200-point scale that provides 27 preference points for development and use of a standard Homeless Management Information System and Point-in-Time counts that support performance measurement (see pp. 37–38), 40 preference points for providing a panel of System Performance Measures for local homeless outcomes (see p.40), and a total of 15 points for each of four target populations for the extent to which grantees achieve reductions in homelessness or make progress toward reducing homelessness as demonstrated for the relevant homeless outcome metrics (see pp. 41–42).
• The National Disaster Resilience Competition, which was HUD’s second-largest competitive grant program in FY16, used evidence about disaster resilience, including benefit/cost analysis, to ensure that disaster funding improves communities’ ability to withstand and recover more quickly from future disasters, hazards, and shocks rather than simply recreating the same vulnerabilities. The tiered funding approach awarded implementation grants after evaluating evidence from the FY15 framing grants.
• HUD partnered with the U.S. Department of Justice in issuing demonstration grants that use Pay for Success financing to reduce homelessness and prisoner recidivism by providing permanent supportive housing through a demonstration using the “housing first” model.

Did the agency use evidence of effectiveness when allocating funds from its 5 largest non-competitive grant programs in FY17? (Examples: Evidence-based funding set-asides; requirements to invest funds in evidence-based activities; Pay for Success provisions)

• HUD’s budget contains 3 large formula grant programs for public housing authorities (PHAs): 1) the Public Housing Operating Fund ($4.4 billion in FY17), 2) the Public Housing Capital Grants ($1.9 billion in FY17), and 3) Housing Choice Voucher (HCV) Administrative Fees ($1.7 billion in FY17). Another 3 formula grant programs serve cities or tribes: 1) Community Development Block Grant Entitlement/Non-Entitlement ($3.0 billion in FY17), 2) HOME Investment Partnerships ($1.0 billion in FY17), and 3) Native American Housing Block Grants ($0.7 billion in FY17).
• Although the funding formulas are prescribed in statute, evaluation-based evidence is central to each program. HUD used evidence from a 2015 Administrative Fee study of the costs that high-performing PHAs incur in administering a HCV program to propose a new FY17 approach for funding Administrative Fees while strengthening PHA incentives to improve HCV outcomes by providing tenant mobility counseling.
• HUD’s funding of public housing is being radically shifted through the evidence-based Rental Assistance Demonstration (RAD), which enables accessing private capital to address the $26 billion backlog of capital needs funding. Based on demonstrated success of RAD, for FY18 HUD proposed removing the cap on the number of public housing developments to be converted to Section 8 contracts. HUD is also conducting a Rent Reform demonstration and a Moving To Work (MTW) demonstration to test efficiencies of changing rent rules.
• HUD conducted an extensive assessment of Native American, Alaska Native, and Native Hawaiian housing needs to strengthen the evidence base for the formula programs.

In FY17, did the agency shift funds away from or within any practice, program, or policy that consistently failed to achieve desired outcomes? (Examples: Requiring low-performing grantees to re-compete for funding; removing ineffective interventions from allowable use of grant funds; proposing the elimination of ineffective programs through annual budget requests)

• HUD’s FY17 budget request included a new formula for funding Housing Choice Voucher Administrative Fees that shifts funding away from inappropriately compensated public housing agencies and increases overall funding according to evidence about actual costs of maintaining a high-performing voucher program. (See here for more info.)
• HUD’s FY17 budget request sought a $11 billion shift (pp.8–9) of resources toward housing vouchers for homeless families based on the rigorous experimental analysis of 4 service options in the Family Options study.
• HUD’s FY18 budget request sought to eliminate funding for Community Development Block Grants. A 2005 PD&R evaluation had shown that targeting of CDBG resources toward communities with greater needs would be greatly enhanced by any of four alternatives to the 1978 statutory formula, but such improvements have not been authorized. An earlier 1995 evaluation found that although CDBG had made a contribution to community development, the neighborhood interventions generally were ad hoc rather than well-coordinated and strategic.