Measuring Health Care Use Outcomes from a Community Development Initiative

Community development has the potential to mitigate neighborhood conditions that undermine residents’ health and create the environment needed for children to lead healthy lives. One community development initiative, Healthy Neighborhoods Healthy Families (HNHF), a multiagency collaboration between Nationwide Children’s Hospital and local, community-based organizations, focuses on enhancing the health and well-being of families in south-side neighborhoods in Columbus, Ohio. Researchers from multiple disciplines collaborated to investigate whether the Healthy Homes program of HNHF, which is aimed at neighborhood revitalization in an area with high vacancy rates, may have also contributed to the health of children residing in the community. They conducted a quasi-experimental study to evaluate differences in health care use by children in the HNHF program neighborhood relative to local comparison neighborhoods.

The researchers used propensity score modeling to identify two comparison neighborhoods with similar baseline characteristics as the target neighborhood so they could measure the program’s impact on people living in the HNHF neighborhood. The measures used to select the comparison neighborhoods included neighborhood-level data on demographics, crime, housing instability, neighborhood housing variables, racial segregation, and pollution.

The researchers then used administrative data for 61,727 Medicaid-eligible children who lived in the three neighborhoods during the preintervention study period (August 2008 to July 2010) or postintervention study period (August 2015 to July 2017). The researchers sought to assess the HNHF program’s impact on health care use by measuring the number of emergency department visits, probability of inpatient hospital admission, and length of stay.

Finally, the researchers applied a difference-in-differences statistical methodology to determine whether changes in children’s health care use in the HNHF neighborhood differed from changes in the comparison neighborhoods. They compared pre- and postintervention trends in the HNHF neighborhood to trends in the two comparison neighborhoods both pooled and separately. Although the analysis revealed some decreases in health care use for children in the HNHF neighborhoods compared with the comparison neighborhoods, most of these results were not statistically significant.

Key findings

• There were no statistically significant differences between the intervention and pooled comparison neighborhoods over time for any of the outcomes.
• Ad hoc analyses found small significant differences in emergency department visits and admission in one of the two comparison neighborhoods.

Although the key findings from this study were inconclusive, the researchers noted several important lessons that confirm existing knowledge about the challenges of evaluating community development initiatives:

• It is feasible to conduct midimplementation evaluative research based on observational data.
• The researchers found it was unrealistic to expect changes in health care use when reducing health care use was not a primary goal of the program. Researches also noted that 10 years may not be long enough to see those effects, especially as many effects of adverse childhood experiences don’t become evident until adulthood.
• Because neighborhoods have unique features, it is challenging to compare one neighborhood to another. More examples and best practices are needed to conduct this type of place-based comparison study.
• Unlike randomized controlled trials, real-world studies on multifaceted interventions cannot control for every factor, as there may be multiple driving forces. However, the researchers point out that “randomized trials do not reflect the real world [and so] there is a place for both.”
• Neighborhood-level outcome studies require multidisciplinary teams, scientific rigor, and patience.

Photo by f11photo/Shutterstock