Estimating the causal effects of policies on health outcomes using observational data poses many challenges for researchers. One such challenge is choosing an analytic method that captures the estimate of interest under appropriate assumptions. Many methods exist that result in valid policy effect estimates with seemingly subtle differences in interpretations, although the construction of each corresponding estimator can vary greatly across the different methods. Four such methods are difference-in-differences (DID), regression models fit with generalized estimating equations (GEE), augmented synthetic controls, and regression models using propensity scores (PS). The complexity of each method and the differences between them increase with the complexity of the data used. Therefore, it is not unlikely to find contradictory results when attempting to estimate a policy effect using all four methods in the setting of staggered treatment adoption.
We present one such scenario, investigating these four methods and differences in the construction of their estimator through their application assessing the effect of expanded health insurance coverage on HIV outcomes. Medicaid is a publicly-funded insurance program that provides healthcare coverage to eligible low-income adults, children, pregnant women, elderly adults, and people with disabilities in the US. A 2012 Supreme Court decision made expansion of Medicaid eligibility optional for states starting on January 1st, 2014. Between 2014 and 2017, 31 states decided to expand Medicaid, providing a quasi-experimental setting that could be utilized to better understand the impact of Medicaid expansion on HIV care continuum outcomes such as retention in care. People with HIV contributing data in the North American AIDS Cohort Collaboration on Research and Design (NA-ACCORD) between 2012 and 2017 were included. Using only the immediate period before and following Medicaid expansion, without covariate adjustments, most methods provide identical effect estimates. However, we obtain contradictory results after including more complex data and covariates, though we do not believe this indicates any unreliability in estimates. Rather, these differences emerge from different target estimands and the construction of each estimator.