Skip to content

Identifying a “Similar” Comparison Group When Randomization Is Not an Option

by on October 25, 2013

Question posted by a member of MEET, a community of practice dedicated to MEASURE Evaluation trainees:

Am working in Namibia and am about to start on an prospective evaluation of HIV prevention programmes and would like to know the best practices when it comes to the selection of a control group. We have two options, both with advantages and disadvantages.

  1. The first option is to select a group from  neighbouring communities.  Unfortunately the country shows great variation and there are concerns about the comparability of these communities.
  2. The second option is to select a comparison group from the same communities. The challenge here is there is a  chance of underestimating the impact of the program due to spillover.

Anyone with comments on how we can best approach this? Any experiences or documentation are most welcome.

Answer from Sian Curtis, MEASURE Evaluation’s Senior Evaluation Advisor:

The dilemma of identifying a “similar” comparison group when randomization is not an option (which I assume it is not from your question) is a common one. I don’t think there is a single right answer as a lot will depend on the specific context of your program and the risk of spillover effects vs the risk of unobserved differences between communities affecting your results.

If the interventions are very individualized, risk of spillover effects will likely be lower than if the interventions are more community-based or likely to spread through social networks. In this case you mention media-based communication and IPC so that suggests that spillover could be a significant issue depending on the reach of the media communication and the social networks being used for IPC. If you go with a comparison group from another community, your fundamental assumption is that the experience of that community represents what would have happened in your intervention population if it had not experienced the intervention. There are observed differences you can potentially control for but the problem is always the unobserved and unobservable factors that could differ between communities and affect the results.

There is no perfect solution to this problem. The program you describe includes a lot of selection criteria – under-served communities in peri-urban and urban areas; younger patients; male; recently started on ARV; ARV adherence problems; discordant relationship. Your comparison area should as similar as possible to the intervention communities, at least in ways that might affect the results, and you will need to recruit comparison participants in the same way using the same criteria as you are using to select intervention participants. Using some kind of matching might help to get as similar a group as possible. For example, match your comparison sites on some observed criteria (using simple or propensity score matching) and then recruit matched controls for the intervention participants as you recruit them. Matching only controls for observable characteristics of course.

If you have the data available, you can also look at past trends in key outcomes in your intervention areas and potential comparison areas and select comparison areas that have similar past trends to your intervention areas to support your assumption that future trends would have been similar in the absence of the intervention. Usually, such data are not available though so that option is often not possible. Another consideration is to collect data on exposure to the types of interventions being implemented (media, IPC etc.) so you can check at the analysis stage how much self-reported exposure you actually have in each group to see whether your intervention group did actually participate in the interventions and whether your comparison group was exposed to any similar interventions.

From → Evaluation, HIV/AIDS

Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: