Understanding Context: Reflections from the Centre for Evaluation Retreat
By: Jamie Lundine
LSHTM MSc Public Health Student
The LSHTM Centre for Evaluation retreat brought together approximately 95 students, faculty members, staff and researchers at the School for a day of discussion on current research and learning in evaluation for public health. The day highlighted two major themes: context and opportunistic evaluations.
The morning’s theme was “assessing context and its influence on outcomes.” We heard from speakers about the importance of understanding context when conducting evaluations. Chris Bonnell reminded us of the importance of understanding not only the context of the programmes we are evaluating, but also the political and organizational context in which our evaluations take place.
He argued that one of the most important question when embarking on an evaluation is to ask our clients or partner organizations “Why do you want to do this review/evaluation?” Understanding the answer to this question reveals insights about the stakeholders, their motivations and the challenges and opportunities that might arise during the evaluation cycle.
Depending on the outcome of your stakeholder analysis, Chris suggested a series of helpful steps that can be taken in order to strengthen the position of evaluators; these include:
- Involve both researchers and policy makers from the donor or client organization that you are working with. Research staff members can act as allies in your evaluation work, supporting your design, data collection and analysis.
- Develop a research protocol and publish or register it. This is arguably easier to implement if the evaluation design involves a Randomized Control Trial. With other evaluation designs, there may be no research body with which to register a protocol, however evaluators can push to publish the research design online or in a journal instead. This will hold all stakeholders accountable to commitments made early on the process. This is particularly important if research findings begin to demonstrate that a policy or programme is not working (or in some cases, is causing harm rather than the desired outcome).
- Establish an independent steering committee, with relevant expertise (e.g. epidemiologists, demographers, economists), as another mechanism for accountability.
- Be kind, but assertive.
- Involve senior colleagues to back-up your position on challenging issues. As a junior researcher, having senior colleagues on the team in advisory roles can position you to have more influence throughout the process.
- Accept that evaluation work can be political and/or have political consequences for you and your career. An evaluation is not neutral and understanding your values and role in the system is important as the process unfolds.
Given that no evaluation is devoid of context, we must seek to explore and understand the context as a crucial step to conducting our research. The afternoon session we were asked to consider that “in evaluation, we arguably never have full control and must compromise. What are we prepared to compromise on, whilst trying to remain “rigorous”?”
In groups, we broke out and considered this question. We discussed the tension between quantitative and qualitative methods and the tendency of both researchers and donors to lean towards quantifying results. As evaluation professionals, we often focus on sample size to power our studies and evaluate impact. Often however, we are often faced with a reality of competing research interests, limited budgets, shifting priorities and scope creep; in this reality, we should not compromise on process evaluation. Understanding the “why” behind what works and what doesn’t in a public health intervention is crucial. If you have a large sample size, but do not understand the underlying process behind why an intervention succeeded or failed, then it will be difficult to learn what went well, what didn’t go well and how you can adapt the intervention and deploy it in a different context.
This is an important conversation, which hopefully continues beyond the end of the retreat.