Putting the Time into Timely Evaluations
By: Sarah Woodhall
LSHTM MSc Public Health student
The theme of the day was timely evaluations. But what does it mean to be timely and what are the considerations for evaluators?
One consideration for evaluators is how rapidly evaluation can be undertaken. Several speakers presented on methods and approaches for making it easier to undertake evaluation either in real-time alongside the implementation of an evaluation, or reducing the time needed to collect data and intelligence for use. For example, Elizabeth Allen (LSHTM) presented on the use of statistical process control. This is a method with its origins in manufacturing and quality control in industrial practices, but provides an intuitive and straightforward way of determining whether observed variation occurs outside of what would be expected. The simplicity of this method makes it one that could be easily employed across a range of scenarios where data are reported on an ongoing basis. Martin Dale presented work by PSI to incorporate interpretations from practitioners in the field into large routine monitoring datasets. This allows external evaluators to get to grips with the data an interpret it more quickly. While this raises some questions about how to deal with potential bias introduced by relying on interpretations from those who actively engage with the reporting system, this approach is making it easier to produce reports more rapidly and alongside ongoing programme rollout.
A second, and perhaps more thorny, consideration is how evaluators can produce evaluations at the right time to inform the policy and programme decisions they seek to influence. In her keynote talk, Annette Boaz (Kingston and St George’s) emphasised the need to engage with stakeholders as early as possible in designing and evaluating an intervention, although much of the discussion focussed on the challenges in achieving this in practice. Several of the breakout groups discussed the idea of working with donors more closely to allow flexibility of programme implementation so that learning from early or real-time evaluations can be incorporated into the intervention during or after scale-up. This would allow evaluations to have impact well beyond the question of ‘does it work’, but perhaps presents a shift in the donor-implementer-evaluator relationship.
And finally, the concept of time was also discussed by Jean Boulton (University of Bath) in her work around complexity. In approaching investigation and evaluation from a perspective that assumes that we exist in a complex system where the success or failure of an intervention will depend on the system around it, evaluators need to investigate both the past and the present and look for signs of what might be to come.
As with evaluation itself, timeliness is clearly a multi-faceted concept. The day brought many of these ideas into sharper focus and gave the audience several practical tools to build on as well as much food for thought to take into their future work.
Blog reports on the Centre for Evaluation’s Symposium: Timely Evaluation for Programme Improvement, held at the Wellcome Trust on 23rd November 2017.Back