Reflections on DOING timely evaluations. Reflections on the Centre for Evaluation’ symposium on “Timely Evaluation for Programme Improvement”

By: Emilie Koum Besson

LSHTM MSc Public Health, Health Economics, student

On the 23rd of November, the LSHTM Centre for Evaluation held a symposium, which brought together academics and professionals from various disciplines and sectors for a day of discussion on “Timely Evaluation for Programme Improvement”.

The first session of the day focused on why doing timely evaluation would improve programmes. It highlighted two important themes: time and flexibility.

The word ‘timely’ refers to something done or occurring at a favourable or useful time. It is about acting at the right moment. Martin Dale, Deputy Director, Monitoring at Population Services International (PSI), offered an unexpected definition of timely evaluation. He argued that it could be an extension of the day-to-day work of implementers. Aly Visram, from Oxford Policy Management, later commented that implementers are often already aware of some of the issues identified by evaluators that the programme is experiencing. They may even have resolved those issues by the time the evaluation report is completed.

Unfortunately, the issues and programme changes are rarely documented because they are the results of informal discussions. I remember that during my work in Eastern Congo. We had two similar projects in two villages. One was working well while the other one was not picking up. It took a discussion with one of the drivers to understand why. The civil society was not happy with the recruitment of people outside the town, and therefore the village was not supportive of the activities. Imagine looking at the data and trying to understand what happened from one month to another, what delayed the programme in the beginning without knowing that story.

In order to reduce duplication and allow a better understanding of the data, Mr. Dale offered a solution to formalise those informal exchanges. The ‘share interpretation’ feature within DHIS2* is aiming at “bridging the gap between data and actions” by allowing implementers to add a narrative to the data they have collected. Mr. Dale used social media as an analogy to explain it. Platforms like Instagram allow people around the globe to tell a story through one image and a short narrative. Others can complete, challenge or learn from this information in real-time. With DHIS2, at a global level, program managers can explain the reasons behind changes, act upon issues as they emerge, keep a track of the information, and share the acquired knowledge with others. But three questions arise:

  1. How long should we wait until we feel like we have accumulated enough data to conduct the appropriate changes?
  2. How rigorous are the information collected?
  3. How much flexibility do we have to conduct the necessary changes?

There is no consensus on the right answer to those questions. Aly Visram’s experience of changes taught him that small-scale improvement is usually easier to conduct and have a greater impact. On the other hand, large-scale changes tend to require more time, more money and be more challenging. The capacity of an organisation to conduct those changes depends on the level of engagement of the staff, the flexibility of the organisation and the donors.

Do we want the project to be fast and cheap or are we only focusing on the outcomes whatever the cost is?

Project implementation is not a linear field and timely evaluation is necessary to improve its effectiveness and cost-effectiveness. For the field to develop, there is a need for donors and organisations to recognise the value of learning from failure. Even with the best evidence, things happen differently to the way we planned. Timely evaluation is there to ensure that the appropriate changes are conducted when needed. The focus should be on the outcomes and not necessarily on the routes by which we reach them.

Blog reports on the Centre for Evaluation’s Symposium: Timely Evaluation for Programme Improvement, held at the Wellcome Trust on 23rd November 2017.

*DHIS 2 is a Health Information System Program used in 60 countries and 23 organizations across four continents