By Holly McKee, published on 18th October 2016
An interview with Mark Myatt and Grant Aaron
There is a lot of buzz among food fortification stakeholders about the importance of assessing program coverage. What is coverage? And why is coverage important to achieving program impact? We spoke to Mark Myatt and Grant Aaron, who over the last 4 years have developed tools to assist fortification stakeholders with program coverage assessments:
Read on to learn more about how they met, and some of the work they have done together that is helping to paint a clearer picture of true impact of nutrition programs.
The beginning: How did you two meet?
Mark Myatt: I had been working for several years with VALID International and a number of NGOs, UNOs, university departments, and Ministries of Health on developing coverage assessment methods for programs treating severe acute malnutrition (SAM). We had, by about 2010, developed a set of proven methods and felt that they could be used for assessing the coverage of a wider range of programs.
In 2013 VALID International ran two technical workshops – one for governments and donor agencies and one for implementing agencies. Grant Aaron came to one of these workshops.
Grant Aaron: I left the workshop absolutely convinced that the topics discussed could be applied to my work at the time at GAIN: large-scale staple food fortification and home fortification (e.g. micronutrient powders) interventions. In this domain of work, efficacy evaluations received far more attention than coverage assessments. In fact, coverage did not get much attention at all
Grant Aaron: Several things. One, I liked how the methods discussed at the workshop were packaged as a complete toolkit. They were conceptually easy to understand and, most importantly, usable by programs and not just academic exercises. Two, the methods incorporated a spatial dynamic to assessing program coverage. It is intuitive that availability, access, and utilization vary across a population geography. Coverage methods should be able to account for this.
What is coverage? What does it even mean?
Mark Myatt: Coverage is the proportion of people who need a service that get that service. With SAM it is the proportion of severely malnourished children in the population that are being treated for SAM using a proven effective treatment protocol. Impact is coverage multiplied by treatment efficacy. With 5% coverage and 100% treatment efficacy impact is only 5%. This is why coverage has been referred to as the “royal road to impact”.
Coverage only delivers impact if you reach those who need the intervention. With fortification programs we can identify households that are likely to be in need using indicators such as low socio-economic status, low dietary diversity, or poor IYCN. We can then look at coverage in households identified as being in need. We call this type of coverage “met need”. It is possible for a program to have quite high levels of raw coverage but to have low levels of met need if the product tends not to be used by (e.g.) poorer households. We also divide coverage up using a “bottlenecks” model. We can look at (e.g.) “message coverage” which for market-based IYCN program would include awareness of the need for good IYCN practices, knowledge of good IYCN practices, and knowledge of the product. These are all coverages in their own right. Without message coverage we cannot expect many carers to purchase and use a product. We have a ladder of coverages. Climbing from message coverage though “contact coverage” (some purchase and use) to “effective coverage” (sustained and frequent purchase and use). It is surprising how many program fail at the most basic level.
What is the spatial dynamic and how is it important?
Mark Myatt: A basic model for epidemiological description is time, place, and person (TPP). Most surveys in the nutrition sector tend to ignore the spatial dimension which is “place” in the TPP model. In terms of coverage the “spatial dynamic” means it is usually very patchy. Distance is a big issue. People who live a long way from services do not tend to access services. With fortification programs we might see coverage depending upon something like dietary patterns that vary over the program area.
Where did you start with assessing coverage of fortification programs?
Grant Aaron: Our first project together was in Ghana to assess coverage of a sales based approach to delivering micronutrient powders (MNPs) to infants and children. We piggy-backed a staple food coverage assessment on as a methods development pilot. We spent July 2013 in Ghana, along with an excellent team from Valid International and the University of Ghana, assessing the coverage of the MNP program and working on methods for the staple food fortification coverage component.
Around the same time, the donor supporting most of the staple-food fortification programming at GAIN, made a request to undertake case studies focusing on assessing program coverage.
This was a big break for methods development as it put coverage front and center for organizational priorities. Out of this work we formalized what we went on to call the Fortification Assessment Coverage Tool (FACT). This was intended to be a standardized tool to assist GAIN, and more broadly fortification stakeholders, with fortification coverage assessments. To date GAIN and its partners have completed close to 20 surveys that have applied these methods or some variant thereof.
What were some of the biggest challenges and/or learnings?
Grant Aaron: From an operational perspective, I would say my biggest challenge was having to move from pilot phase to scale-up rather hastily. It’s been difficult to balance organizational demands for running surveys vs. taking the necessary time to finalize a tool. In terms of broad learning from the body of work, two things stand out: one there is no substitute for adequate due diligence and program design prior to program roll-out; and two, we have to be focused on ensuring that programs achieve high coverage in the at-risk population groups.
Mark Myatt: The main challenge during development was the push to move very quickly from small-scale pilots to large-scale surveys. Now the challenge is to get survey partners to use spatial sampling. Most partners like to stick with what they know and what they know is simple cluster surveys.
Learning for methodological development is to insist on maintaining a development plan. The use of third party peer review was very useful. Learning from the surveys (the point of the toolkit) is that proper program design, ongoing monitoring and evaluation, and a focus on achieving coverage and meeting needs are key to impact. I think donors and implementing partners just assumed that their programs were good.
You contacted BioAnalyt to assist with specimen analyses for many of these surveys. Tell us a bit about this component of the work.
Grant Aaron: Since we were dealing with such a large number of food specimens, thousands of samples across a number of geographies, we wanted to ensure consistency, high quality, and rapid turnaround, while maintaining low-cost. BioAnalyt has assisted in all these regards.
Mark Myatt: Large numbers of specimens of flour, oil, salt, and so-on. We wanted these to be analyzed properly (i.e. with good accuracy and precision) and quickly. BioAnalyt helped us with this. I think this has proven to be a good choice of collaborator.
Grant Aaron: We’ve been working on quite a few publications from this collective body of work. The goal is to get everything in public domain!
Mark Myatt: I think in addition to scientific publications, we have to produce a new handbook for the methods. I’d like to see an organization like the Coverage Monitoring Network established to do or support quality FACT surveys for all agencies with Infant & Young Child Nutrition (IYCN) and fortification programs.