Tag: evaluation

This study sought to evaluate the impacts of World Agroforestry’s (ICRAF) Agroforestry Food Security Programme, Phase II (AFSPII) in Malawi (2012-2016). Given the absence of a well-planned impact evaluation strategy, a quasi-experimental design grounded in AFSPII’s theory of change was designed to take advantage of this programme’s geographically staggered roll-out. This was complemented by semi-structured interviews with 40 programme participants and discussions with district-level implementing partners to corroborate the results. The rigour of this evaluation strategy was undermined by the fact that many in the planned comparison group—participants residing in purposively matched villages engaged in AFSPII during its third year—had actually commenced their participation in the first year. Since many participants interviewed in Years 1 and 3 actually resided in the same villages, comparing the two was problematic because: (i) they likely differed in both observable and unobservable ways relevant to the study’s outcomes of interest; and (ii) the Year 3 participants from these villages may have been influenced by their interactions with Year 1 participants. This issue was exacerbated by the fact that only 53% of the targeted sample size was attained and baseline data could only be reconstructed for a few key variables (e.g. household assets). While several econometric models were used to control for observable differences between the two groups, these limitations are explicitly recognized. Nevertheless, several relevant lessons were generated, particularly from the study’s examination of the uptake of promoted agroforestry practices among AFSPII participants, as well as the contextual factors influencing this. Only a small proportion of these participants were able to realize the targets set for the four promoted agroforestry practices. For example, for the programme’s core intervention—fertilizer tree promotion—only 14% of the Year 1 participants incorporated either a moderate or significant amount of fertilizer tree biomass into their fields prior to maize planting in both Years 2 and 3. It is therefore unsurprising that no significant average differences for the various outcome and impact measures were found in the comparison of Year 1 (n=402) and Year 3 (n=229) participants, except for a modest decrease in time spent collecting firewood. The study concludes with three recommendations: (1) Build in systematic provisions for participatory monitoring, follow-up support, and adaptive management when scaling complex agronomic and natural resource management interventions. (2) Expand the research cycle to encompass the scaling effort, particularly when there is uncertainty about technology and practice uptake or performance, including how they interact with the socio-economic and biophysical characteristics of “real world” settings. (3) Ensure that an appropriate impact evaluation strategy is pursued from the outset, viewing it as an integral part of the larger research effort.