Advancing Ex-Post Impact Assessment of Environmental and Social Impacts of CGIAR Research (scoping study)
Donors want to move beyond directly measurable economic impacts to account for environmental and social indicator (ESI) impacts—negative and positive. Joint SPIA/IAFP work has evolved to consider NRM research and poverty impacts but is still heavily focused on economic impacts expressed through markets, with a few important exceptions—pesticides and health, carbon emissions. This activity emerged from the IAFP meeting in Nairobi which confirmed the need for exploring and applying ESI using new approaches. The initial phase of this study employed two broad approaches:
A. Extended Cost-Benefit Analysis (ECBA) expanded to include non-market ESI where all benefits/costs are converted into monetary terms, using revealed preferences (hedonic pricing) and stated preference (contingent valuation, choice modelling). Jeff Bennett at ANU is the consultant.
B. Indicators and other measures of ESI that are considered separately with market costs and benefits using weighted indices, cost effectiveness, etc. (Goran Djurfeldt, Torbjörn Fagerström and Axel Fredholm are the consultants).
There are three stages to the study: (i) Desk review of methods and identify relevant case studies; (ii) Case studies led by centers; (iii) Synthesis of lessons and development of guidelines. Phase one has just recently been completed with the finalization of the two consultancy reports after peer review. SPIA has now developed and circulated for comments (to IAFPs, consultants and SC members) a proposal for moving forward with Phase two but focusing exclusively on environmental impacts assessment.
The workplan, sequence/timeframe, and budget for this study were discussed at SPIA 35. A Planning Workshop is being held at FAO in Rome on 22-23 June to discuss the methodologies for advancing five Centre case studies.
Crop germplasm improvement: updating the impact initiative
Despite the considerable amount of impact assessment of crop improvement research activity over the years that proved essential in documenting widespread impacts of the CGIAR’s investments, there is today relatively little effort being made to sustain and update that work. The Evenson and Gollen study (2003), the most comprehensive assessment to-date, used data from the Centers up to 1997, and some even earlier. At the recent IAFP meeting, a representative of the Bill and Melinda Gates Foundation challenged the group by asking what is comprehensively known about the diffusion of core technologies from major CGIAR programs. If crop improvement research is considered the major success story, even today, it is essential to update the Evenson and Gollin study. (Note: there are important exceptions with at least three centers having updated varietal adoption data in 2007 and 2008). Basic data on adoption of improved varieties must be kept current: it should be collected on a regular and systematic basis. There was general consensus amongst SPIA and the IAFPs that this deserved priority attention. At a follow-up meeting with many of the IAFPs attending, participants agreed that investing in surveys of varietal diffusion in several commodities, countries, and regions was necessary to supplement and validate national and subnational expert opinion that was the basis for the earlier work. A strong pitch was made for a new initiative in this respect under SPIA’s leadership—similar to last time—although SPIA had emphasized that the motivation and leadership for this update initially should come from the IAFPs and Gates. In subsequent discussions with the Gates Foundation, SPIA agreed to consider coordinating this activity.
Recently, the Gates Foundation organized a two-day workshop of CGIAR IAFPs in Seattle to initiate this study. All centers involved in crop genetic improvement were invited and sent representatives—Africa Rice, IITA, ICRISAT, CIMMYT, IRRI, ICARDA, CIP, CIAT, and IFPRI, in addition to the SPIA chair and IA experts. The objective of the workshop was to design and prepare a proposal to the Bill and Melinda Gates Foundation (BMGF) to support work on the impacts of CGIAR crop genetic improvement (CGI) in Sub-Saharan Africa and Asia. Thus, the objective of this proposal is to update and institutionalize impact assessment on CGI, standardize procedures and data bases, and to develop and test methods for improving impact assessment. Key results from the workshop include:
Main results
1. A concept note circulated prior to the meeting, was developed by Tom Walker (consultant) in close consultation and inputs from BMGF (Greg Traxler) and SPIA.
2. If approved BMGF will provide about $2.8 m through the Science Council to support the work in sub-Saharan Africa over a period of 2.5 years. A further $ 0.5 million is likely to be provided for similar work in rainfed Asia, either through the SC or directly through IRRI or ICRISAT or even ICAR. In addition, ACIAR will carry out an impact assessment of CGI for rice, with a likely focus on Southeast Asia.
3. The proposal will consist of three components.
a. Collection of base data on varietal releases, pedigrees, resources to CGI in NARS, and expert opinions on varietal adoption for all major country-commodity combinations (greater than 25,000 ha of a commodity in a specific country).
b. Surveys of varietal diffusion in selected countries. These surveys are primarily to verify estimates of expert opinion, as well as provide some additional information on varietal diffusion, access to seed, and traits of interest to farmers.
c. Studies in selected countries to develop and test innovative approaches to measuring adoption and impacts of CGI on poverty, gender, food security and the environment.
4. The funds for the first component will be pre-allocated in the proposal to each center according to the number of country-commodity combinations (centers with bigger areas and more countries get more funds, although overall funding for this component is modest). Funds for the second component will be pre-allocated equitably to each center ($100-$150K) to conduct a country diffusion study. However, centers leading the diffusion surveys for a country will be expected to collaborate with other centers to collect information on all crops of interest to the system.
5. Funds for the third component would be allocated on a competitive basis following the established procedures developed by SPIA. IFPRI’s Harvest Choice in collaboration with the centers is expected to be a strong contender for these funds.
6. Tom Walker, who has worked with SPIA on the PORIA impact studies has agreed to coordinate the study, with about one third to one half of his time allocated to it over the next 2.5 years. He would be contracted by SPIA. However, SPIA would not be involved in day to day management so as to preserve overall independence and oversee the peer review to ensure quality of the final product.
7. Full details of the separate grant for South Asia and the ACIAR evaluation of rice CGI will be developed in the coming weeks, and likely finalized when all parties are in Beijing in August.
Next Steps
8. Tom Walker will work with Greg Traxler and myself (and Mywish) over the next week to develop a full draft of the proposal. This will be circulated within BMGF for comment and external review, and will be sent to the centers and the SC for final comments.
9. The draft will then be finalized in early July and submitted with a cover letter from the SC.
10. On this schedule, funds would be available by Oct 15th, 2009 to initiate work. The SC Secretariat needs to ensure that we have the financial instruments in place to accept the funds and quickly send on the pre-allocated funds to each center.
11. A detail planning and methods workshop will be held in Africa before the end of 2009.
Randomized control trials (RCTs)
RCTs use experimental methods to eliminate ‘selection bias’ between treatment and control group through random assignment. They are being increasingly used for their strong counterfactual and high internal validity, i.e., they depend less on assumptions regarding steps along the impact pathway. The method appeals to donors and many academics in development economics, but RCTs are controversial and criticized by many in the evaluation field. Whether they are useful for documenting impact (as opposed to simply establishing efficacy) may depend on validity of assumptions about or ease of scaling up, type of intervention considered -- simple vs. complex. Other concerns relate to cost – RCTs are expensive to implement (though the cost of a wrong answer by other methods is very high). Arguments in favouring of exploring or expanding use of RCTs in the CGIAR include: i) Centers are already doing them; ii) scope for designing new experiments to address economic, social and environmental impacts together exists (though expensive); and iii) there are new sources of funding for this approach and new venues for reporting “intelligent failures” to overcome publication bias. A major comment during a plenary discussion of the IAFP meeting focused on applicability – the approach seems more relevant for establishing efficacy in a limited environment than one for documenting impact ex-post. SPIA member Mywish Maredia is developing a scoping paper to explore this issue further.
Enhancing the Value and Use of Ex post Evaluation of Outcomes and Impacts in the CGIAR
Study Terms of Reference
Summary of Key Issues
Summary of Responses to Questions in TORs
Advancing Ex-Post Impact Assessment of Environmental and Social Impacts of CGIAR Research
<//span>
| 
