Category Archives: Impact Evalution

The state of art in M&E and IE practices — reflections on the UK Evaluation Society Annual Conference (2013)


The UK Evaluation Society (UKES) Annual Conference is a unique opportunity to share knowledge and experience with other Monitoring and Evaluation (M&E) and Impact Evaluation (IE) professionals. Three main points for reflection inspired by the two-days of presentations and debates of this year’s edition that I would like to highlight here:

  1. The definition of Value For Money (VFM), that during a session facilitated by Save The Children was broken down in economic vs. operational VFM — the former as economic benefit coming from the implementation of the programme, the latter as the intrinsic value of the service (often having a public nature) provided by the activity (having better quality education is good per se, not dependent on the cost of provision);
  2. The IE community now is ready to leave behind the (sterile?) debate qualitative vs. quantitative methodologies for IE – a theory based mixed approach is considered to tacke more efficiently with the complexity of society; also, different methodologies serve different proposes — it is up to the evaluator to select the ‘right one’ each time;  
  3. The challenge that M&E and IE specialists encounter not to be considered as a threat but a resource by the organisations they work in; this is a reciprocal learning process, in which the M&E specialist deeply understands rationale, complexities, assumptions and risks of the activities and the programme managers use rigorously tested practices to improve delivery and operational and economic VFM. 

The Conference was also an opportunity to present INASP‘s approach to ensure both that IEs are carried out in a rigorous way and that the learning is shared with and informs the activity of relevant stakeholders.

As INASP’s M&E Officer, my poster Promoting Change in Heterogeneous and Dynamic Environments: A Case-Study From Uganda (awarded as best Conference Poster), presented INASP’s approach in carrying out an ex-post small N IE of effectiveness of training delivered by INASP and the Consortium of Uganda University Libraries (CUUL) in the period 2010-2012.

2013 UKES award poster

First the evaluation questions were formulated and methodology was defined  — a triangulation-approach to tackle with issues of small population size (small N) and self-selection in the treatment group; second, data collection and analysis was carried out — by using ‘cross-fire’ online questionnaires administered to four different target groups and identifying both common trends and contradictions in responses ; third, the findings were discussed and a set of actions (policy recommendations) was planned during a validation workshop facilitated by INASP with the head of the libraries of member institutions of CUUL.

If you want to know more about the discussed issues, you may find useful the following resources:

On VFM — Emmi et Al. 2011, ITAD 2010; On Theory-Based approaches — 3ie 2009 and LSE 2012; on methodologies — DFID 2013DFID 2012, Better Evaluation and Newman 2012.

I will appreciate any other perspective.

To Practice 7 – I’m not the World Bank


Before starting my Nicaraguan experience, dreams of glory flooded my ego. Reviewing all my studies on data collection, I knew it would have been a priority to collect data from households survey as fast as possible. Because interviewing the first household of the community today and the last one in 6-months time means introducing a significant source of bias in the research, especially in rural economies subject to seasonality. However, I’m realising that I was naive. Because we’re just two interviewers, because I didn’t consider my stomach could have left me in my room for one entire week, because I go to and from the community using my stick, not my jeep. In other words, because I’m not the World Bank, nor Indiana Jones. To sum up, my goal is to have enough large control group to match with the treatment of the households who participate to a micro credit scheme – the ‘Small Business Program. I have two possibilities: surrendering to the extended collection time bias or opting for a second best solution. I’m trying to move toward the second option. In parallel with the community survey, I’m working with students of a technical high school to assess the program Education That Pays For Itself, run by the English NGO, Teach A Man To Fish. Among the NGO’s outcomes of interests, there is the socioeconomic condition of the students’ families. Hence, I’ training the students to interview their families (belonging to other communities though), in order to increase quickly the treatment group. Therefore, I can focus my interviews in the community on treatment group mainly, shortening the data collection timing, and getting information on students’ families. The assumption is that from the students’ families, coming from rural background in Nicaragua, a PSM approach will allow me to select part of my control group. I fancy to better solutions…

P.S. Edgar, I know you’re thinking about the horse!

To practice 6 – Ex Post Considerations on Vulnerability Measurement


After carrying out and attending several interviews, questions are arising in me about the vulnerability section. Vulnerability – considered in my study “the magnitude of the threat of future poverty” (Calvo and Dercon 2005, p.5) – is a complex concept, indeed enriching a poverty profile.

BUT:

A man who was shot in an armed robbery 3 months ago, told me his level of worry for being assaulted in the future is very low, a woman who hasn’t been assaulted in the last 5 years, stated is extremely worried of being assaulted in the future.

Now, let’s put apart probabilistic calculations and think in a mere qualitative way, how do we compare their different levels of perceived vulnerability to assaults? Measuring the poverty ex-ante, a real challenge. Literature on this issue is warmly welcomed, I need a reliable vulnerability index…

To Practice (5) – Health Conditions and Opportunity Cost of Time


IS MORTALITY IN DEVELOPING COUNTRIES PROCYCLICAL? HEALTH PRODUCTION AND THE VALUE OF TIME IN COLOMBIA’S COFFEE-GROWING REGIONS (Miller and Urdinola, 2007) is a paper testing the hypothesis of countercyclical infant mortality in poor coffee crops area in rural Colombia, i.e. the mortality rate being decreasing during economic growth. That is to say, during economic upturns (high coffee price), coffee growers would have a higher propensity to focus on their children’s health conditions due to higher disposable income. Meanwhile, during economic downturns (low coffee price), poor coffee growers would give priority to other expenditures than the ones put towards their children’s health status – due to consumption smoothing mechanisms. However, the authors find evidence of procyclical infant mortality rate. Namely, the mortality rate increases during positive economic shocks (high coffee price), when the coffee growers are supposed to have more resources they can assign to health care. In developed countries, too, evidences suggest that mortality rate is procyclical. This could be explained by three main factors, holding during economic upturns:

  1. Increasing consumption of harmful normal goods like tobacco and  alcohol (Miller and Urdinola, 2007)
  2. Increasing pollutant emissions and higher traffic fatality rates (ibid.)
  3. Increasing opportunity cost of time (ibid.)

The Colombian context – where the rates of tobacco and alcohol addition are supposed to be close to zero among infants and the geographic and environmental features make pollution and traffic an minor problem –   suggests reasons 1. and 2. cannot hold. On the other hand, factor 3., the increased opportunity cost of time, looks plausible.

Jinotega department in Nicaragua is a coffee-growing region. Yesterday I was interviewing a household’s head and, at the question whether he was worried by bad health conditions of him or a family member, he complined about the waste of time for going to the health center. Did he read Miller and Urdinola, too?

Don’t hesitate to tell me I make things too simple!

To Practice (4) – God’s Willingness and the Ethic of Risk


Interview 2, 3 & 4 completed. I shortened the time, from 3 to 2 hours to complete the questionnaire! Like the first time, I brought back home both material and spiritual enrichment.  Again, the vulnerability section is giving unexpected reactions. At the question whether the hh’s head was afraid of different shocks affecting familiar economic activity, the perceived fear decreased in the time-horizon, counter-intuitively. ‘What God keeps today, he will give it back tomorrow’ is the respondent’s explanation.  In some ways, this attitude could bring individuals to invest in riskier activities, faithful in divine justice at sometime; on the other hand, individuals could have an incentive to inactivity, being everything decided – reminiscences of Bachelor’s courses on Webber and The Protestant Ethic of Capitalism come to me.‎ I’ll seek to let the data tell me this story.

In the picture below, Questionnaire  with ocote (my second still life!), a pine resin used to light up the house – not reached by electric light.

To practice (2) – Like a fly


The household questionnaire is complete, composed by 10 sections:

  1. Households’ composition;
  2. Migration
  3. Physical living standards
  4. Social capital
  5. Health conditions
  6. Economic activity
  7. Consumption and savings
  8. Agriculture and farming
  9. Small Business Program
  10. Vulnerability

A questionnaire should take into account  the context and the research questions.  Also, I tried to plan analytically the interview process and organising quick, meticulous and efficient data collection and input. Finally, a responsible data collection and input allows dealing with ethical issues of privacy, veracity and objectivity and the one-shot nature of the data collection (there is no possibility, in the short run, to come back to the same family to ask questions again).

Now it’s time to take a deep breath, knock the door and entering, like a fly, in the community households’ life. Like a fly, insignificant, hoping they will not see me as a disturbing element, at least not too much.

Impressions of April


One of my most common activities here, during never-ending hours without light and internet connection is reflecting, sometimes trivially, other  (supposedly) deeply. I think about distance, from the people I love, from my routines, from what I consider my  home. In bad moments I would like to  teleport to the other side of the ocean, hug my beloved, and then come back here, charged for a new day. However, I am gradually auto-convincing, maybe moved by a primitive surviving instinct, I like to taste the sweet and sour feeling of waiting and live entirely my new life. In good moments, I believe that teleportation would make the journey an empty experience indeed. ‘Radio Cure’ by Wilco is better than any other word.