The state of art in M&E and IE practices — reflections on the UK Evaluation Society Annual Conference (2013)

The UK Evaluation Society (UKES) Annual Conference is a unique opportunity to share knowledge and experience with other Monitoring and Evaluation (M&E) and Impact Evaluation (IE) professionals. Three main points for reflection inspired by the two-days of presentations and debates of this year’s edition that I would like to highlight here:

1. The definition of Value For Money (VFM), that during a session facilitated by Save The Children was broken down in economic vs. operational VFM — the former as economic benefit coming from the implementation of the programme, the latter as the intrinsic value of the service (often having a public nature) provided by the activity (having better quality education is good per se, not dependent on the cost of provision);
2. The IE community now is ready to leave behind the (sterile?) debate qualitative vs. quantitative methodologies for IE – a theory based mixed approach is considered to tacke more efficiently with the complexity of society; also, different methodologies serve different proposes — it is up to the evaluator to select the ‘right one’ each time;  
3. The challenge that M&E and IE specialists encounter not to be considered as a threat but a resource by the organisations they work in; this is a reciprocal learning process, in which the M&E specialist deeply understands rationale, complexities, assumptions and risks of the activities and the programme managers use rigorously tested practices to improve delivery and operational and economic VFM. 

The Conference was also an opportunity to present INASP‘s approach to ensure both that IEs are carried out in a rigorous way and that the learning is shared with and informs the activity of relevant stakeholders.

As INASP’s M&E Officer, my poster Promoting Change in Heterogeneous and Dynamic Environments: A Case-Study From Uganda (awarded as best Conference Poster), presented INASP’s approach in carrying out an ex-post small N IE of effectiveness of training delivered by INASP and the Consortium of Uganda University Libraries (CUUL) in the period 2010-2012.

First the evaluation questions were formulated and methodology was defined  — a triangulation-approach to tackle with issues of small population size (small N) and self-selection in the treatment group; second, data collection and analysis was carried out — by using ‘cross-fire’ online questionnaires administered to four different target groups and identifying both common trends and contradictions in responses ; third, the findings were discussed and a set of actions (policy recommendations) was planned during a validation workshop facilitated by INASP with the head of the libraries of member institutions of CUUL.

If you want to know more about the discussed issues, you may find useful the following resources:

On VFM — Emmi et Al. 2011, ITAD 2010; On Theory-Based approaches — 3ie 2009 and LSE 2012; on methodologies — DFID 2013, DFID 2012, Better Evaluation and Newman 2012.

I will appreciate any other perspective.

On Norway

Can we find a reason for the Norwegian tragedy? Northern European societies are considered among the most tolerant worldwide. Newton’s third law of dynamics states that for each action corresponds an equal and opposite reaction. Can we transfer this principle to sociology? The “Christian” terrorist who killed more than 90 people is a product of the tolerant Norwegian society indeed. Is his action the price that Norway paid to be so open and multicultural? I’m not a sociologist and I know that it is easy and dangerous to simplify in social sciences. However, what I believe is that Norway (and all Scandinavian countries) should prevent in the future these events without giving into the blackmail of the violence. That is, without renouncing to principles of tolerance, freedom and open multiculturalism which, maybe, contributed to determine the equal and opposite reaction on Utoeya island and in Oslo’s center.

I apologise if I offended anyone with my trivial analysis – all my solidarity goes to Norwegian people.

Viva l’Italia! (2)

The figure above is taken from the website Lavoce.info, a community of economists discussing especially on Italy and EU. On the X axis is per capita GDP, on the Y axis average MPs’ annual salaries in the EU (the currency is Euros).

The “I” outlier stands for Italy. Do you need further comments?

Viva l’Italia!

In Italy, from now on, if you feel you are victim of mobbing or unjust firing from your employer and you want to take him/her/it (big companies) to the labour court, you can by paying EUR233, not refundable (even in case you win the case). I suggest to the Italian government to extend this criterion on trials for  mafia, terrorism, money-laundering, corruption, etc. Probably, in a couple of years time we could get the documented cases of violation of workers’ rights, rackets, etc. to decrease significantly. Eventually, we will get rid of useless judges, courts, and Constitution. Reaching the first realised utopia of a country without crime. Honest people, migrate to our country, to live in harmony in the cradle of Roman law!

Italy is a Democratic Republic, founded on work. Sovereignty belongs to the people and is exercised by the people in the forms and within the limits of the Constitution. (Constitution of The Italian Republic, Art. 1)

Einstein today

We should be on our guard not to overestimate science and scientific methods when it is a question of human problems; and we should not assume that experts are the only ones who have a right to express themselves on questions affecting the organization of society.

I am convinced there is only one way to eliminate these grave evils, namely through the establishment of a socialist economy, accompanied by an educational system which would be oriented toward social goals. In such an economy, the means of production are owned by society itself and are utilized in a planned fashion. A planned economy, which adjusts production to the needs of the community, would distribute the work to be done among all those able to work and would guarantee a livelihood to every man, woman, and child. The education of the individual, in addition to promoting his own innate abilities, would attempt to develop in him a sense of responsibility for his fellow men in place of the glorification of power and success in our present society.

Above, two extracts from an article by Albert Einstein written in 1949. In this, you can enjoy the greatness of a complete thinker, gifted with the ability to decompose the complexity of the society and give back a consistent theory of it. What would be his analysis of our society/ies? Is it possible to re-think socialism nowadays, really inspired by solidarity among human beings? Where do we locate the Chinese model in this debate?

Pesticides, IQ, social climbing – A pesticide poverty trap

Two different studies by Berkelay and Columbia university confirm that children of women who assumed food with pesticides during their pregnancy, after 7 year from birth, perform worse in IQ tests than peer whose mothers ate free-pesticide food in the same period. The researchers controlled for eduction and environmental elements which could bias the results, too. Unfortunately, the paper is not downloadable freely (it would be interesting to analyse for some selection bias issues). As a development economist, I am concerned with poverty issues. That is, it is likely that poorer families are more exposed to pesticides than wealthier ones, pushing poorer children into a pesticide poverty trap which does not allow (at best, obstacles) them to social climbing. Especially, I am thinking to developing countries or BRICS, where need for growth could lead government to subside cropping technologies which spur agricultural productivity at the expense of health conditions.

China’s growth – the trade-off investments vs consumption and Japanese lessons

Martin Wolf on the Financial Times  provides an analysis about how China’s growth could slow down and, eventually, end in a bump because of the investments-led growth strategy. There are signals that the investments are lowering their returns and the consumption is growing too slowly in order to fill up the gap. Among the other arguments given by the author, there are the ‘middle-income trap’ and the size of China. I need some expert macro-economist telling me whether the parallelism with Japan is too rash. It is worth for a deep reading!

When will China overtake USA, truly?

Yao Yang on Project Syndicate about the issues emerging when comparing China’s and US’ economies in terms of purchasing power parity (PPP). Complex concepts, tough explained so neatly. Enjoy it!

Berlusconi’s defeat in local elections – The end of an era?

Belusconi’s empire in Italy seems at its end. In mayors’ elections, his party (PDL) lost a symbol of the Italian right wing, Milan, the Italian economic capital (PM’s home town, too!). The end of an era, maybe. However, nothing can erase the fact that he ruled Italy for almost 2 decades, influencing with his conflict of interest and numerous charges, his verbal violence and his populism. Berlusconi is one among the cause of Italian decadence nowadays, but it was the effect of the of the empty spaces left from politics in the 90s. Now, a new approach is necessary. Politics has to find its lost nobility in Italy (perhaps, not only in Italy), without spirit of revenge or deafness towards opponents’ reasons, though building a dialectic and democratic dialogue, based on shared principles, written in our Constitution.

The Italian vote on: Bloomerg, BBC, CNN, La Repubblica, Il Giornale.

To Practice 7 – I’m not the World Bank

Before starting my Nicaraguan experience, dreams of glory flooded my ego. Reviewing all my studies on data collection, I knew it would have been a priority to collect data from households survey as fast as possible. Because interviewing the first household of the community today and the last one in 6-months time means introducing a significant source of bias in the research, especially in rural economies subject to seasonality. However, I’m realising that I was naive. Because we’re just two interviewers, because I didn’t consider my stomach could have left me in my room for one entire week, because I go to and from the community using my stick, not my jeep. In other words, because I’m not the World Bank, nor Indiana Jones. To sum up, my goal is to have enough large control group to match with the treatment of the households who participate to a micro credit scheme – the ‘Small Business Program. I have two possibilities: surrendering to the extended collection time bias or opting for a second best solution. I’m trying to move toward the second option. In parallel with the community survey, I’m working with students of a technical high school to assess the program Education That Pays For Itself, run by the English NGO, Teach A Man To Fish. Among the NGO’s outcomes of interests, there is the socioeconomic condition of the students’ families. Hence, I’ training the students to interview their families (belonging to other communities though), in order to increase quickly the treatment group. Therefore, I can focus my interviews in the community on treatment group mainly, shortening the data collection timing, and getting information on students’ families. The assumption is that from the students’ families, coming from rural background in Nicaragua, a PSM approach will allow me to select part of my control group. I fancy to better solutions…

P.S. Edgar, I know you’re thinking about the horse!