Session 2 defines and discusses concepts usable in studies of impact effect in general, but it goes more deeply into the randomization issues, too. As suggested by the title, the session is focused on the importance of the randomization process in order to define treatment and control group. Below, some basic notions underlined by Dan Levy.
1) We are dealing with random selection when each individual in the population has the same probability to be selected into the treatment or the control group. A random sampling occurs when we select from the population, a random assignement when individuals are allocated randomly to treatment and control group. When randomly selected, on average, treatment and control group are statistically identical.
2) Randomization allows to get rid of sometimes weak economic assumption on which the traditional econometric studies base their conclusions. Moreover, when a random assignment takes place, any difference between treatment and control group after the implementation of the program can be attributed to the effect of the same on the treatment.
3) The counter-factual is what would have been happened in the absence of the program, i.e. it has to be measured at the same moment in time with the treatment. Its proxy is the control group; it is not observable in the reality; it is built randomly and composed by unities (economic individuals) not affected by the program; it has to be defined before the program is implemented.
4) Internal validity is the credibility of the impact design (3ie 2009), external validity is the extent to which the results of a particular experiment can be generalized.
5) A spill-over occurs when treatment and control group interact (causing a bias, luckily negative, on the estimated impact of the program), a cross-over takes place when a unit of the experiment jumps from the control to the treatment and vice-versa.
6) Steps in conducting a successful experiment: (I) detailed design of the study (involving evaluators and evaluated); (II) random assignment; (III) collection of baseline statistics; (IV) verify that the assignment looks random (using tables and hypotheses testing); (V) monitoring the process (following up, avoiding spill- and cross-over); (VI) collection of follow-up statistics; (VII) estimation of the impact; (VIII) statistical and practical assessment of the program.
REFLECTIONS & QUESTIONS
– Is it reasonable to hypothesize that the treatment and the control group change after the assignment with respect to some variant characteristics of the individuals (motivation, enthusiasm, optimism, etc.)? How to account for this source of bias?
– Is it possible to design a system of disincentive to cross-over (e.g. a part of the budget of a project could be used to refund the treatment group for not receiving the program benefits)?
See you soon!!!