Re-evaluating RCTs with nightlights - an example from biometric smartcards in India

Abstract

Satellite data and randomized controlled trials (RCTs) are a powerful combination for analyzing causal effects beyond traditional survey-based indicators. The usage of remotely collected data for evaluating RCTs is cost-effective, objective and possible for anyone with treatment assignment data. By re-evaluating one of the largest RCTs - the smartcard intervention of Muralidharan et al. (2016) covering 20 million people - with Indian nighttime luminosity, this paper finds that nightlights as a specific type of satellite data likely often are too noisy to evaluate RCTs. Building upon a post-treatment and a Difference-in-Differences approach, we do not find any statistically significant effects of the biometric smartcards on nightlights, contrasting Muralidharan et al. (2017)'s results of higher income level in treated areas. This can be mainly explained either with the noisiness-caused inability of nightlights to specifically capture economic effects or the absence of an increased economic activity due to a simple redistributive effect of the intervention. The former is more likely when looking at GDP implications of the noisiness in the luminosity data. Per head estimates, sensitivity checks for spillovers, subdistrict-level instead of village-level observations and different time-wise aggregations of nightlight data do not lead to changed results. Although limited with nightlights, nonetheless, the potential for re-evaluating RCTs with satellite data in general is enormous in three ways: (1) For confirming claimed treatment effects, (2) to understand additional impacts and (3) for cost-effectively understanding long-term impacts of interventions. Using daytime imagery for analyzing RCTs is a promising direction for future research.