Improving last-mile service delivery using phone-based monitoring - Pubdocs.worldbank.org.
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Improving last-mile service delivery using
phone-based monitoring
Karthik Muralidharan1 Paul Niehaus1 Sandip Sukhtankar2
Jeff Weaver3
1 UC San Diego
2 University of Virginia
3 University of Southern California
February 11, 2021
Managing the last mile
I Improving last mile public service delivery in developing
countries is a recurring challenge
I e.g. fraud, corruption, absenteeism, low effort
I Likely to be large returns to improving personnel management
in the public sector (Lemos, Muralidharan, and Scur 2021)
Monitoring Last-mile Service Delivery February 11, 2021 2 / 38
Managing the last mile
I Improving last mile public service delivery in developing
countries is a recurring challenge
I e.g. fraud, corruption, absenteeism, low effort
I Likely to be large returns to improving personnel management
in the public sector (Lemos, Muralidharan, and Scur 2021)
I Key challenge: “can only manage what you can measure”,
and measurement is a challenge
I Hard to measure public sector performance generally, especially
true in low capacity states
I Massively scaled problem across many communities
Monitoring Last-mile Service Delivery February 11, 2021 2 / 38Existing approaches fall short
I Internal reporting by lower layer bureaucrats susceptible to
spin ond overstating performance or understating problems
I Audits are costly, only applicable for some services, and
provide data with significant lags
I Feedback hotlines increasingly popular but underused and
yield non-representative data
Monitoring Last-mile Service Delivery February 11, 2021 3 / 38Phone-based monitoring
I Mobile phone penetration in low-income countries rose from
1% in 2002 to 62% in 2017
I We examine whether this general-purpose technology can be
used to improve public sector performance
I Call beneficiaries and ask about receipt of services
I Leverage government databases of phone numbers and
geographic information to match to individual bureaucrats
I Provides channel to obtain quick, cheap, independent
information about last-mile service delivery across sectors
I Some governments have started using these systems (e.g.
Pakistan’s Citizen Feedback Model)
Monitoring Last-mile Service Delivery February 11, 2021 4 / 38This paper
I In 2018, the Indian state of Telangana created program to
distribute $1.8 billion annually in lump-sum payments to 5.7
million farmers
I We worked with them to implement and evaluate a
phone-based monitoring system
I In a randomly selected set of 132 of 548 mandals, officials
were informed that performance will be measured via calls to
program beneficiaries
I At-scale experiment in terms of implementation, outcome
measurement, and unit of randomization
Monitoring Last-mile Service Delivery February 11, 2021 5 / 38Main results
I Increased the rate of “on-time” delivery of transfers by 3.3%
($3.9 million dollars)
Monitoring Last-mile Service Delivery February 11, 2021 6 / 38Main results
I Increased the rate of “on-time” delivery of transfers by 3.3%
($3.9 million dollars)
I 7.6% reduction in number of farmers who did not get
transfers (17,000 farmers)
Monitoring Last-mile Service Delivery February 11, 2021 6 / 38Main results
I Increased the rate of “on-time” delivery of transfers by 3.3%
($3.9 million dollars)
I 7.6% reduction in number of farmers who did not get
transfers (17,000 farmers)
I Progressive: biggest improvements among smallest farmers
Monitoring Last-mile Service Delivery February 11, 2021 6 / 38Main results
I Increased the rate of “on-time” delivery of transfers by 3.3%
($3.9 million dollars)
I 7.6% reduction in number of farmers who did not get
transfers (17,000 farmers)
I Progressive: biggest improvements among smallest farmers
I Highly cost-effective: cost per dollar delivered of 3.6 cents;
cost per dollar delivered on time was under 1 cent
Monitoring Last-mile Service Delivery February 11, 2021 6 / 38Main results
I Increased the rate of “on-time” delivery of transfers by 3.3%
($3.9 million dollars)
I 7.6% reduction in number of farmers who did not get
transfers (17,000 farmers)
I Progressive: biggest improvements among smallest farmers
I Highly cost-effective: cost per dollar delivered of 3.6 cents;
cost per dollar delivered on time was under 1 cent
I Reasons to think these estimates are conservative as to the
potential of phone-based monitoring more broadly
Monitoring Last-mile Service Delivery February 11, 2021 6 / 38Why is this exciting?
I Solves many problems with collecting actionable performance
information
I Independent source/difficult for targeted bureaucrat to disrupt
I Can scale across wide range of places, programs, and outcomes
I Information can be collected and processed in close to real-time
I Low fixed and variable costs to deploy
Monitoring Last-mile Service Delivery February 11, 2021 7 / 38Why is this exciting?
I Solves many problems with collecting actionable performance
information
I Independent source/difficult for targeted bureaucrat to disrupt
I Can scale across wide range of places, programs, and outcomes
I Information can be collected and processed in close to real-time
I Low fixed and variable costs to deploy
I Top-down monitoring can be effective (Olken, 2007), but
constrained by cost and difficulty of measurement
I Phone-based monitoring increases feasibility of top-down
approaches by reducing measurement costs
Monitoring Last-mile Service Delivery February 11, 2021 7 / 38Why is this exciting?
I Solves many problems with collecting actionable performance
information
I Independent source/difficult for targeted bureaucrat to disrupt
I Can scale across wide range of places, programs, and outcomes
I Information can be collected and processed in close to real-time
I Low fixed and variable costs to deploy
I Top-down monitoring can be effective (Olken, 2007), but
constrained by cost and difficulty of measurement
I Phone-based monitoring increases feasibility of top-down
approaches by reducing measurement costs
I Very nimble evaluation
I Study completed in 3 months (results); 6 months (paper)
I Cost less than USD 50,000
I Yet, done at scale - randomized across 5.7 million beneficiaries
as part of scaled rollout
Monitoring Last-mile Service Delivery February 11, 2021 7 / 38Agenda
Context
Intervention
Empirical Strategy
Results
Effects on program performance
Tallying costs and benefits
Conclusion
Monitoring Last-mile Service Delivery February 11, 2021 7 / 38Agenda
Context
Intervention
Empirical Strategy
Results
Effects on program performance
Tallying costs and benefits
Conclusion
Monitoring Last-mile Service Delivery February 11, 2021 7 / 38Rythu Bandhu Scheme
I Rythu Bandhu Scheme (RBS) launched in May 2018 to
provide lump-sum, unconditional cash transfers to farmers
I Rs. 4000 ($60) per acre to every landowning farmer in both
cropping cycles
I Transfers given as “order checks”, exchangeable for cash or
deposits at designated banks
I $1.8 billion to be disbursed annually (7% of state budget) to
5.7 million farmers
I Mandal (sub-district) agricultural officers (MAOs) were
responsible for check distribution within their mandal(s)
I Average mandal has 10,000 beneficiaries, 60,000 residents
I MAOs organized distribution through meetings in each village
to distribute the checks
Monitoring Last-mile Service Delivery February 11, 2021 8 / 38Rythu Bandhu Scheme
I Many implementation concerns, particularly given the
government of Telangana had never done this before
I Non-delivery of checks to intended beneficiaries
I Late delivery of checks, forcing farmers to take out loans for
time-sensitive agricultural inputs and making it costly to get
the check
I Corruption during the distribution process
Monitoring Last-mile Service Delivery February 11, 2021 9 / 38Rythu Bandhu Scheme
Monitoring Last-mile Service Delivery February 11, 2021 10 / 38Rythu Bandhu Scheme
Monitoring Last-mile Service Delivery February 11, 2021 11 / 38Agenda
Context
Intervention
Empirical Strategy
Results
Effects on program performance
Tallying costs and benefits
Conclusion
Monitoring Last-mile Service Delivery February 11, 2021 11 / 38Phone-based monitoring intervention
I State government had collected phone numbers during land
record digitization (3.5 million numbers, 61% of farmers)
I Contracted private vendor to call random sample of farmers
I Sampled 150 farmers per treatment mandal
I Response rate of ∼ 48%
I Calls collected information on:
I Whether and when farmers received their check
I Whether and when encashed check
I Problems receiving or encashing check
I Overall satisfaction with the program
Monitoring Last-mile Service Delivery February 11, 2021 12 / 38Phone-based monitoring intervention
I Treatment MAOs were informed about intervention one week
before distribution of cheques
I Held video conference (90% attendance), sent letter with
details (68% received)
I Explained how the system worked, information to be collected
I Informed that reports would be issued to them and supervisors
I No explicit incentives based on these reports
I Reports divided performance into: % Checks delivered, speed
of receipt, satisfaction, encashment, corruption
I Gives color-coded grade, raw statistics, and compared to
district and state averages
Monitoring Last-mile Service Delivery February 11, 2021 13 / 38Intervention Report Card
Category
In Mandal District Average State Average Overall Rating
Percent of Checks Recieved
Fair
Percent of Checks Delivered by May 20 Poor
Percent of Satisfied Beneficiaries Excellent
Percent Successfully Encashed Check Good
Percent reporting corruption Excellent
Monitoring Last-mile Service Delivery February 11, 2021 14 / 38Timeline of intervention
I Feb 28, 2018: Rythu Bandhu Scheme is announced
I April 1: Our first meeting with Government of Telangana
I May 2: Treatment MAOs informed of intervention
I May 8: Check distribution begins
I May 23: Half of checks distributed and encashed
I May 29 - June 15: Call center collects data
I June 15: ∼75% of checks had been distributed and encashed
I July 9: Reports sent to treatment MAOs and supervisors
Monitoring Last-mile Service Delivery February 11, 2021 15 / 38Agenda
Context
Intervention
Empirical Strategy
Results
Effects on program performance
Tallying costs and benefits
Conclusion
Monitoring Last-mile Service Delivery February 11, 2021 15 / 38Opportunity to evaluate implementation at scale
I Conducted evaluation in 30/31 districts of Telangana
(excluding Hyderabad)
I Randomized 548 mandals into 132 treatment (122 MAOs)
and 416 control (376 MAOs), randomizing at the MAO level
I Average mandal has ∼10,000 beneficiaries and ∼60,000
residents
I 1.3 million beneficiaries in treatment, 4.3 million beneficiaries
in control
I Estimates are representative at the state level (35 million
residents)
Monitoring Last-mile Service Delivery February 11, 2021 16 / 38Geographic distribution of intervention
Control
Treatment
Monitoring Last-mile Service Delivery February 11, 2021 17 / 38Balance
Control vs. Treatment on pre-determined characteristics
Variable Control mean Treatment mean Difference (SE)
Land registry data
Land size (acres) 2.21 2.18 -0.01 (0.05)
Median land size 1.57 1.56 0.00 (0.05)
Land size - 25th percentile 0.65 0.66 0.02 (0.04)
Land size - 75th percentile 2.96 2.93 -0.03 (0.06)
Registered mobile numbers 0.61 0.61 0.01 (0.01)
Farmer population 11345 10935 -249 (389)
Census 2011 data
Literacy rate 0.60 0.60 -0.00 (0.01)
Share of rural population 0.86 0.85 0.01 (0.02)
Share of working population 0.51 0.51 0.01 (0.00)
Share of SC population 0.18 0.18 -0.00 (0.00)
Share of ST population 0.13 0.14 0.02* (0.01)
Share of irrigated land 0.52 0.51 -0.01(0.04)
Share of electrified villages 0.95 0.94 -0.02(0.02)
Average village distance from Hyderabad 135.91 134.76 0.32(2.09)
Observations 417 131 548
Differences in column (3) are estimated through regressions on a treatment indicator, with fixed effects at the
randomization strata level. Standard errors are clustered
Monitoring at the
Last-mile MAODelivery
Service level and reported inFebruary
parentheses.
11, 2021 18 / 38Balance
Control vs. Treatment on pre-determined characteristics
Variable Control mean Treatment mean Difference (SE)
Other Mandal characteristics
Number of banks in mandal 3.52 4.12 -0.26 (0.35)
Average distance to nearest ATM 12.72 12.43 -0.18 (0.47)
Share of HHs using banking services 0.45 0.43 -0.01 (0.02)
Average distance to nearest bank 7.51 7.22 -0.15 (0.31)
Share of villages with all-weather road 0.91 0.91 0.00 (0.01)
Share of HHs owning mobile phones 0.52 0.50 -0.01 (0.01)
Average rainfall in mandal 2013-17 (mm) 707.35 714.26 8.76 (10.19)
MAO Characteristics
Age of MAO 35.57 36.36 0.89 (0.77)
Gender of MAO (Female = 1) 0.30 0.33 0.02 (0.05)
Number of SHC samples (2017) 1030.01 961.11 -64.65 (54.10)
No. of farmers covered by SHCs (2017) 4470.67 4572.42 34.66 (251.47)
SHC tests conducted (2017) 976.04 924.91 -44.77 (52.32)
SHCs produced (2017) 4176.72 4332.26 85.98 (240.81)
Joint test (p-value) (.225)
Observations 417 131 548
Differences in column (3) are estimated through regressions on a treatment indicator, with fixed effects at the
randomization strata level. Standard errors are clustered
Monitoring at the
Last-mile MAODelivery
Service level and reported inFebruary
parentheses.
11, 2021 19 / 38Outcomes
I The pre-registered primary outcomes of interest for the study
were:
I Check Distribution and Encashment
I Speed of Check Distribution and Encashment
I Beneficiary Satisfaction
I Corruption
Monitoring Last-mile Service Delivery February 11, 2021 20 / 38Outcomes
I The pre-registered primary outcomes of interest for the study
were:
I Check Distribution and Encashment
I Speed of Check Distribution and Encashment
I Beneficiary Satisfaction
I Corruption
I Program was very well implemented, limiting our ability to
observe changes in the latter two outcomes
I 92% of beneficiares were satisfied, corruption reported by only
1.5%
I 83% of checks were encashed, 69% encashed by June 8
I Given the relatively successful implementation, estimates are
likely conservative as to potential effects of phone-based
monitoring
Monitoring Last-mile Service Delivery February 11, 2021 20 / 38Data
I We primarily measure outcomes using administrative records:
I Land register of all 5.7M landholders in the state
I Individual-level bank records of check encashment for all 5.7M
landholders: encashment status and date of encashment
I Secondary sources:
I Call center data (22,565 beneficiaries)
I Data from a phone survey of 142 MAOs
I Individual-level records of check distribution maintained by
MAOs: distribution status and date of distribution
I Soil Health Cards program information from 2016 to 2018
Monitoring Last-mile Service Delivery February 11, 2021 21 / 38Estimation
yivmsd = α + βTmsd + δsd + γXivmsd + eivmsd
I y is an outcome, T is an indicator for treatment assignment
and X is pre-determined covariates
I Indices denote individual i in village v in mandal m in stratum
s in district d
I δsd are randomization strata fixed effects
I Standard errors are clustered at the level of randomization,
i.e. MAO
Monitoring Last-mile Service Delivery February 11, 2021 22 / 38Agenda
Context
Intervention
Empirical Strategy
Results
Effects on program performance
Tallying costs and benefits
Conclusion
Monitoring Last-mile Service Delivery February 11, 2021 22 / 38Treatment Effect On Encashment by Date
.06
.04
Treatment Effect (βTreatment)
.02
0
Coefficient
95% Confidence interval
-.02
07 May 28 May 18 Jun 09 Jul 30 Jul
Date of Encashment
Monitoring Last-mile Service Delivery February 11, 2021 23 / 38Cumulative rates of encashment in treatment and control mandals
.8
.6
% of Checks Encashed
.4
.2
Treatment mandals
Control mandals
0
07 May 28 May 18 Jun 09 Jul 30 Jul
Monitoring Last-mile Service Delivery February 11, 2021 24 / 38On-Time Encashment
Encashed before Ever encashed
June 8th
(1) (2) (3) (4) (5)
Treatment Control Treatment Control Obs.
mean mean
Overall 0.0231∗∗∗ 0.69 5,645,937
(0.00807)
Outcome in header. All specifications include fixed effects at the randomization strata level. Standard errors are
clustered at the MAO level and reported in parentheses.
Monitoring Last-mile Service Delivery February 11, 2021 25 / 38On-Time Encashment
Encashed before Ever encashed
June 8th
(1) (2) (3) (4) (5)
Treatment Control Treatment Control Obs.
mean mean
Overall 0.0231∗∗∗ 0.69 5,645,937
(0.00807)
Land quartiles
Quartile 1 0.0278∗∗∗ 0.52 1,449,482
(0.00960)
Quartile 2 0.0248∗∗∗ 0.71 1,460,294
(0.00791)
Quartile 3 0.0241∗∗∗ 0.76 1,443,788
(0.00755)
Quartile 4 0.0208∗∗ 0.77 1,443,836
(0.00803)
Outcome in header. All specifications include fixed effects at the randomization strata level. Standard errors are
clustered at the MAO level and reported in parentheses.
Monitoring Last-mile Service Delivery February 11, 2021 26 / 38On-Time Encashment
Encashed before Ever encashed
June 8th
(1) (2) (3) (4) (5)
Treatment Control Treatment Control Obs.
mean mean
Overall 0.0231∗∗∗ 0.69 0.0126∗ 0.83 5,645,937
(0.00807) (0.00655)
Land quartiles
Quartile 1 0.0278∗∗∗ 0.52 1,449,482
(0.00960)
Quartile 2 0.0248∗∗∗ 0.71 1,460,294
(0.00791)
Quartile 3 0.0241∗∗∗ 0.76 1,443,788
(0.00755)
Quartile 4 0.0208∗∗ 0.77 1,443,836
(0.00803)
Outcome in header. All specifications include fixed effects at the randomization strata level. Standard errors are
clustered at the MAO level and reported in parentheses.
Monitoring Last-mile Service Delivery February 11, 2021 27 / 38Encashment
Encashed before Ever encashed
June 8th
(1) (2) (3) (4) (5)
Treatment Control Treatment Control Obs.
mean mean
Overall 0.0231∗∗∗ 0.69 0.0126∗ 0.83 5,645,937
(0.00807) (0.00655)
Land quartiles
Quartile 1 0.0278∗∗∗ 0.52 0.0224∗∗ 0.68 1,449,482
(0.00960) (0.00932)
Quartile 2 0.0248∗∗∗ 0.71 0.0145∗∗∗ 0.85 1,460,294
(0.00791) (0.00631)
Quartile 3 0.0241∗∗∗ 0.76 0.0113∗ 0.88 1,443,788
(0.00755) (0.00601)
Quartile 4 0.0208∗∗ 0.77 0.00699 0.89 1,443,836
(0.00803) (0.00621)
Outcome in header. All specifications include fixed effects at the randomization strata level. Standard errors are
clustered at the MAO level and reported in parentheses.
Monitoring Last-mile Service Delivery February 11, 2021 28 / 38Effect of intervention corresponding to encashment dates
Treatment Effect (βTreatment) Land size Quartile 1 Land size Quartile 2
Treatment Effect (βTreatment)
.06
.06
.04
.04
.02
.02
0
0
-.02
-.02
01 May 01 Jun 01 Jul 01 Aug 01 May 01 Jun 01 Jul 01 Aug
Date of Encashment Date of Encashment
Land size Quartile 3 Land size Quartile 4
Treatment Effect (βTreatment)
Treatment Effect (βTreatment)
.06
.06
.04
.04
.02
.02
0
0
-.02
-.02
01 May 01 Jun 01 Jul 01 Aug 01 May 01 Jun 01 Jul 01 Aug
Date of Encashment Date of Encashment
Coefficient
CI Upper/CI Lower
Monitoring Last-mile Service Delivery February 11, 2021 29 / 38Time to Encashment
Effects on distribution and encashment
Days till encashed
(1) (2) (3)
Treatment Control mean Observations
Overall -0.759* 20.16 4,663,678
(0.388)
Land quartiles
Quartile 1 -0.655 23.99 984,251
(0.511)
Quartile 2 -0.676* 20.08 1,239,604
(0.383)
Quartile 3 -0.842** 18.71 1,278,096
(0.359)
Quartile 4 -0.982*** 18.79 1,284,734
(0.367)
Outcome in header. Days elapsed before encashment (conditional on encashment) are counted from 8 May 2018.
All specifications include fixed effects at the randomization strata level. Standard errors are clustered at the MAO
level and reported in parentheses.
Monitoring Last-mile Service Delivery February 11, 2021 30 / 38Phone Ownership and Multi-tasking
I We cannot reject that the treatment effect was the same for
those with and without phones Go
Monitoring Last-mile Service Delivery February 11, 2021 31 / 38Phone Ownership and Multi-tasking
I We cannot reject that the treatment effect was the same for
those with and without phones Go
I Concern that this may have led to worse performance on other
tasks (multi-tasking): test for differences in performance on
2018 production of “Soil Health Cards”
(1) (2) (3) (4)
Number of SHC Number of farmers SHCs available
SHC tests conducted
samples entered covered by SHCs on portal
Treatment -43.54 -138.8 -26.55 -124.9
(46.99) (205.7) (45.27) (203.4)
Control Mean 906.02 4259.53 873.73 3891.78
Observations 512 512 512 512
Monitoring Last-mile Service Delivery February 11, 2021 31 / 38Agreement between call center and administrative data
I In long run, need call center data to be accurate to use for
performance management
I Unique opportunity to evaluate accuracy in this context due to
administrative data
I Agree in 88.6% of cases on whether farmer encashed check
I Key is agreement in aggregate, MAO-level performance
ratings
I Relative performance of each pair (m, m0 ) of MAOs within a
district
I Determine whether relative ranking of a pair agrees across
datasets
I Accounting for sampling variation, disagreement in only 9% of
pairs Go
Monitoring Last-mile Service Delivery February 11, 2021 32 / 38Agenda
Context
Intervention
Empirical Strategy
Results
Effects on program performance
Tallying costs and benefits
Conclusion
Monitoring Last-mile Service Delivery February 11, 2021 32 / 38Cost-effectiveness: cost per incremental dollar delivered
I ITT estimate: increased amount delivered on time by Rs. 203
per farmer; amount ever delivered by Rs. 54 per farmer
I Implies an additional ∼ 3.9 million USD delivered to farmers
on time and ∼1 million USD delivered overall
I Contract with call center cost GoTS $36,000
I ⇒ cost per incremental dollar delivered was 3.6 cents
Monitoring Last-mile Service Delivery February 11, 2021 33 / 38Cost-effectiveness: cost per incremental dollar delivered
I ITT estimate: increased amount delivered on time by Rs. 203
per farmer; amount ever delivered by Rs. 54 per farmer
I Implies an additional ∼ 3.9 million USD delivered to farmers
on time and ∼1 million USD delivered overall
I Contract with call center cost GoTS $36,000
I ⇒ cost per incremental dollar delivered was 3.6 cents
I If extended to the entire state and both crop cycles ⇒ $33.1
million delivered on-time and $8.6 million more delivered
annually
Monitoring Last-mile Service Delivery February 11, 2021 33 / 38Cost-benefit: overall welfare
I As a second exercise, we price the value of getting capital to
farmers during the planting season, instead of sitting with the
gov’t
I Assume 25% return for farmers (loan rate), 5% for
government Go
I Based on these parameters, we estimate that phone-based
monitoring generated Rs. 10 million (∼$150,000) in benefits,
roughly four times our (conservative) estimate of the cost.
We reject the null of no benefit (p = 0.03)
Monitoring Last-mile Service Delivery February 11, 2021 34 / 38Interpretation
I The estimates here are likely conservative in several senses
I Scope for improvement was modest, given excellent
performance in control (83% encashment)
I Effect of measurement and light-touch accountability, but
phone-based monitoring can also be used for
information/alongside formal incentives
I Already was a source of high-quality administrative data
I “Incomplete” compliance, where some treated officials report
being unsure they were treated MAO Survey
Monitoring Last-mile Service Delivery February 11, 2021 35 / 38Conclusion
I Calling beneficiaries to measure their experiences was
cost-effective at improving program performance
I Approach can be applied to a wide-range of service delivery
settings
I Some examples of outbound call-centers, but inbound centers
are far more common and often unused
I Solves many problems with collecting actionable performance
information
I May increase feasibility of top-down monitoring by lowering
cost
I Flexibility to scale across wide range of place, programs, and
outcomes, as well as adapt to new circumstances
Monitoring Last-mile Service Delivery February 11, 2021 36 / 38Next steps
I Long-term, we are building a research and policy agenda
around phone-based monitoring for public performance
management
I We have started testing phone-based monitoring in other
settings
I Testing across different states and sectors with greater scope
for performance improvement
I Measure incentive, informational, and selection effects
I Build phone-based measurement into routinized functioning
and tie outcomes systematically to personnel management
I Test impacts of sharing information at different levels of
government and the public
Monitoring Last-mile Service Delivery February 11, 2021 37 / 38Agreement in aggregate
Agreement between phone and admin data on MAO performance
(1) (2) (3)
Agreement
Actual Residual dis-
rate from
agreement agreement
sampling
rate rate
variation
Pair-wise order of
68.6% 77.6% 9.0%
rankings
Bottom 20% in PD
found in bottom 20% 43.0% 61.7% 18.7%
of AD
Bottom 20% in PD
found in bottom 50% 83.0% 92.7% 9.7%
of AD
Back
Monitoring Last-mile Service Delivery February 11, 2021 38 / 38Impacts across data sources
Comparing administrative and phone-survey data
Administrative data Phone data
(1) (2) (3) (4) (5)
All With With With Reached
phones phones, phones,
sampled reached
Panel A: Distribution status
Treatment 0.801 0.874 0.873 0.878 0.885
Control 0.793 0.870 0.876 0.884 0.880
Difference 0.00901 0.00614 0.00384 0.00326 0.00389
(0.00744) (0.00609) (0.00669) (0.00719) (0.00644)
Panel B: Encashment status
Treatment 0.657 0.727 0.731 0.743 0.757
Control 0.630 0.700 0.711 0.732 0.754
Difference 0.0254*** 0.0229** 0.0221* 0.0128 0.00204
(0.00912) (0.0101) (0.0115) (0.0115) (0.0102)
Observations 5,536,538 3,356,249 44,690 21,835 21,835
Outcome variables reflect distribution and encashment status as of the date the call was made to the respondent.
For respondents who were not called in the survey, the median date of calls made to their district is used as the
cut-off date. All specifications include fixed effects at the randomization strata levels. Standard errors are clustered
at MAO level and reported in parentheses.
Monitoring Last-mile Service Delivery February 11, 2021 39 / 38Cost-effectiveness analysis
I We price the value of putting getting capital to farmers during
the planting season, instead of letting it sit with the gov’t
I Total value of a unit of capital held by the gov’t until time t
and then by the farmer from time t until T is defined as
v (t ) = e rg t e rf (T −t )
where rg is the return on capital held by the gov’t and rf is
the interest rate on loans
I Given a distribution F of check encashment dates, the total
social value created is
Z
W (F ) = v (t )dF (t )
Monitoring Last-mile Service Delivery February 11, 2021 40 / 38Cost-effectiveness analysis
I Based on these parameters, we estimate that phone-based
monitoring generated Rs. 10 million (∼$150,000) in benefits,
roughly four times our (conservative) estimate of the cost.
We reject the null of no benefit (p = 0.03)
I Estimated benefits are 0 by definition at T = 0 and then
increase steadily as we increase T and exceed the costs of the
intervention for any δ (difference between rf and rg ) after 5
June
Monitoring Last-mile Service Delivery February 11, 2021 41 / 38Sensitivity of cost-effectiveness estimates
Cost-benefit vs. total time period (T )
BackMAO Survey
I We surveyed 88 of 122 treatment MAOs, sample of 54 control
MAOs Return
I Incomplete adherence in treatment
I 90% of treatment MAOs had heard of intervention
I Only 28% were sure had been done in their area, 28% were
unsure, 35% said had not
I May be strategic misrepresentation, but suggests results are
lower bound
I Minimal contamination of the control group
I 52% of control MAOs had heard about the intervention
I But only 4% believed themselves treated, 8% were unsure
Monitoring Last-mile Service Delivery February 11, 2021 43 / 38Did Benefits Only Accrue to Those with Phones?
I In general, we cannot reject that the treatment effect was the
same for those with and without phones Go
Heterogeneous effects by phone coverage
Encashed by June 8 Ever encashed Days till encashed
(1) (2) (3) (4) (5) (6)
Treatment Control Treatment Control Treatment Control
mean mean mean
Phone coverage
No listed phone 0.0229∗∗ 0.57 0.00691 0.72 -1.295∗∗∗ 22.14
(0.0116) (0.0116) (0.475)
Listed phone 0.0202∗∗ 0.76 0.0128∗∗ 0.90 -0.475 19.13
(0.00821) (0.00554) (0.396)
All specifications include fixed effects at the randomization strata fixed level. Standard errors are clustered at the
MAO level and reported in parentheses. The bottom row reports the F-statistic and p-value from a test of the null
that coefficients are statistically similar across both categories.
Monitoring Last-mile Service Delivery February 11, 2021 44 / 38Spillovers
Potential Spillover Effect of Intervention
(1) (2)
Ever distributed Ever encashed
Number of treatment mandals in revenue division 0.000679 0.00847
(0.00473) (0.00551)
Constant 0.874 0.818
(0.00696) (0.00807)
Observations 399 399
Tests for the possibility that supervisors of MAOs focused more attention on treatment MAOs. Districts in Telangana
are divided into “revenue divisions”, which each contain several mandals. Although roughly the same fraction of
mandals were treated in each district, we did not stratify the randomization at the revenue division level. As a
result, there is random variation in the fraction of MAOs within each revenue division that are treated. If there
were diversion of revenue division supervisor-level attention and attention matters for performance, we should expect
worse performance among control MAOs with more treated MAOs in their revenue division, as these control MAOs
would get less attention paid to them. Outcome in header. All specifications include fixed effects for districts and
number of mandals in the revenue division. Standard errors in parentheses and clustered at the revenue division level.
17 mandals could not be matched to revenue divisions, so were not included.
Monitoring Last-mile Service Delivery February 11, 2021 45 / 38You can also read
