Conclusion

This concluding chapter offers some guide on how to run a field study. First, researchers should think about where, when, and with whom they will run their experiment, and make sure these parameters fit the underlying idea or theory they intend to test. Second, every question in a survey should have a purpose. Researchers should be mindful that subtle features of a survey like response scales and order of questions can influence the results. Third, researchers should make sure that their implementing partner understands what it will take to conduct a research. Fourth, researchers should make an intentional decision about how, and how much, to incorporate technology into their survey. Fifth, researchers should not assume people will sign up to receive a program or service. They should find out directly whenever possible by piloting or otherwise gauging demand for their intervention.

Poultry Loans

This chapter looks at the Poultry Loan, a credit-to-savings intervention. Through an informational and promotional campaign, the researchers recruited fifty participants willing to try poultry farming and launched a pilot. At this stage the research did not include an impact evaluation but just sought to determine whether the model was operationally viable. Strictly speaking, this case is not a failure. It is actually a prime example of an appropriate pilot. That said, even the limited pilot offers failures worth discussing, all related to the research setting. First, at the front line of implementation, local financial services organization Sahastradhara KGFS's unexpectedly lengthy software update caused a timing problem. Second, the complexity of the intervention made it challenging to roll out all at once. The third lesson goes back to the beginning of the story, before Poultry Loans were even an idea.

Interest Rate Sensitivity

This chapter details a study conducted with Opportunity International Savings and Loans, Ltd. (OISL)—one of Ghana's largest microfinance institutions—which analyzes the implications of interest rate for both revenue and outreach. The basic concept was simple: market loans to different people using a range of interest rates and observe how many and what kinds of people respond to the offer. The single biggest hang-up was the guarantor requirement. Most applicants had a hard time finding family or friends who could commit to cover a loan; it was also a hassle to do the paperwork. On the surface, this is a simple case of low participation. Far fewer clients took loans than was projected in the pilot, slashing the study's power. That so many clients dropped out because of the sheer duration of the application process suggests a second kind of failure: the study placed too high a burden on OISL's staff.

Survey and Measurement Execution Problems

This chapter assesses survey and measurement execution problems in field research. Until recently, the vast majority of surveys in development field studies were done the old-fashioned way, on clipboards with pen and paper. The past five years have seen a huge shift toward electronic data collection using laptops, personal digital assistants (PDAs), or even smartphones. This has several advantages but also poses risks. It requires electricity to charge devices, often a challenge in rural areas of developing countries. Still, even as laptops, tablets, PDAs, and other technologies are incorporated, surveying remains a very human process. On the upside: surveyors can adapt, interpret, and problem-solve when necessary. On the downside: surveyors can adapt, interpret, and problem-solve whenever they want, which can substantially impact respondents' answers. Meanwhile, some researchers prefer to use measurement tools that capture data directly, without asking questions. The problem with measurement tools is that they do not always work as advertised.

Partner Organization Challenges

This chapter looks at the challenges that arise from working with partner organizations. Evaluations provide evidence about effectiveness and impact, and offer operational insights that can spur improvements to products and processes. From a researcher's perspective, partnerships with practitioners offer access to the people who live and breathe the issues in question. However, along with the mutual benefits of partnerships come unique challenges, one of which is limited staff flexibility and bandwidth. Partner organization staff typically have tremendous expertise, but often experimental protocols require making changes to familiar tasks. This raises the question of staff capacity and appetite to learn new skills and routines. As such, researchers should seek out partners who genuinely want to learn about their programs and products; who are ready, willing, and able to dedicate an appropriate amount of organizational capacity to research; and who are open to the possibility that not all the answers will be positive.

Bundling Credit and Insurance

This chapter assesses a study conducted with SKS Microfinance and insurer ICICI-Lombard where the researchers added a mandatory health insurance policy to SKS microloans to test the theory that bundling policies with other products creates a viable pool of clients for insurers. SKS's bundling of insurance with microloans proved so problematic that, at the end of the day, there were not enough insured clients for researchers to study the impact of getting insurance on health experience or financial performance. The obvious failure here is low participation after randomization. The deeper question is why low participation became an issue. This points to two contributing failures. First, there was a partner organization burden around learning new skills. The second contributing failure can be traced all the way back to the project's inception. Before the study began, SKS had never bundled insurance with its loans. In terms of research setting, they were dealing with an immature product.

Child Health and Business Training with Credit

This chapter examines a study conducted by a microfinance institution (MFI) where they began developing educational supplements for their client base of poor women on the topics of infant/child health and business training. With tailored materials ready, the MFI launched the program in about half of its branches, using an “integrated model” in which loan officers delivered the trainings during their weekly repayment meetings. As it turned out, only a portion of the groups assigned to receive training were actually receiving it, and often at lower intensity than was intended. The underlying failure is that both problems—missed trainings and trainings given to the wrong groups—went unchecked for so long. Moreover, front-line staff members involved in the study faced competing priorities. If loan officers had been more aware of and invested in the research or managers more vigilant, they might have caught these challenges and addressed them before it was too late.

Credit and Financial Literacy Training

This chapter examines a study conducted with the Peruvian microfinance institution Arariwa, which explores a number of questions regarding technology's potential and proper role as a development tool by implementing and testing a multimedia financial education program for clients. In this case, there are two major areas of failures: research setting and partner organization challenges. There were a few distinct instances of the former. First, the field sites presented challenges to the use of technology. Second, the intervention itself was deceptively complex. Finally, there was an element of bad timing in the flooding that caused repayment problems for some clients, thus adding stress to loan officers' already full plates. Indeed, competing priorities were a key partner organization challenge in this case. Loan officers were expected to deliver trainings without any lapses in, or relief from, their basic duties.

Low Participation Rates

This chapter focuses on low participation rates. Low participation rates squeeze the effective sample size for a test, making it more difficult, statistically, to identify a positive treatment effect. There are two moments in which low participation rates can materialize: during the intake process to a study or intervention, or after random assignment to treatment or control. Low participation during the intake process often occurs when marketing a program to the general public. Researchers working in the field with partner organizations often face inflexible constraints in trying to cope with low participation during intake. The second type of low participation—that which occurs after subjects have been randomly assigned to treatment or control—is a more daunting problem and is less likely solvable than low participation at the intake phase.

Technical Design Flaws

This chapter examines technical design flaws. There are two common issues to avoid when it comes to survey design: bloated surveys, particularly without a clear analysis plan for all questions; and poorly designed survey items. No less important than the survey and other data collection tools is the plan to deploy them. As such, researchers should field test survey questions before launch. Also, debrief regularly with field survey teams to find out which questions respondents are struggling with, which parts of the survey are hardest to administer, and the like. The chapter then considers mistakes in randomization, power, and necessary sample size calculations in RCT design. Power and necessary sample size calculations rely on parameters that are hard to observe or guess. The best advice is to run these calculations multiple times, imagining a range of scenarios in the field and using a corresponding range of values for key parameters.