Chapter 18 Monitoring and Evaluation

• Lead disaster agency
• Agencies involved in reconstruction
• Project managers
• Affected communities
  

• As with assessments, monitoring and evaluation can take place at whatever level is relevant to the organization seeking the information, which will be similar levels to those at which assessments are conducted.
• Define and agree with stakeholders what will be monitored and evaluated early in project development.
• A mix of qualitative and quantitative approaches is likely to be the most useful for monitoring and evaluation in a post-disaster situation. Participatory performance monitoring and households surveys are two especially useful qualitative tools.
• Assessment data are a critical source of baseline information for evaluation, another reason to promote the sharing of this information among agencies.
• Government can simplify the task of tracking reconstruction if it provides agencies with guidance on the indicators it wishes to be monitored at the project level. The indicators to be monitored should be based on the reconstruction policy.
• Good monitoring and evaluation principles are not different in a post-disaster situation, but to apply them may require more flexibility and imagination.
• If government is not prepared to aggregate data collection from multiple agencies to monitor reconstruction, agencies in one sector or region should consider coordinating the monitoring among themselves.

There are countless audiences for the information that comes from the monitoring and evaluation of post-disaster projects, including funders, government, executing agencies, the general public, and—of course—the affected community.

In Chapter 2, Assessing Damage and Setting Reconstruction Policy, it is suggested that information gathered and produced in a post-disaster assessment might be looked at as a “public good.” A similar case can be made for monitoring and evaluation results, given the large number of stakeholders for most reconstruction projects.

Yet monitoring and evaluation of humanitarian and development activities, while often attempted, are not always that effective as tools to communicate results. Monitoring and evaluation can be even more difficult for disaster-related projects: project assessments and designs may have been hastily prepared, baselines are often not established, and the necessary data might be hard to collect. But good monitoring and evaluation not only improve project outcomes for stakeholders, they have the potential to contribute to international understanding of what “works” in reconstruction—knowledge that is still in somewhat short supply.

Many good tools and resources are available for monitoring and evaluation under “normal conditions.” However, few methodologies have been adapted specifically to the disaster environment. Even so, this chapter argues for a rigorous, yet participatory and flexible approach to monitoring and evaluation in all aspects of housing and community reconstruction.

1. The lead disaster agency should decide how monitoring and evaluation will be carried out within the reconstruction program.
2. The lead disaster agency, in consultation with agencies involved in reconstruction, should decide how information on expenditures and progress at the project level will be tracked and reported, in order to facilitate consolidation.
3. Agencies involved in reconstruction should jointly define protocols for collecting and consolidating sector information, in the absence of government guidance.
4. Agencies involved in reconstruction should decide how they can involve affected communities in monitoring and evaluation activities.
5. Agencies involved in reconstruction should decide how the results of monitoring and evaluation activities will be shared with the affected community and the general public.
6. Affected communities should demand that monitoring and evaluation provide objective project results, which may imply contracting third-party evaluators to conduct them.
   

Monitoring, like assessments, may be an area where there are efficiencies in collaboration, but not necessarily the right incentives. Government should consider establishing protocols for the collection and reporting of post-disaster data, in order to facilitate collection, consolidation, and analysis at the national level. Rules may also be needed to establish minimum parameters for the M&E of projects and to require the disclosure of results. With these rules in place, government can track the progress and the effectiveness of all expenditures related to the disaster and of all the projects being carried out.

• Monitoring: to assess whether a program is being implemented as was planned. A program monitoring system enables continuous feedback on the status of program implementation, identifying specific problems as they arise.
• Process evaluation: to analyze how the program operates. Focuses on problems in service delivery.
• Cost-benefit or cost-effectiveness evaluation: to assess program costs (monetary or non-monetary), in particular their relation to alternative uses of the same resources and to the benefits being produced by the program.
• Impact evaluation: to determine whether the program had the desired effects on individuals, households, and institutions, and whether those effects are attributable to the program intervention. (A detailed discussion of impact evaluation is found inAnnex 1, How to Do It: Conducting an Impact Evaluation of a Reconstruction Project.)
  

In an evaluation of the 1998 Armenia, Colombia, earthquake reconstruction,^[3] Gonzalo Lizarralde proposes the following as the aspects of post-disaster housing project to evaluate:

• National reconstruction program (multi-sectoral) M&E
• Housing and community sector-level M&E
• Program or project-level M&E for a specific reconstruction project (not an assessment level)
• Household-level M&E (generally collected using household surveys) (see thecase study, below, on the unexpected results of a household survey following the 2004 Indian Ocean tsunami reconstruction in Indonesia)

The following table compares the separate characteristics of M&E at each level and shows the responsible party.

The results framework. Monitoring and evaluation take place in the context of a strategic dialogue among development agencies and their governmental clients about “aid effectiveness.” Many development agencies, including the World Bank, have in the past few years oriented their development interventions to conform and contribute to the “Managing for Development Results” agenda. This approach combines a coherent framework for development effectiveness with practical tools for strategic planning, risk management, progress monitoring, and outcome evaluation. For maximum effect, it requires:

• objectives that are clearly stated in terms of expected outcomes and beneficiaries;
• intermediate and higher-order outcome indicators and targets;
• systematic monitoring and reporting;
• demand for results by partner countries and development agencies alike;
• an effective and continuous dialogue on results; and
• strengthening of country capacity to manage for results.

These principles were endorsed in the Rome Declaration on Harmonization in February 2003 and further developed by the Organisation for Economic Co-Operation and Development (OECD) in various reference materials.^[10]

As a result of these agreements, a number of agencies, including the World Bank and the U.S. Agency for International Development, now use the “results framework” to organize and report on project processes and outcomes. Results-based management and results frameworks are similar to logical frameworks (discussed below), but they take a broader look at the context of the project in an organization. While often used for strategic planning, these resources are useful for project-level design as well.

A results-based approach aims to improve management effectiveness and accountability by defining realistic expected results, monitoring progress toward the achievement of expected results, integrating lessons learned into management decisions, and reporting on performance. Inputs and the activities that transform them into outputs reflect the process of implementing projects and program rather than desirable end results in themselves. The results framework presents project objectives and indicators in the following format.

This matrix is accompanied by a second matrix that describes in detail the baseline data for key indicators, the target values, and the data collection and reporting arrangements.

The logical framework matrix. The logical framework matrix (LFM) is a project “snapshot” that is still used by a number of international agencies. It is an instrument for arranging the 10 questions listed above in a logical, succinct way, to define project, program, or policy objectives, and to identify expected causal links (the “program logic”), outcomes, and impact. It also helps identify indicators for M&E at each stage, as well as potential risks.

The case study, below, on the reconstruction in Colombia following the 1999 Armenia earthquake, shows how the results of the project were reported using an LFM.

Availability of monitoring and evaluation data. M&E should cover processes, costs and benefits, and impacts. The project’s design and results framework or logical framework will help define what specifically should be monitored and evaluated. “Output” and “activity” data will be generated by the project’s own monitoring and financial systems; the project should be set up to facilitate the collection of these data. Other data may come from the national-level tracking system and/or surveys and data-gathering exercises that government and donors may conduct jointly. The following table shows some potential sources of baseline and M&E data.

The disaster environment. Above all, the disaster environment itself may be what makes M&E so difficult. The World Bank states that an impact evaluation is intended to determine whether a program had the desired effects on individuals, households, communities, and institutions, and whether those effects are attributable to the program intervention. But when there are multiple agencies implementing multiple interventions in the same locality, it may be difficult to attribute impact to any one project. In addition, some of the results sought from post-disaster projects are qualitative or difficult to measure (“commitment to building back better” or even “greater community participation”). Disaster projects are sometimes designed rapidly with insufficient information, necessitating adjustments during implementation, and making agencies reticent to have their work “judged.” And, there is apt to be turnover and inexperience in the executing agency and higher priorities in government than providing census or other data to donors. Therefore, project designers should be practical when identifying indicators and means of verification for post-disaster projects. Third parties may be needed to collect the data or run the monitoring program altogether. But government should not hesitate to establish rules for monitoring and to send the message that agencies must be accountable for the resources they are spending. For a detailed discussion of post-disaster impact evaluation, see Annex 1, How to Do It: Conducting an Impact Evaluation of a Reconstruction Project.

• the legality and regularity of project expenditures and income, in accordance with laws, regulations, and contracts, such as loan contracts and accounting rules;
• the efficiency of the use of project funds measured against accepted financial practices; and
• the effectiveness of the use of project funds, that is, whether they were used for the intended purposes.

SeeChapter 19, Mitigating the Risk of Corruption, for a discussion of the purposes of audits and the types that may be useful in post-disaster reconstruction projects.

Social audits. Social audits are a special form of audit that is used for “participatory performance monitoring” purposes. With social audits, the public and the affected community oversee and report on an organization’s activities or a reconstruction project. With proper supervision, participatory performance monitoring can be used to collect either qualitative or quantitative information for M&E. For details on conducting a social audit and a summary of other participatory performance monitoring mechanisms, seeAnnex 2, How to Do It: Conducting a Social Audit of a Reconstruction Project.

• Poor coordination within the housing sector or with government prevents data from being aggregated across the sector.
• Assessment data are not shared among agencies, government, and others stakeholders, causing inconsistencies and excess data collection costs.
• Lack of baseline data for projects or insufficient time to develop baselines.
• Project staff lacks commitment to monitoring, leading to delays in the implementation and limited availability of M&E information by project managers.
• Participatory M&E methods are not employed because of limited capacity or lack of commitment.
• Multiple baselines are developed and multiple monitoring indicators are used.
• Evaluations are conducted “in-house” and don’t convey actual project results.
• Information from M&E systems is not shared so learning about which reconstruction interventions are effective does not take place.

1. Take M&E seriously in housing and community reconstruction, in spite of the complexities of the post-disaster environment.
2. At the same time, be realistic about the challenges and design a monitoring system that is easily manageable while producing reliable results.
3. Consider conducting an impact evaluation using qualitative methods, alone or in combination with quantitative methods.
4. Take advantage of the knowledge gained during assessment when designing the M&E processes and establishing the project baseline.
5. Work with government and other donors to harmonize M&E indicators, so that information on project results can be compared and aggregated.
6. To avoid bias when conducting program evaluations, use objective, experienced evaluators. It is most likely that this will entail hiring outside evaluation experts.
7. Don’t just monitor the affected community. Involve them in project M&E, using social audits or other participatory monitoring methods, and make sure the community receives the results.

The Indonesian organization Urban Poor Linkage Indonesia (UPLINK) is a national coalition of NGOs and community-based organizations that focuses on urban poor issues. UPLINK provided emergency help and housing reconstruction assistance to 25 villages in one of the coastal areas that was most affected by the 2004 tsunami in Aceh, Indonesia. The project completed more than 3,300 houses using a participatory approach. The reconstruction work of UPLINK won an international award and was recognized by various national and international technical evaluations for the outstanding quality of the new houses.

While expressing general satisfaction with UPLINK’s work, people voiced some reservations about its high quality. In fact, a survey carried out in 2007 revealed that a significant number of people considered such factors as size, number of rooms, provision of a kitchen or a porch and furniture, and an overall “modern” house more important than quality or protection from future disasters. Houses with inferior quality but with free furniture were more appreciated than the high-quality, unfurnished houses provided by UPLINK. In addition, people expressed a willingness to forgo the participatory process used by UPLINK if a contractor-built house was bigger and looked more modern! This indicates that in evaluating housing assistance options, people evaluate numerous features, including size, design, and amenities of the housing package, and that important considerations for funding agencies, such as the quality and safety of construction, are not necessarily a priority for the homeowners and accordingly are unlikely to be considered when families begin expanding their homes. In the case of UPLINK, people surveyed considered the quality as “excessive” and would have preferred a more standard quality in exchange for a little extra space, a kitchen, or a porch.

Source: Jennifer Duyne Barenstein et al., 2008, People-Driven Reconstruction and Rehabilitation in Aceh: A Review of Uplink’s Concepts, Strategies and Achievements (Aachen: Misereor).

Damages from the 1999 Armenia, Colombia, earthquake were estimated at US$1.6 billion dollars. Some 560,000 people suffered direct earthquake losses, and 1.5 million more residing in 5 departments (provinces) and 28 municipalities in the region were affected indirectly.

The development objective for the World Bank’s Earthquake Recovery Project for reconstruction after the earthquake was “to assist project beneficiaries to normalize economic and social activities through the restoration of essential housing and basic infrastructure built according to adequate seismic standards.” Components of the project included (1) grants of up to US$6,000 for shelter assistance to homeowners who met established criteria and for new houses for renters in vulnerable groups, (2) rehabilitation and retrofitting of social infrastructure, (3) rehabilitation of public infrastructure, (4) capacity building for natural disaster management, (5) social capital restoration, and (6) project management. The project was approved on March 21, 2000, and closed on August 20, 2002.

The reconstruction program was under permanent control, monitoring, and auditing by public entities, such as the General Controller’s Office, as well as private entities and citizen oversight groups, which permitted a guarantee from all stakeholders that the projects were being properly executed both physically and financially. Official project monitoring responsibility was contracted to a consortium of universities. Bank staff conducted nine monitoring missions to Colombia during execution of the loan.

The project was given a highly satisfactory rating in the Implementation Completion and Results Report dated January 10, 2003.^[13] The following were the results of the project.

Source: World Bank, 2003, “Implementation Completion Report (FSLT-70090), Colombia Earthquake Recovery Project,” http://web.worldbank.org/external/projects/main?menuPK=51447259&pagePK=51351007&piPK=64675967&theSitePK=40941&menuPK=64187510&searchMenuPK=51351213&theSitePK=40941&entityID=000094946_0301300401260&searchMenuPK=51351213&theSitePK=40941.

Catley, Andrew, et al. 2008. Participatory Impact Assessment—A Guide for Practitioners. Boston: Feinstein International Center at Tufts University. https://wikis.uit.tufts.edu/confluence/display/FIC/Participatory+Impact+Assessment--+a+Guide+for+Practitioners.

Field, Erica and Michael Kremer. 2006. Impact Evaluation for Slum Upgrading Interventions. Washington, DC: World Bank. http://siteresources.worldbank.org/INTISPMA/Resources/383704-1146752240884/Doing_ie_series_03.pdf.

Goicoechea, Ana. 2008. “Preparing Surveys for Urban Upgrading Interventions: Prototype Survey Instrument and User Guide.” http://siteresources.worldbank.org/INTURBANDEVELOPMENT/Resources/336387-1169585750379/UP-6_Surveys.pdf.

Molund, Stefan, and Göran Schill. 2007. Looking Back, Moving Forward: SIDA Evaluation Manual. 2nd ed. Stockholm: Swedish International Development Agency. http://www.sida.se/sida/jsp/sida.jsp?d=118&a=3148&language=en_US.

Organisation for Economic Co-operation and Development. Principles for Evaluation of Development Assistance. Paris: OECD. http://www.oecd.org/dataoecd/31/12/2755284.pdf.

ProVention Consortium. n.d. M&E Sourcebook. Geneva: ProVention Consortium Secretariat IFRC. http://www.proventionconsortium.org/?pageid=62.

UN-HABITAT. 2001. Guidelines for the Evaluation of Post Disaster Programmes: A Resource Guide. Nairobi: UNCHS (UN-HABITAT). http://www.unhabitat.org/content.asp?cid=1264&catid=286&typeid=16&subMenuId=0.

World Bank. 1996. Performance Monitoring Indicators: A Handbook for Task Managers. Washington, DC: World Bank. http://www.worldbank.org/html/opr/pmi/pmi.pdf.

World Bank. 2000. Key Performance Indicator Handbook. Washington, DC: World Bank. http://info.worldbank.org/etools/library/view_p.asp?lprogram=3&objectid=38956.  

World Bank. 2004. Monitoring and Evaluation: Some Tools, Methods and Approaches. Washington, DC: World Bank. http://lnweb90.worldbank.org/oed/oeddoclib.nsf/InterLandingPagesByUNID/A5EFBB5D776B67D285256B1E0079C9A3.

World Bank. 2004. Ten Steps to a Results-Based Monitoring and Evaluation System. Washington, DC: World Bank. http://go.worldbank.org/C5TSRIQPR0.

World Bank. “Impact Evaluation,” http://go.worldbank.org/2DHMCRFFT2. Contains extensive tools and resources on impact evaluation. 

The elements of a comprehensive project evaluation are the following.

• Monitoring: to assess whether a program is being implemented as was planned. A program monitoring system enables continuous feedback on the status of program implementation, identifying specific problems as they arise.
• Process evaluation: to analyze how the program operates. Focuses on problems in service delivery.
• Cost-benefit or cost-effectiveness evaluation: to assess program costs (monetary or non- monetary), in particular their relation to alternative uses of the same resources and to the benefits being produced by the program.
• Impact evaluation: to determine more broadly whether the program had the desired effects on individuals, households, and institutions, and whether those effects are attributable to the program intervention.
  

The additional effort and resources required for conducting impact evaluations are best mobilized when the project is innovative and replicable, involves substantial resource allocations, and has well-defined interventions. Impact evaluations can also explore unintended consequences, whether positive or negative, on beneficiaries.

Before carrying out an impact evaluation, it must be determined whether one is warranted. The costs and benefits should be assessed, and consideration should be given as to whether another approach, such as monitoring of key performance indicators or a process evaluation, would be adequate or more appropriate. An impact evaluation requires:

• a need and desire to assess the causality associated with the project;
• strong political and financial support; and
• an ability to “net out” the effect of the interventions from other factors through the use of comparison or control groups.

Qualitative techniques are also used for carrying out impact evaluation, without attempting to make a causal connection. The focus is on understanding processes, behaviors, and conditions as they are perceived by the individuals or groups being studied. There are risks associated with using qualitative methods alone to evaluate impact, including subjectivity in data collection, the lack of statistical robustness, and generally small sample sizes, which make it difficult to generalize to a larger, representative population. The validity and reliability of qualitative data are very dependent on the skill of the evaluator and field staff in interpreting the information they collect.

However, there are benefits from using qualitative information that might be especially relevant in the post-disaster context, where a quantitative impact evaluation may be impossible. Qualitative assessments are flexible, can be carried out using rapid techniques, and can employ novel data collection approaches. They can also enhance other elements of the impact evaluation by providing an understanding of stakeholder perceptions and priorities that may in turn have affected program impact. The affected population can even play a role in qualitative evaluation, using such tools as participatory monitoring. Three participatory monitoring techniques are described in this chapter in Annex 2, How to Do It: Conducting a Social Audit of a Reconstruction Project.

A logical framework or results framework can provide the basis for identifying the goals of the project and the information needs for the evaluation. If either of these has been prepared during project preparation, it should serve as the starting point for defining objectives and issues for the evaluation. If not, it can be developed in preparation for the evaluation. Reviewing other evaluation components, such as cost-effectiveness or process evaluations, may also be important objectives of a study and can complement the impact evaluation. A process evaluation can assess the procedures, dynamics, norms, and constraints under which a particular program is carried out. Qualitative and participatory methods can also be used to assess impact.

No evaluation technique or set of techniques is perfect. The evaluator must make decisions about the tradeoffs for each method chosen during the planning of the evaluation.

Although each impact evaluation will have unique characteristics requiring different approaches, a best practice impact evaluation should include:

• an estimate of the counterfactual by (1) using random assignment to create a control group (experimental design), and (2) appropriately and carefully using other methods, such as matching to create a comparison group (quasi-experimental design);
• to control for pre- and post-program differences in participants, and to establish program impacts, relevant data collected at baseline and follow-up (including sufficient time frame to allow for program impacts);
• sufficiently sized treatment and comparison groups to establish statistical inferences with minimal attrition;
• cost-benefit or cost-effectiveness analysis to measure project efficiency; and
• qualitative techniques to allow for the triangulation of findings.

Identifying a control group is challenging under ordinary circumstances; for a post-disaster project, it may be considerably harder. This may be due to the fact, for instance, that the project takes place only in a specific location (where the disaster occurred), that those who participate have special characteristics (all members of the affected population whose houses were destroyed), or that it may not be ethical to withhold assistance from some who were affected. Selecting the control group can be accomplished using methodologies that fall into two broad categories: experimental designs (randomized) and quasi-experimental designs (nonrandomized).

1. During Project Identification and Preparation
2. Clarify objectives of the evaluation
3. Explore data availability
4. Design the evaluation
5. Form the evaluation team
6. If data will be collected:
   1. Select sample design
   2. Develop data collection instrument
   3. Staff and train fieldwork personnel
   4. Pilot test
   5. Data collection
   6. Data management and access

During Project Implementation

1. Collect data on an ongoing basis
2. Analyze the data
3. Write up the findings and discuss them with policy makers and other stakeholders
4. Incorporate the findings in project design

According to the World Bank,^[15] slum upgrading consists of physical, social, economic, organizational, and environmental improvements within neighborhoods. These projects may be undertaken by citizens, community groups, businesses, and local and national authorities. Typical actions include:

• regularizing security of tenure through property mapping, titling and registration;
• installing or improving basic infrastructure, including water, waste collection, storm drainage, electricity, security lighting, and public telephones;
• removal or mitigation of environmental hazards;
• providing incentives for community management and maintenance;
• constructing or rehabilitating community facilities, such as nurseries, health posts, and community centers;
• home improvement, including material upgrading, new construction, and expansion of existing structures;
• improving access to health care and education and programs to address community issues, such as crime and substance abuse;
• enhancement of income-earning opportunities through training and micro-credit; and
• crime control.

Some of the challenges faced in slum upgrading evaluations that are relevant to evaluations of housing and community projects are discussed below.

There are multiple data sources that may help reduce data collection costs for an impact evaluation, such as administrative data, household survey data, census data, facility survey data, industry data, specialized survey data, participatory assessments, and geographic information system and global positioning system data.

Household surveys are essential analysis tools for collecting information on satisfaction and other project results at the household level. A census covers the whole population in the country. A survey covers only a subset—generally a small fraction—of all households. Common survey types include single-topic surveys, multi-topic surveys, demographic and health surveys, employment surveys, rapid monitoring surveys, service satisfaction surveys, and specialized surveys. ^[19]

The use of household surveys has become increasingly widespread around the world, as has the effort to standardize survey design and survey indicators. Government may be able to supply survey data for an impact evaluation. There are also a number of online sources of survey data, as shown below.

An impact evaluation should be carried out only by evaluation experts. The team may include a combination of international and national consultants. A proposed evaluation design should be provided to the consultants or should be developed by them and approved by the agency contracting the consultancy.

A basic impact evaluation team includes an impact evaluation manager, a lead researcher and a research assistant, field supervisors, and interviewers.

• The manager clarifies the objectives of the evaluation, taking into account the client’s needs, drafting the terms of reference for the team members, reaching agreement among the team members and the client about implications of the implementation of the evaluation, and coordinating the field work.
• The lead researcher, usually an economist, selects the evaluation methodology; defines and supervises the data collection strategy. including survey and sampling design; supervises the field work; conducts the quantitative analysis; and writes the evaluation report.
• The research assistant gives support to the lead researcher, especially when it comes to designing the data entry programs and processing data to produce basic results for the qualitative analysis.
• Field supervisors oversee the interviewees and other data collection in the field. Use of national consultants to carry out the field work is a common practice to overcome language and cultural barriers.
  

Some evaluation teams include a sociologist and/or an anthropologist to ensure community participation and to perform the qualitative analysis. A fieldwork manager may be needed to supervise data collection, including scheduling the field supervisors and interviewers. The team should coordinate its work with government officials in relevant sectors. Examples of terms of reference for an impact evaluation are available from the World Bank. ^[20]