Define Measures and Create a Monitoring and Evaluation Plan

Finalize Outcomes and Develop Measures
Define Measures and Create a Monitoring and Evaluation Plan

Purpose

Develop monitoring and evaluation plans for selected indicators to track and assess strategy implementation, outcomes, and impacts to nature and people, inform adaptive management, influence key actors, mitigate risk, and contribute to the evidence base.

Products

List of indicators to be monitored
List of measures to be generated
Monitoring and evaluation plans

Any conservation strategy needs to be able to demonstrate progress towards its ultimate goal if stakeholders, teams, and donors are to stay engaged and committed. In the case of strategies that seek to catalyze systemic change, that progress can be subtle, even incremental at times. Monitoring and evaluation can help you test for hypothesized changes, identify capacity gaps and learning needs, and make adaptive management decisions as your strategy progresses.

Show All

Define monitoring and measures needs.

Start by defining and prioritizing the information you will need. Your needs might include:

Reporting impact of your project or program to your organization. How is your organization tracking and summarizing impact across projects and programs? How do partners and others report impact? Is there a need for reporting in a common “currency” for area impacted, types of impacts, degrees of impacts?

Reporting to funders or other philanthropic uses. What, if any, are the funders’ requirements? Is your project designed to be used as a ‘proof of concept’ to solicit additional funding? If so, what proof will likely be required? What are the requirements for reporting progress on activities and results such as tool development, analytical reports, etc.?

Influencing key actors, including building the evidence base for conservation. Is there a group whose behavior you are hoping to change? What type of evidence do they require? The best way to assess this is generally to directly ask members of this group. Additionally, assessing how information has led to changes in behavior in the past can provide an understanding of the required quality and quantity of evidence. Note that one group of key actors is the conservation community. Evidence that a strategy works is important to influence broader adoption of the strategy by the conservation community.

Mitigating risk of negative impacts and other reputational risks. Are there potential, credible negative impacts to nature and people? Are there public promises for which proof of delivering on these promises is required?

Mitigating legal risk (if applicable). What legal guarantees, if any, are you responsible for? What evidence would be necessary to defend your organization in court

External Resources

Conservation Measures Partnership

The Conservation Measures Partnership (CMP) is a partnership of conservation organizations that seek better ways to design, manage, and measure the impacts of their conservation actions.

Read More

Measures Demystified

Learn more about measures by taking the Measures Demystified training on ConservationTraining.org. TNC staff can learn more about measures by following the Measures Community on CONNECT.

Read More

Ten steps to a results-based monitoring and evaluation system

The World Bank has provided this useful resource on designing M&E plans:Ten steps to a results-based monitoring and evaluation system a handbook for development practitioners.

Read More
Define and prioritize information needs for adaptive management.

Define milestones, specific key points in the results chain, and other information needs that are critical to making management decisions. Be explicit with what information is needed for these points. For each information need, consider whether indicators are needed. Note that activities and outputs are generally more commonly measured and reported to donors and in management reviews, but intermediate results and impacts are the results that are necessary to measure to indicate conservation results.

Processes or activities describe project actions, such as engaging in meetings, working with partners, conducting lobbying activities, etc. Indicators for these track activities and participation. They are generally qualitative in describing status, such as whether an activity is completed, ongoing and going well, has some issues, has major issues, or has not been started.

Outputs describe the major products that are completed by the conservation activity. These may be reports, or tools that were developed. Indicators for these are qualitative and generally related to the completion and delivery of a product. Progress in completing outputs can be generated similarly to process or activity indicators.

Intermediate results describe what we intend to accomplish that is a prerequisite for achieving conservation goals or outcomes. Intermediate results can be defined for several major steps in a sequence within a strategy. Intermediate results may relate to changes to or establishment of policy, governance, sustainable finance, partnership development, a social behavioral change, or implementation of management activities. Intermediate results are often referred to as “leading indicators” since their completion suggests that impacts will occur in the future.

Impacts describe what changes to people and nature are ultimately being achieved as a result of the conservation strategy. Impacts are related to goals; specifically, our goals are to achieve a certain level of impact. Impacts can be described in terms of the scope of an impact (how many hectares and/or kilometers are protected, restored, improved, how many people benefited, etc.) and/or the degree of impact (increase in population size, changes in species diversity, changes in water quality, changes in income, life expectancy, etc.). Impacts are often referred to as “lagging indicators” since they can take time to be realized and/or monitored.

Tools You Can Use

Guidance for evaluating systems change

Evaluating Systems Change - A planning guide (Mathematica 2010) provides guidance on planning effective evaluations of system change interventions. Incorporating systems theory and dynamics into evaluation planning can improve an evaluation’s design by capturing system conditions, dynamics, and points of influence that affect the operation and impact of a system change intervention.

Guidance for incorporating human well-being into strategy design and implementation

Strengthening the Social Impacts of Sustainable Landscapes Programs is a practitioner's guide to strengthening and monitoring human well-being outcomes. It focuses on developing strategies for sustainable landscapes -- for people whose livelihoods depend on surrounding land and waters.

Guidance for urban conservation strategies

The Field Guide to Conservation in Cities in North America has a whole section on steps to understanding, monitoring, evaluation and learning. It's set in the context of urban conservation, but its tips and case studies have lessons for everyone.
Identify indicators and measures.
For each of the information needs identified above, identify an indicator.

Example Indicators
Pro Tips
- Adopting or adapting existing tools and indicators can save time and money.
  
  Consult the literature, specialists, or colleagues for survey templates or previous tools that have been employed in past projects. Be sure to check if survey templates or tools are context appropriate and will collect relevant information.
- Considering novel indicators and monitoring approaches can save money.
  
  For instance, household surveys may be common when collecting socioeconomic data, but they can be expensive and require specialized knowledge. A less expensive option may be to use mobile phone-based survey methods, focus groups and key informant interviews, and participatory rural appraisal methods.
External Resources

What do we want to know? Selecting Indicators.

Indicators specify what needs to be measured. To learn more about selecting indicators, see Mayoux, L. (2002). What do we want to know? Selecting Indicators.

Read More

For all indicators, define the following:
Pro Tips
- If an indicator or measure doesn’t fill an information need, don’t measure it.
  
  Consider the full range of information needs that you have identified, and make sure that the indicators and measures you select are focused on priorities for fulfilling those needs.
Be explicit about how the indicator will be used.

As stated above, the purpose of measuring each indicator should be specified, including the intended audience and use. In particular, it is useful to specify the status or quantitative values of the indicator that would signify adequate progress in activities, completing outputs, achieving intermediate results, leverage through generating influence and progress toward goals, or unacceptable impacts, or lack of progress that would trigger adaptive management or other decisions. If this cannot be done, consider whether you have selected the wrong indicator, or whether it would actually be useful to measure that indicator.

External Resources

Outcome Mapping Learning Community

Outcome Mapping is an approach to planning, monitoring and evaluation that puts people at the centre, defines outcomes as changes in behaviour, and helps measure contribution to complex change processes.

Read More
Select indicators that inform multiple audience needs first.

Doing so can save resources. Table 4 demonstrates that in some cases one indicator can provide relevant information to many audiences (i.e., for Reduced conflict and Increased food security), and in other cases, different audiences may need different information (i.e., multiple indicators are required to measure Increased employment).

Table 4: Indicators by outcome and audience for the Northern Rangelands Trust
Consult with local stakeholders.

Indicators, especially those related to human well-being should be drafted in consultation with, and be validated by, local stakeholders to ensure they accurately reflect their priorities and experiences.
Evaluate the indicators.

If you have several indicators measuring the same concept, rate the indicators based on a set of criteria and only use the top rated indicators. The most well-known indicator criteria are SMART (for quantitative) and SPICED (for qualitative).

SMART indicator criteria are:

SPICED indicator criteria are:

Evaluate needs and available human and data resources.
The following are required for effective monitoring and evaluation. Staff must assess whether they have the expertise and capacity for each step.
Pro Tips
- Separately assess social science and ecological monitoring and evaluation capabilities.
  
  While some social science and ecological data collection and analyses can be combined, developing monitoring and evaluation plans require specialized knowledge. If critical audiences need statements with high certainty (i.e., the most rigorous monitoring and evaluation plans), significant research experience (i.e., PhD-level training) or certifications (i.e., analytical labs for certain monitoring needs) may be required.

Monitoring and evaluation plans should make use of data already being collected by others.
This may include data being collected by governments, academics, industries, indigenous organizations, community organizations and NGOs, and may include survey data, government statistics, model results, experimental results, and remotely sensed data. Consider partnerships with entities that have already invested in infrastructure to support monitoring and evaluation activities. However, it is important to assess the monitoring and evaluation design and information needs with data from existing efforts, and the probability that these efforts will or will not continue to provide needed information, before deciding to depend on others to supply data.
Pro Tips
- Ask academics, agencies, partners and research entities about data they are acquiring and about availability.
  
  Some data are restricted, but in many cases data sharing agreements can be put in place.
- Ask others involved in the strategy about existing sources of information.
  
  Partners and key actors often have familiarity with data sources useful to address information needs.
- Conduct web searches for data sources that will continue to provide updated information.
  
  Many government agencies regularly collect high quality data. Explore local, state, and federal government agencies for data needs.
Consider hiring contractors to fill capacity gaps.

Hiring contractors to design and implement data collection, evaluation, and/or measures reporting may be necessary. It can be difficult to evaluate the qualifications of contractors in areas where you lack expertise. Take special care in selecting the right contractor by reviewing the contractor’s previous work, talking directly with their prior clients, and soliciting external expertise to vet contractors if necessary. Second, you will need to be explicit about the information needs of critical audiences: what information is truly needed, when information is necessary, and how it should be reported. The contract should specify the content, timing and format of products delivered by the contractor. Finally, consider the importance of consistency in data collection and analysis. Is it possible to use the same contractor for the length of the project? And if not, will others be able to reliably duplicate their methods?
Consider community-based monitoring.

Engaging local communities in monitoring can engender interest and support for the project and empower communities through participation, as well as providing necessary information.

Develop monitoring and evaluation plans with appropriate research design.
Monitoring and evaluation plans should clearly articulate who, what, when, how, and why information should be collected, analyzed, and used.
Pro Tips
- Baseline data is important.
  
  It is difficult to detect change when you don’t know where you started. You can’t go back in time to collect baseline data, so make sure that all relevant baseline data is collected.
- Consider the communications needs of indigenous and local communities.
  
  All communication should consider the language, literacy levels, format, and other needs of these audiences.
- Extra precautions are required for any research involving human subjects.
  
  There is consensus in the research community about the ethics and appropriate protocols and standards for any research involving human subjects. Large universities have dedicated offices that provide this information for their academic community (e.g., Stanford University, University of Washington, University of Minnesota), and federally funded projects and grants that include research on human subjects require ethical approval. Conservancy staff should follow the Conservancy’s Standard Operating Procedure (SOP) for research involving human subjects , which applies to all research involving human subjects sponsored by the Conservancy, whether conducted by Conservancy employees, volunteers or agents (such as a contractor), or by any third party using the Conservancy’s facilities. The SOP includes specific guidance for researchers on research principles, permissible research and approval requirements, risk, informed consent, data security, and government funding.

Your monitoring and evaluation plan should specify the following elements:

Tools You Can Use

A guide for planning and evaluation of urban conservation projects

Focusing on the human impacts of conservation strategies must be central to how we define success in conservation work in cities. Whole Measures for Urban Conservation Framework
Be clear when data are needed to inform key decisions.

The monitoring and evaluation plan should clearly articulate the questions that should be answered through the monitoring and evaluation activities. Identify the decisions or actions you want to influence. This defines the realm of issues that you want to affect, for which you need to develop measures. Then identify the information that you will need to inform those decisions or actions and assess impact. This is what determines what you want to monitor and why.

Tools You Can Use

Tools for identifying data needs

This toolkit from Spark Policy Institute is designed to help you determine which questions to ask in order to get the data you need and how to use data to take action.
Implement the social safeguards.
Monitoring and evaluation plans should pass the 11 social safeguard questions outlined in Appendix C.
Pro Tips
- Refer to Appendix C
  
  Appendix C
Conduct peer review.

Monitoring and evaluation plans are strengthened when they are reviewed by peers. Peer reviews can be useful for identifying issues with monitoring and evaluation plans, whether there are duplicated efforts by others (e.g., government agencies collecting similar data), or whether monitoring and evaluation activities are sufficiently rigorous or realistic given the available resources.
Consider the timeframe of expected changes.

If, even under the best circumstances, conservation outcomes and impacts will take years to become apparent, don’t spend a lot of time and money monitoring to confirm the lack of change. Treat monitoring as a way to test for hypothesized changes. Design monitoring and evaluation plans that have realistic timeframes and geographic scales. Note that this mainly applies to ongoing monitoring activities that are required to detect subsequent change. Also note that the relevant timeframe for negative impacts may be different than the time frame relevant for desired change, and may require sooner or more frequent monitoring.
Use a qualified expert to help with the research design.

Developing an appropriate research design — taking into account requirements for temporal and spatial sampling, replication, controls, and counterfactuals — before initiating data collection is critical, for several reasons. These include avoiding wasting resources on unnecessary monitoring and accurately budgeting for required monitoring. Further, research design should inform monitoring decisions (e.g. frequency, sample size and other methodological decisions). Have a qualified scientist and/or statistician develop or review the research design to ensure that it provides the necessary rigor to match the desired level of certainty.
Select a research design commensurate with the level of certainty required for your audience.
It is often helpful to think about the certainty statements a program would like to make about its impact. If a program is interested in attributing outcomes to program activities (Certain statements), then a rigorous monitoring and evaluation plan should be implemented. This would likely require a greater level of investment than a plan that would attribute program impacts to anecdotal evidence (Cautious statements). Consider the types of statements the primary audience will need to be informed, or convinced, by a monitoring and evaluation plan. For instance, local stakeholders may be satisfied with anecdotes about a program’s impact, while donors may require causal statements about a program’s impact. Please see Appendix G for examples of research designs required to meet the three levels of certainty described below.

Table 5: Certainty Statements
Pro Tips
- Refer to Appendix G
  
  Appendix G

Minimum Standard Questions

Will the monitoring and evaluation plan ensure that essential information gaps are filled?
Has the existing strength of evidence informed a conservation strategy’s risk and leverage as it relates to the level of investment in monitoring and evaluation plans?
Are there indicators for both positive and potential negative outcomes for both people and nature, which were developed in partnership with potentially impacted stakeholders?
Does the monitoring and evaluation plan include collecting baseline data?
Has there been full exploration and consideration of secondary data from government, NGO, indigenous organizations, community organizations, and other firms or agencies for environmental and socioeconomic data to avoid duplicating data collection efforts and opportunities to fill gaps?
Have specific audiences, the intended use of information, and needed level of rigor been defined for each indicator?
Have plans been created that clearly define the design, collection, management, evaluation, and reporting procedures and responsibilities for data?
Do the monitoring and evaluation plans clearly articulate how the data will be analyzed, updated, and then shared to relevant audiences in culturally appropriate ways?
Do the monitoring and evaluation plans provide a realistic budget sufficient for monitoring over a long enough period of time to detect anticipated outcomes and impacts?
Will monitoring design and evaluation approaches (e.g. research design and statistical methods) and sampling be conducted by qualified professionals?

FAQS Show All

If we can collect information with no defined user or purpose right now, but that might useful in the future, should we collect it?

If we have a step in a theory of change or results chain that has not been identified as having a measures or monitoring need, is that a problem?

What extra precautions are required for research involving human subjects?

Define Measures and Create a Monitoring and Evaluation Plan

Define monitoring and measures needs.

Define and prioritize information needs for adaptive management.

Identify indicators and measures.

For all indicators, define the following:

Be explicit about how the indicator will be used.

Select indicators that inform multiple audience needs first.

Consult with local stakeholders.

Evaluate the indicators.

Evaluate needs and available human and data resources.

Monitoring and evaluation plans should make use of data already being collected by others.

Consider hiring contractors to fill capacity gaps.

Consider community-based monitoring.

Develop monitoring and evaluation plans with appropriate research design.

Your monitoring and evaluation plan should specify the following elements:

Be clear when data are needed to inform key decisions.

Implement the social safeguards.

Conduct peer review.

Consider the timeframe of expected changes.

Use a qualified expert to help with the research design.

Select a research design commensurate with the level of certainty required for your audience.

Submit a Comment