Appendix G – Additional Instruction for Modeling Design

Monitoring can be designed to make a variety of comparisons, resulting in different analytical strengths. Cottingham et al. (2005) provide eight alternative monitoring frameworks to evaluate environmental responses to environmental flow management, listed in increasing strength of inference. These designs are applicable to a broad range of monitoring needs.

Cautious (low rigor/anecdotal)

  1. Impact-only design. An activity has already been implemented (no before-activity data exist) and there are no spatial controls or reference sites, watersheds or groups being used for comparison; monitoring is limited to the site, watershed, or group where activities have been implemented. These responses can be evaluated against specific predictions based on the conceptual model. Causal links between temporal changes in responses are difficult if not impossible to determine because the changes might have occurred without the activities being implemented.
  2. Reference-Impact design. A modification of (1) above, where there are no before-activity implementation data but the same parameters are monitored through time in a reference and impact site, watershed, or group, which represents the desired direction of change for the impact. This design provides slightly better evidence for a causal link between temporal change in response, because natural changes through time can be measured at reference sites as well. It is also possible to assess whether the trend of change at the impact location is towards the refer­ence condition, if that is desired.
  3. Control-Impact design. Similar to (2) above except that comparison is with a control site, watershed, or group. This design provides stronger inference about causality because comparison with the spatial control reduces the likelihood that effects from activities are statistically confounded with natural change.

Somewhat certain (medium rigor/correlation)

  1. Control-Reference-Impact design. This is a combination of (2) and (3) above. Statistical analyses test for divergence in temporal trends between the impact and the control, and for convergence in temporal trends between the impact and the reference site, watershed, or group. This design provides causal strength similar to (3), with the added advantage of assessing whether the trends are moving toward reference conditions, if that is desired.
  2. Before-After-Impact design. This is a standard "impact analysis" design comparing parameter values before versus after activities have been implemented. The "before" data provide baseline or temporal control conditions. Evidence for causal links is limited by lack of spatial controls, therefore it is unclear whether or not the change would have occurred independently of the activities being implemented. This design is also difficult to use if activities are implemented gradually, if there is a long lag-time for impacts to occur, or if the difference that occurs is not large.
  3. Before-After, Reference-Impact (BARI) design. This is similar to (5) but with a spatial compo­nent -a reference site, watershed or group that provides some measure of whether natural changes coincide with changes seen in the impact site. This design also allows assessment of whether the trend of a response is towards the reference condition. The test of interest is whether any before-after difference at the impact location is the same as at the reference location. The causal inference associated with this design is limited because the reference and impact sites, watersheds, or groups have different conditions prior to activity implementation. This makes it difficult to rule out a response to other factors coinciding with the start of the implementation of the activity.
  4. Before-After Control-Impact (BACI) design. Similar to (6), but using a spatial control instead of a reference. This design provides strong inference about causality because comparisons with spatial and temporal controls reduce the likelihood of confounding effects with natural spatial and temporal changes.

Certain (high rigor/counterfactual analysis)

  1. Before-After Control-Reference- Impact (BACRI) design. A combination of (6) and (7) that provides strong evidence for causal links between activity and response, and also measures whether the change is towards reference condition, if that is desired.

Note that for all of these designs, inclusion of replicates improves the validity of control-impact contrasts. If you want to learn more about impact evaluation, consider this useful series of video lectures from the International Initiative for Impact Evaluation.

Submit a Comment

Your email address will not be published.