Appendix G — Additional Instruction for Modeling Design

Mon­i­tor­ing can be designed to make a vari­ety of com­par­isons, result­ing in dif­fer­ent ana­lyt­i­cal strengths. Cot­ting­ham et al. (2005) pro­vide eight alter­na­tive mon­i­tor­ing frame­works to eval­u­ate envi­ron­men­tal respons­es to envi­ron­men­tal flow man­age­ment, list­ed in increas­ing strength of infer­ence. These designs are applic­a­ble to a broad range of mon­i­tor­ing needs.

Cau­tious (low rigor/anecdotal)

1. Impact-only design. An activ­i­ty has already been imple­ment­ed (no before-activ­i­ty data exist) and there are no spa­tial con­trols or ref­er­ence sites, water­sheds or groups being used for com­par­i­son; mon­i­tor­ing is lim­it­ed to the site, water­shed, or group where activ­i­ties have been imple­ment­ed. These respons­es can be eval­u­at­ed against spe­cif­ic pre­dic­tions based on the con­cep­tu­al mod­el. Causal links between tem­po­ral changes in respons­es are dif­fi­cult if not impos­si­ble to deter­mine because the changes might have occurred with­out the activ­i­ties being implemented.
2. Ref­er­ence-Impact design. A mod­i­fi­ca­tion of (1) above, where there are no before-activ­i­ty imple­men­ta­tion data but the same para­me­ters are mon­i­tored through time in a ref­er­ence and impact site, water­shed, or group, which rep­re­sents the desired direc­tion of change for the impact. This design pro­vides slight­ly bet­ter evi­dence for a causal link between tem­po­ral change in response, because nat­ur­al changes through time can be mea­sured at ref­er­ence sites as well. It is also pos­si­ble to assess whether the trend of change at the impact loca­tion is towards the refer­ence con­di­tion, if that is desired.
3. Con­trol-Impact design. Sim­i­lar to (2) above except that com­par­i­son is with a con­trol site, water­shed, or group. This design pro­vides stronger infer­ence about causal­i­ty because com­par­i­son with the spa­tial con­trol reduces the like­li­hood that effects from activ­i­ties are sta­tis­ti­cal­ly con­found­ed with nat­ur­al change.

Some­what cer­tain (medi­um rigor/correlation)

4. Con­trol-Ref­er­ence-Impact design. This is a com­bi­na­tion of (2) and (3) above. Sta­tis­ti­cal analy­ses test for diver­gence in tem­po­ral trends between the impact and the con­trol, and for con­ver­gence in tem­po­ral trends between the impact and the ref­er­ence site, water­shed, or group. This design pro­vides causal strength sim­i­lar to (3), with the added advan­tage of assess­ing whether the trends are mov­ing toward ref­er­ence con­di­tions, if that is desired.
5. Before-After-Impact design. This is a stan­dard “impact analy­sis” design com­par­ing para­me­ter val­ues before ver­sus after activ­i­ties have been imple­ment­ed. The “before” data pro­vide base­line or tem­po­ral con­trol con­di­tions. Evi­dence for causal links is lim­it­ed by lack of spa­tial con­trols, there­fore it is unclear whether or not the change would have occurred inde­pen­dent­ly of the activ­i­ties being imple­ment­ed. This design is also dif­fi­cult to use if activ­i­ties are imple­ment­ed grad­u­al­ly, if there is a long lag-time for impacts to occur, or if the dif­fer­ence that occurs is not large.
6. Before-After, Ref­er­ence-Impact (BARI) design. This is sim­i­lar to (5) but with a spa­tial compo­nent ‑a ref­er­ence site, water­shed or group that pro­vides some mea­sure of whether nat­ur­al changes coin­cide with changes seen in the impact site. This design also allows assess­ment of whether the trend of a response is towards the ref­er­ence con­di­tion. The test of inter­est is whether any before-after dif­fer­ence at the impact loca­tion is the same as at the ref­er­ence loca­tion. The causal infer­ence asso­ci­at­ed with this design is lim­it­ed because the ref­er­ence and impact sites, water­sheds, or groups have dif­fer­ent con­di­tions pri­or to activ­i­ty imple­men­ta­tion. This makes it dif­fi­cult to rule out a response to oth­er fac­tors coin­cid­ing with the start of the imple­men­ta­tion of the activity.
7. Before-After Con­trol-Impact (BACI) design. Sim­i­lar to (6), but using a spa­tial con­trol instead of a ref­er­ence. This design pro­vides strong infer­ence about causal­i­ty because com­par­isons with spa­tial and tem­po­ral con­trols reduce the like­li­hood of con­found­ing effects with nat­ur­al spa­tial and tem­po­ral changes.

Cer­tain (high rigor/counterfactual analysis)

8. Before-After Con­trol-Ref­er­ence- Impact (BACRI) design. A com­bi­na­tion of (6) and (7) that pro­vides strong evi­dence for causal links between activ­i­ty and response, and also mea­sures whether the change is towards ref­er­ence con­di­tion, if that is desired.

Note that for all of these designs, inclu­sion of repli­cates improves the valid­i­ty of con­trol-impact con­trasts. If you want to learn more about impact eval­u­a­tion, con­sid­er this use­ful series of video lec­tures from the Inter­na­tion­al Ini­tia­tive for Impact Evaluation.