Types of Evaluation

The most widely used typology of monitoring, including the United States Forest Service and Bureau of Land Management, is that of Noss and Cooperrider (1994). They divided monitoring into four basic types: baseline, implementation, effectiveness, and validation. Monitoring activities have expanded rapidly since they wrote, and we now have enough experience to make some important distinctions.

Compliance Evaluation/ Monitoring

Compliance monitoring simply assesses the degree to which regulations or standards are met. Noss and Cooperrider (1994) note that compliance and implementation monitoring are sometimes treated as synonyms. That is true but unfortunate, and greater distinction has recently been made between the two. In both compliance and implementation monitoring no mechanisms are explicitly invoked and the only comparison is to a specific externally derived target in the regulation or standard.

Compliance monitoring can be a useful regulatory device. However, it assumes the regulation or standard is appropriate, does not evaluate effectiveness, and contributes nothing to the expansion of knowledge. An example would be documenting whether wildlife tree patches meet existing standards, without also documenting whether wildlife tree patches were attaining their expected outcomes.

Compliance monitoring fails to address two essential management questions: Are we going in the right direction? How do we change if the direction is wrong?

In the Wildlife Tree Evaluation study, compliance monitoring was used to determine the timber supply impacts of wildlife tree retention practices. It was also combined with other types of monitoring to assess how effectively current wildlife tree retention practices met the ecological and administrative guiding principles specified in the Provincial Wildlife Tree Policy and Management Recommendations.

Implementation Evaluation/ Monitoring

Implementation monitoring is undertaken simply to record rates of progress towards a specific goal, including rates or adoption of new practices, and whether they were implemented as planned. Corporate or agency targets usually are involved. Industry frequently uses implementation monitoring. Examples include monitoring operations to see whether 80% of cutblocks use variable retention within the expected time period, or whether 10% retention blocks actually retain 10% of the trees.

Resource agencies might wish to use implementation monitoring in evaluating progress on action plans for which the scope can be defined clearly, such as living rivers or protected areas strategies. Again no mechanisms are explicitly involved and comparisons are to an arbitrary, external target. The terms implementation monitoring and performance monitoring are sometimes used interchangeably because both reflect progress towards a goal. It is better to use implementation because performance monitoring is also used as a synonym for effectiveness monitoring.

Implementation monitoring was used in the Wildlife Tree Evaluation study to evaluate how current wildlife tree guidelines were implemented and how these practices are changing over time.

Effectiveness Evaluation/ Monitoring

Effectiveness monitoring is used to determine whether the plans or practices implemented are actually meeting the anticipated outcomes; that is, are they effective? By definition, effectiveness monitoring must sample actual practices, with the goal of obtaining the most precise estimate of operational effectiveness.

Effectiveness monitoring is a necessary part of operational feedback in adaptive management, but it provides limited opportunities to learn new information to refine future activities or standards. Some comparison or target that defines effectiveness must be invoked. Provided that target is well founded, effectiveness monitoring answers the managers' question: Are we going in the right direction? Such monitoring indicates that the practices are effective in meeting the anticipated outcomes, thus serve to validate the practices. Thus some may term the approach validation monitoring.

Noss and Cooperrider (1994) note baseline monitoring as a separate approach to establish baseline reference conditions that can be used to quantify a change that might be due to management activities. Often, areas included in the baseline set are deemed to be affected in a minimal way by management (Overton et al. 1995). In such cases, the underlying assumption is that differences in measurements taken in reference areas and managed areas can be attributed to management activities.

Little is gained by elevating this form of monitoring to a separate conceptual approach, because it usually implies little more than the location of sampling when making comparisons for effectiveness monitoring. Given that comparisons are being made, the same sampling protocol must be used for the baseline as for the managed area. Baseline simply reflects one potential form of comparison, not a separate form of monitoring.

The Wildlife Tree Evaluation study also involved effectiveness monitoring because the ecological guiding principles were broad and set objectives for the retention that would result in the best habitat being maintained. As long as the assumptions underlying the principles are sound, then assessing these principles is effectiveness monitoring. However, to truly evaluate effectiveness, this study would need to evaluate whether the habitat attributes the policy requested be retained are in fact beneficial to biodiversity.

Improvement/ Validation Evaluation/ Monitoring

Monitoring to improve, also called monitoring to learn, subsumes the validation monitoring of Noss and Cooperrider (1994). They noted that validation monitoring seeks to verify the assumed causal linkages between cause and effect. That is, validation monitoring is intended to validate the basic assumptions under which the management direction was developed. It is true that better understanding of underlying assumptions permits improvement but it is not the only way.

Monitoring to improve can also be based on a wider range of comparisons. Monitoring to improve may simply attempt to sample the widest range of available practices, including rare but informative extremes or combinations. Creating learning opportunities through experimental treatments beyond the normal operational range is an integral part of monitoring for learning how to improve. While experimental approaches may serve to validate causal linkages, they can guide improvement even when linkages remain poorly understood.

Both effectiveness monitoring and monitoring to improve are part of adaptive management, but because they imply somewhat different monitoring designs, a conscious decision needs to be made on the effort allocated to each in designing the monitoring program. Explanatory mechanisms are most likely to be invoked when monitoring to improve, but extending the range of comparisons also can attain improvements.

The two approaches to improvement also have different implications to statistical design. Given that validation monitoring is only one route to improvement, but monitoring to improve is greatly different from effectiveness monitoring, greater clarity is attained by including validation monitoring within the term monitoring to improve. Note also that monitoring to improve is unlikely to be attained unless the monitoring program as been developed to the point that the managers' last question is addressed: How do we change if the direction is wrong?

The Wildlife Tree Evaluation project was not broad enough to include improvement monitoring. If it were to include this type of monitoring, the study would have had to assess practices outside the range of those recommended by the Guidebook to identify alternatives that maintain stand level biodiversity as well or better than wildlife tree practices as presently prescribed. Improvement monitoring may also include studies that increase our understanding of why and how retention of stand structures impacts biodiversity, such as a study that assesses how large snags retention affect certain key species.

Combining Approaches

Noss and Cooperrider (1994) noted that effectiveness monitoring and validation monitoring (one aspect of monitoring to improve) reach their full potential only when used in combination. That is, knowing that the overall goals are or are not being met (effectiveness monitoring) is rather pointless without some assurance that the observed effect is due to the management activities (validation monitoring). Similarly, while it is comforting to believe that one's understanding of the world is correct (validation monitoring), that is of little utility in determining whether overall goals are being met (effectiveness monitoring). The two types of monitoring are highly complementary when fully integrated.

There is a common misconception that effectiveness monitoring is somehow easier or less rigorous than monitoring to improve, whether it is validation monitoring or not. Noss and Cooperrider (1994) noted that this misconception has led to suggestions that effectiveness monitoring can be readily handled by government agencies, while validation monitoring is the role of research institutions.

The suggestions reveal two fallacies:

The message here is that cooperative approaches involving the private sector, research institutions and government will prove most effective in developing monitoring approaches.

Summary

The key points about conventional monitoring approaches can be summarized as follows: