Model validation:
A search for the Magic Grove or the Magic Model

STAND DENSITY MANAGEMENT:  Planning and Implementation Conference
Edmonton, November 6 - 7, 1997

Section 2 - Growth and Yield Responses and Monitoring

James W. Goudie, Stand Modelling Group
Forest Productivity and Decision Support Section
Research Branch, British Columbia Ministry of Forests

Back to References

Abstract - While growth and yield models cannot be proven valid, models must be evaluated or tested against available data to identify potential biases. This paper reports on the results of testing the Tree and Stand Simulator (TASS), a spatially explicit model used in the silvicultural and forest-level planning in British Columbia, against a portion of the plots and installations addressing various thinning strategies that have been analysed to date. The system generally performs very well but may moderately overestimate the response to thinning in some cases.

Introduction

In an old Persian folk tale called "The Magic Grove", a young man is given a quantity of gold and sent to the city by his father to purchase seed for trees that would beautify the family farm. Along the way he happens upon a camel caravan carrying cages of beautiful exotic birds destined for a culinary demise at the table of the evil emperor Kahn. The young man decides to use the gold to rescue the birds and set them free. That night, the magic birds in turn, collect twigs from the forest and plant them in the holes originally prepared for the seed. Magically, they flush and overnight grow into a grove of trees that not only beautifies the farm and provides home and cover for the birds (enhancing biodiversity), but also helps to protect the farm from a band of marauding Kahn soldiers. For young people listening to this entertaining folk tale, the important moral model that kindness is rewarded by kindness is validated.

Rather than folk tales or anecdotes, computer models must be tested against data, experience, and sometimes other models. The model testing process should not be a search for the magic grove to justify a model. Neither should responsible forest management be a search for the magic model to justify activities. The Tree and Stand Simulator (TASS) (Mitchell 1975) is used in stand and forest-level planning efforts in British Columbia and requires a thorough test against thinned stands since it is frequently applied to pre-commercial and commercial thinning regimes.

Literature Review

Model testing.

Testing forest growth model behaviour against particular datasets has had a lengthy history in the forestry literature. "Verification", "validation", "testing", and "checking" are terms commonly associated with such activity. Unfortunately, most of these terms are value laden and imply a truth (Oreskes et al. 1994). Like statistical models, computer models can never be proven valid or true (Curtis 1982, Goulding 1979, Vanclay and Skovsgaard 1997). Popper (1963) points out that such testing may in fact be redundant since we know in advance that models are false (cited by Caswell 1976). Goulding (1979) suggests that the goal for validation of a growth model should simply be to increase confidence about the growth of stands.

Vanclay and Skovsgaard (1997) propose using the term model "evaluation" because it is less loaded with value. They suggest the following five categories should be checked:

• biological and theoretical logic - the structure should be parsimonious, biologically realistic, consistent with existing theories of forest growth and predict sensible responses to management activities;
• statistical properties in relation to the data - the nature of error terms and the properties of the model parameters should be known ;
• error characterisation - evaluate the accuracy, residual patterns, confidence intervals and contribution of individual model components to the total error;
• statistical tests - test the bias and precision of the model and components, goodness-of-fit of predicted distributions, patterns and distribution of residuals, and correlations over time;
• sensitivity analyses - determine how model components influence predictions, the sensitivity of the model to its inputs, and how errors propagate through the model.

This paper concentrates on preliminary results from tests of the Tree and Stand Simulator (TASS) conducted under the statistical test category.

The model.

The Tree and Stand Simulator, TASS, is a distance dependent (i.e. spatially explicit) individual tree model initiated in the early 1960s. Mitchell (1965) developed the initial structure while studying white spruce (Picea glauca [Moench] Voss) as a PhD student at Yale. Development continued as emphasis shifted to Douglas-fir (Pseudotsuga menziesii [Mirb] Franco) (Mitchell 1975), lodgepole pine (Pinus contorta ssp latifolia Engl.) (Goudie 1980) and other species native to British Columbia. The emphasis has always been on quantifying the biology and morphology of tree growth, including crown structure and and inter-tree competition. However, stand-level yields remain its most important output.

In TASS, simulated trees can be established and controlled in virtually any realistic manner. Thinning, pruning and fertilisation are management activities usually projected. Simulated crowns expand, interact and compete for growing space. Figure 1 represents two live trees of differing vigour competing for space and a third dead tree that was overtopped earlier in the hypothetical simulation. Tree crowns are represented by simple concentric shells that represent foliage of differing ages. IN simple terms, the crown with the highest surface occupies grid locations under contention, however, the algorithm that governs competition for growing space is the most complex component of the model.

Figure 1. Graphical representation of the tree growth and competition in TASS. Two live trees (solid crown profiles) and one overtopped dead tree (dashed crown profile) are displayed.

Most of the relationships in TASS are derived from detailed dissections of individual trees (boles, branches and foliage). Untreated permanent sample plot data are used to generate mortality algorithms and ensure that yields conform to known data. Figure 2 illustrates the process for a small portion of the plots in natural-origin coastal Douglas-fir. TASS projections with varying initial densities are compared the calibration data for number of trees (A) and total standing volume (B). Note that the independent variable in the graphs (and many others in this paper) is top height which reduces the variation across site. This method has been used in the European literature to display even-aged stands for many years since it has been long known that yield for a given species and management regime is primarily a function of height (Assmann 1970).

A B

Figure 2. TASS projections (dashed) of number of stems/ha (A) and total volume (B) at six initial densities compared to a subset of the untreated, natural-stand permanent plot calibration data (solid) for coastal Douglas-fir.

A total of 11 989 permanent plots (comprised of 43 799 measurements) established for B.C. species growing in North America, Europe and New Zealand have been consistently summarised and classified by species and treatments including control, thinned, fertilised, and thinned and fertilised. Approximately 3200 coastal and 6200 interior plots from untreated natural stands or plantations were used to calibrate yields for various species. Pure Douglas-fir (885 plots) and western hemlock (Tsuga heterophylla [Raf.] Sarg. ) (1115 plots) on the coast, and lodgepole pine (2160 plots) and spruce (456 plots) in the interior dominate the calibration data.

Once the scale and shape of the natural runs conform to untreated natural stands, runs designed to mimic plantation espacement trials are compared to data. Figure 3 compares TASS with one source of espacement trials in coastal Douglas-fir growing on very good site at the University of British Columbia Research Forest near Haney, B.C. Mortality is underestimated beyond about 25m for the wider spacings (A) for this trial. The corresponding overestimate of total volume by TASS shown in Figure 2b illustrates the importance of mortality rates when comparing models to data, models to models, or data to data. These example plots represent only four of the 139 permanent plots addressing plantation espacement available for calibration and testing the Douglas-fir version of the model. Thinning data are rarely used in the calibration process and are thus independent, however, many of the control plots in the experiments contribute to the calibration process.

TASS has been used to generate managed stand yield tables in British Columbia for over a decade (e.g. Mitchell and Cameron, 1985). A digital subset of TASS yield tables addressing initial density (natural and planted) and precommercial spacing are now accessed by the Table Interpolation Program for Stand Yield (TIPSY) (Mitchell et. al 1992). Foresters developing pre-harvest silvicultural prescriptions and crop plans use TIPSY to evaluate a number of common regimes. Commercial thinning will be likely added in the near future. A sub-program (the TIPSY Economist, Stone et al., 1996) provides economic information. Batch versions of TIPSY are used to generate yield tables for forest-level planning. Currently TASS is calibrated for even-aged stands but ongoing work is expanding to more complex forests with multiple species, crown layers and age classes.

A B

Figure 3. TASS projections of number of stems/ha (A) and total volume (B) at four initial plantation densities compared to one espacement trial.

TASS is linked to a bucking routine, a sawmill simulator (SAWSIM) and a financial analysis system (FAN$Y) that allows the simulation of products and their value. This collection of models, known as SYLVER (Mitchell and Polsson 1993), is used for biological and economic analysis for a variety of regimes not currently covered in TIPSY. TASS has also been linked to a disease model (Mitchell and Bloomberg 1986), an insect attack simulator (Alfaro et al. 1996) and a light interception model (TRAYCI, Brunner, In review).

The use of components of SYLVER to investigate the effects of density management is increasing. Stone (1993) and Stone (1996) evaluated the biological and economic effects of commercial thinning in Douglas-fir and lodgepole pine, respectively. Until now, TASS has not been adequately tested against thinned stands to determine if such projections are reasonable.

The Thinning Validation Data

The availability of evaluation data is constrained by the history of research projects in any particular region. The earliest research trials in B.C. were established in the 1920s and 30s and consisted of either individual growth and yield plots or thinning installations designed after the European model of frequent, light thinnings. The Schenstrom plots established in 1929 on Vancouver Island (Warrack 1979) are the most famous. Expenditures and activities ebbed during the 1940s until Canada's involvement in World War II ended. The prosperous post-war years of the 1950s resulted in an explosion of research activity in B.C. At least 19 studies involving thinning trials were established, and research in artificial regeneration methods also dramatically increased (genetics, nursery, seedling survival, and plantation espacement trials).

The relatively few studies initiated in the 1960s were primarily related to genetics and tree improvement. Only three thinning trials established in the 1960s have been identified. The 1970s brought increased genetics research as well as new research on fertilisation, physiology and ecological classification. Ecological research, remote sensing, wildlife-forest interactions dominated the 1980s and interest in partial cutting and thinning began to renew. Environmental activism and other pressures have recently lead to at least 16 new studies being established in the 1990s involving partial cutting, selective logging, new forestry, gap creation and other thinning activities .

Thus, the available testing data is complicated by societal factors (wars, recessions, environmental movements), political environment (funding priorities, political philosophy), scientific developments (statistical designs, conventional wisdom), personnel developments (retirements, mobility, personal interest, publication demands) and popular methods of professional forest management. Acceptable "fits" to the available long-term data does not ensure an unbiased projection of current or future activities.

Methods

The data

The permanent sample plot data on file were surveyed for the following characteristics:

• primarily a single species
• at least one thinning treatment
• availability of unthinned controls
• approximately 15 years of measurements

Table 1 displays the available data currently on file that met the criteria. We have tested TASS against 23 of the 30 sources but this paper displays the results of just the six most likely to be of primary interest to this conference: Schenstrom plots, three Alberta experiments, experimental project (EP) number 385 in southeastern B.C., and a wide-ranging fertilizer-thinning experiment on the coast.

Plots and installations with many years of establishment are very valuable and informative. We include the famous Schenstrom plots (Warrack 1979) at Cowichan Lake Research Station on Vancouver Island because they are the longest-term trials we have available (15 measurements over 66 years). The installation consists of a control plot and four large, treated plots that have had up to seven thinnings since they were established (two levels of crown thinning and two levels of low thinning).

Three trials from Alberta (Gregg Burn, Teepee Pole Creek and Mackay ) are located nearest to the site of the conference (see Johnstone 1981a, 1982 and 1981b, respectively, for details). The Gregg Burn and Teepee Pole Creek experiments are identically-designed pre-commercial thinning experiments in very dense fire-origin lodgepole pine stands but thinned at different times (ages seven and 25, respectively). Each trial has five residual densities (494-7907 trees/ha), two replications per site and three sites. Leave trees were selected as the largest tree within a 45-cm distance to 10X10 grid locations. Pseudo-controls were established in 1996 for comparison but not used here.

The Mackay installation consists of a control plus five plots thinned at age 22. They replicated the treatments in each of three blocks on one site. An additional wide spacing (746/ha) was established in a separate, large single plot.

Commercial thinning is becoming very popular in western Canada. One of the few commercial thinning trials for interior species is EP 385. It was, before it burned down, located in southeastern British Columbia in a natural lodgepole pine stand and consisted of five plots thinned at age 53 (residual densities from 1121 to 2843 stems/ha) plus control.

EP 703 from coastal B.C. (Darling and Omule 1989, Omule 1990, 1991, Stone 1994) is of particular interest since it has the largest number of available plots and the widest geographic range of any source. It was originally intended to be a 3X3 factoral experiment investigating three levels of fertiliser (0, 20 and 400 kgN/ha) and three levels of thinning (0, 20 and 35% basal area removal). Changes in design occurred over the five years of establishment such that some later installations added two levels of fertiliser (600 and 900 kgN/ha) and a heavier thinning (50% basal area removal).

Source¹

Species²

No. Plots

First year est.

Mean age at est.

Mean number of years measured

Regen. method

Treatment³

5

1929

19

66

15

1951

50

35

49

1958

3

22

6

1952

53

26

6

1952

53

26

20

1953

56

35

17

1953

13

36

16

1956

17

27

58

1966

21

28

156

1970

34

20

68

1970

42

18

27

1970

25

26

27

1969

22

26

46

1970

25

18

30

1964

7

32

30

1967

25

29

19

1954

22

32

9

1967

13

25

6

1969

8

23

3

1966

18

26

10

1972

26

24

3

1919

9

54

3

1934

23

50

4

1929

27

54

12

1978

10

14

12

1978

13

14

12

1978

20

14

15

1972

17

19

24

1975

12

16

15

1975

14

17

723

25

24

A total of 940 plots were established in 85 installations. Unfortunately, none of the installations contained all treatments due to site and stand limitations. Only two sites had a complete 4X4 factoral and the original 3X3 factoral was established in just eight locations. In fact, 16 did not receive a control treatment. We selected installations with at least 70% of either western hemlock or Douglas-fir by volume prior to thinning and only selected those installations containing control plots to ensure the ability to test response to thinning. Our sample consisted of 224 unfertilized plots from 37 installations.

The Simulations

TASS does not project tree lists and must begin with bare ground due to the large amount of information retained for each tree that is unavailable in permanent plot data. Input variables for the TASS runs were:

• average pre- and post- treatment density
• plot size (the sum of all replications)
• site index
• thinning method.

We matched pre-treatment density data (or control plot data if pre-thinning measurements were unavailable) by iterating initial conditions until pre-treatment density was judged to be close enough to the plot statistics (normally, within 5%). The default method of thinning in TASS, which attempts to mimic a forester walking through the stand and selecting the largest diameter tree within a particular distance, was used unless unique methods were known in a particular experiment. For example, the Gregg Burn and Teepee Pole Creek trials required a special thinning algorithm because of their unique tree selection methodology.

Results and Discussion

This paper primarily focuses on the performance of TASS in terms of total standing volume relative to dominant height growth. An implicit assumption in this methodology is that dominant height growth, defined as top height, is unaffected by density. This assumption has been tested in the forestry literature for decades and density effects are usually non-significant (e.g. Evert 1971). The available permanent plot data we have generally supports this assumption for densities less than about 15 000 per ha, however, a few exceptions do exist. The well-known Wind River espacement plots in southern Washington State (Reukema 1979) and the Shawnigan Lake fertilizer-thinning trials on southern Vancouver Island (McWilliams and Therien 1996), both located on relatively low-site Douglas-fir sites, show significant reductions in dominant height at much lower densities than 15 000/ha.

Schenstrom plots

Figure 4A shows the trends of total volume vs. top height for the control and four thinned plots for both TASS and the plot data. The TASS projections are generally below the levels of both treated and untreated plots. Note that some of the between-plot variance is a result of site variation. Interpolating results to common top heights attempts to remove such site variation, especially important in older non-replicated trials such as the Schenstrom plots. The cross-section at the last common top height of 47.3 indicates very close correspondence between TASS and the trials (Figure 4B). It also supports the long-held contention in the European literature that total yield (standing+thinning volumes) are not increased by thinning (Assman, 1970).

The average response (thinned-control) at each measurement for standing live volume is very similar (Figure 5A). However, TASS over-predicts standing+thinning volume by about 40-50 cu.m/ha (Figure 5B) because the post-thinning growth of leave trees may be too great, resulting in volumes that are slightly high.

A B

Figure 4. Comparison of Schenstrom Plot data and TASS. Standing volume against
top height (A) and distribution of volume at the last common top height of 47.3m (B).

A B

Figure 5. Mean response of Schenstrom thinned plots and TASS projections
for standing live volume (A) and standing plus all previous thinnings (B).

Alberta experiments.

Gregg Burn results. The mean standing volume against top height for the Gregg Burn trials is displayed in Figure 6A. Figure 6B shows the treatment differences interpolated to the last common top height (11.7m). The densest treatment (7907/ha) is underestimated and the next widest (3951/ha) is slightly overestimated. The other densities are very close.

A B

Figure 6. Comparison of Gregg Burn plot data and TASS. Standing volume against
top height (A) and distribution of volume at the last common top height of 11.7m (B).

Teepee Pole Creek. Similar presentations for the Teepee Pole Creek data are shown in Figures 7A and 7B. TASS underestimates total standing volume of the dense plots and overestimates the widest treatments. Examinations of diameter distributions (not shown) indicate that the dense TASS runs have too many small trees (DBH7.5cm) after thinning and this difference is maintained throughout the duration of the plot measurements.

Differences between the plot data and projections in merchantable volume (12.5cm DBH+) are considerably less because the extra volume in the 10 cm class indicated by the plot data is ignored (Figure 7C). The overestimate of widely-spaced plots is due to both reduced mortality predicted by TASS and greater diameter growth of simulated widely-grown trees. The merchantable volume of the wide spacings is also overestimated.

Mackay plots. Mean values of total volume versus top height are displayed for the Mackay plots in Figure 8A. The results at the final common top height of 17.6m are shown in Figure 8B. The standing+thinning volume of the densest regimes are projected well but the widest regime is overestimated. When adjusted for the differences in the control plots, the standing+thinning yields for TASS are between -8.2 and +6.9 percent of that shown by the data for the four densest regimes (4070, 2691, 2663->1440 (i.e. two thinnings), and 1494 per ha) but 26.0 % high for the widest treatment. The standing only yields are +7.3 to +14.3 high for the four dense regimes and +27.8 percent high for the 764/ha.

A B

C

Figure 7. Comparison of Teepee Pole Creek plot data and TASS. Standing
volume against top height (A) and distribution of total volume (B) and
merchantable volume (C) at the last common top height of 13.7m.

A B

Figure 8. Volume-top height relationship (A) and component volume
distribution at the final common top height of 17.6m (B) for the Mackay plot data.

EP 385.

The comparison for EP 385 is shown in Figure 9A (standing volume over top height) and Figure 9B (standing, thinned and mortality volumes at the last common top height of 21.9m). The thinned plot standing volume and standing+thinning volumes were an average of 8.0 percent low compared to the data at 21.9m of top height and the control plot standing volume was 3.5 percent high.

A B

Figure 9. Volume-top height relationship (A) and volume distribution
at the final common top height of 21.9m (B) for the EP 385 plot data.

EP 703.

Figures 10 (Douglas-fir) and 11 (western hemlock) show the mean response (treated-control) by 2m height growth class of the three levels of thinning (20%, 35%, 50% basal area removal) for both the data and TASS projections. The independent variable is top height growth since establishment so that different sites can be compared.

Figure 10. Mean total volume response (thinned-control) for
simulated (TASS) and actual (EP 703) coastal Douglas fir
with 20 (a), 35 (b) and 50 (c) percent basal area removed. The
independent variable is control plot dominant height growth.
The number of installations for each height growth class is also shown.

Figure 11. Mean total volume response (thinned-control) for simulated
(TASS) and actual (EP 703) coastal western hemlock with
20 (a), 35 (b) and 50 (c) percent basal area removed. The independent
variable is control plot dominant height growth. The number of
installations for each height growth class is also shown.

The height growth of the control plot was used since covariance tests showed no significant treatment effects on top height development (not shown). Differences were statistically non-significant for all height growth classes and both species, however, some trends are apparent.

The response predicted by TASS for thinned Douglas-fir is slightly high for most height growth classes. The mean response of the 50 percent removal thinning treatment is predicted very accurately but this comprises only five installations. The response predicted by the western hemlock version of the model is consistently high early (less than 6m of top height growth) but tended to be low thereafter.

Conclusions

The work reported here will continue. We plan to evaluate residuals, diameter distributions, merchantable volume, mortality and other factors in the future. However, one tentative conclusion can be drawn based on data analysed to date. TASS tends to slightly over-predict thinning response relative to controls - perhaps because we do not include many factors that may differentially affect thinned plots (e.g. disease, pests, windthrow, morphological changes, branchiness).

TASS, and probably other models as well, perform best when tested against well-designed experiments. Researchers and foresters should be planning and installing thinning experiments now that are well replicated, have large enough plots, and cover a broad range of stand and treatment conditions. Future modellers and foresters will benefit from such data, and we can avoid searches for both magic groves and magic models.

Literature cited

Alfaro, R.I., G. Brown, K.J. Mitchell, K.R. Polsson and R.N. Macdonald. 1996. SWAT: A decision support system for spruce weevil management. pp. 31-44. In: T.L. Shore and D.L. MacLean [Eds.]. Decision support systems for forest pest management. FRDA Report No. 260.

Arnott, J. T. and D. Beddows. 1981. Levels-of-growing-stock cooperative study in Douglas-fir. Report No. 6: Sayward Forest, Shawnigaan Lake. p. 52. In Env. Can. Can. For Serv. Inf. Report BC-X-223, Victoria, B.C., Pac. For. Res. Centre.

Assmann, E. 1970. The Principles of Forest Yield Study. Pergamon Press, New York.

Brunner, A. In review. A light model for spatially explicit forest stand models. 42p.

Caswell, H. 1976. The validation problem. p. 313-325. In Systems Analysis and Simulation in Ecology, Vol. 4, Patten, B. C. (eds.). Academic Press, New York. 601 p.

Curtis, R. O. 1982. Some notes on validation of stand simualtion models. p. 6. In Meeting of Western Forest Mensurationsists, Nov. 29, 1982, Portland, OR,

Darling, L. M. and S. A. Y. Omule. 1989. Extensive studies of fertilizing and thinning coastal Douglas-fir and western hemlock: an establishment report. p. 17. B.C. Min. For., Res. Br., FRDA Rep. No. 054, Victoria, BC.

Evert, F. 1971. Spacing studies - a review. p. 95 p. Can. For. Serv. Inf. Rep. FMR-X-37,

Goudie, J. W. 1980. Yield tables for managed stands of lodgepole pine in northern Idaho and southeastern British Columbia. University of Idaho, Moscow, Idaho. 111 p.

Goulding, C. J. 1979. Validation of growth models used in forest management. N. Z. J. For. 24(1):108-124.

Johnstone, W. D. 1981. Effects of spacing 7-year-old lodgepole pine in west-central Alberta. Can. For. Serv. Inf. Rep NOR-X-236, Edmonton, Alberta.

Johnstone, W. D. 1982. Juvenile spacing of 25-year-old lodgepole pine in western Alberta. p. 19. Envir. Can., Can. For. Serv., North. For. Res. Cent., Inf. Rep NOR-X-244, Edmonton, AB.

Johnstone, W. D. 1981. Precommercial thinnig speeds growth and development of lodgepole pine: 25-year results. p. 30. Can. For. Serv. Inf. Rep NOR-X-237, Edmonton, Alberta.

McWilliams, E. R. G. and G. Therien. 1996. Fertilization and thinning effects on a Douglas-fir ecosystem at Shawnigan Lake: 24-year growth response. (Revised 1997) Economic & Regional Develop. Agree., B.C. Min. For. and Lands, Res. Br., FRDA Rep. No. 269. 44 p.

Mitchell, K. J. 1975. Dynamics and simulated yield of Douglas-fir. p. 39. In For. Sci. Monogr. 17,

Mitchell, K. J. 1969. Simulation of the growth of even-aged stands of white spruce. p. 48. In Yale Univ. School of Forestry Bull. No 75, Yale University, New Haven, CN.

Mitchell, K. J. and W. J. Bloomberg. 1987. Expanding concepts of growth and yield modelling to disease impacts and forest products. p. 3-11. In Second Can. For. Serv. Modelling Workshop, Richardson, J. (eds.). Can. For. Serv., Hull, Que.

Mitchell, K. J. and I. R. Cameron. 1985. Managed stand yield tables for coastal Douglas-fir: initial density and precommercial thinning. p. 69. B.C. Min. For., Res. Branch, Land Manage. Rep No. 31, Victoria, B.C.

Mitchell, K. J., S. E. Grout, R. N. Macdonald and C. A. Watmough. 1992. User's guide for TIPSY: A Table Interpolation for Stand Yields. p. 78. (Draft Manuscript), B.C. Min. For., Victoria, B.C.

Mitchell, K. J. and K. R. Polsson. 1993. SYLVER: A decision-support system for silviculture. p. 34-44. In Les modeles de croissanceforestiere et leurs utilisations = Forest growth models and their uses, November 18-19, 1993, Quebec City, PQ, Canada, (ed) Ung, C. (eds.). Natural Resources Canada, Canadian Forest Service, Sainte-Foy, Quebec.

Omule, S. A. Y. 1987. Comparative height growth to age 28 for seven species in the CWHD subzone. Economic & Regional Develop. Agree., B.C. Min. For. and Lands, Res. Br., FRDA Rep. 005, Victoria, BC.

Omule, S. A. Y. 1988. Growth and yield 32 years after commercially thinning 56-year-old western hemlock. Economic & Regional Develop. Agree., B.C. Min. For. and Lands, Res. Br., FRDA Rep. 029, Victoria, B.C.

Omule, S. A. Y. 1990. Net basal area response 9 years after fertilizing thinned and unthinned Douglas-fir. Economic & Regional Develop. Agree., B.C. Min. For. and Lands, Res. Br., FRDA Rep. 097, Victoria, B.C. Canada.

Omule, S. A. Y. 1985. Response of coastal douglas-fir to precommercial thinning on a medium site in British Columbia. p. 22. B.C. Min. For. Res. Note No. 100, Victoria, B.C.

Omule, S. A. Y. 1984. Results from a correlated curve trend experiment on spacing and thinning of coastal Douglas-fir. p. 22. B.C. Min. For., Res. Br., Res. Note No. 93, Victoria, B.C.

Omule, S. A. Y. and G. M. Britton. 1991. Basal area response nine years after fertilizing and thinning western hemlock. p. 33. B.C. Min. For., Res. Br., FRDA Rep. No. 137,

Oreskes, N., K. Shrader-Frechette and K. Belitz. 1994. Verification, validation, and confirmation of numerical models in the earth sciences. Science 263:641-645.

Reukema, D. L. 1979. Fifty-year development of Douglas-fir planted at various spacings. p. 21. In USDA Forest Serv. Res. Pap. PNW-253, PNW For. and Range Exp. Stn., Portland, OR.

Steele, R. W. 1955. Growth after precommercial thinning in two stands of Douglas-fir. USDA For. Serv. Res. Note PNW-117:6 p.

Stone, J. N. 1994. Extensive studies of fertilizing and thinning coastal Dougals-fir and western hemlock: An installation report. Economic & Regional Develop. Agree., B.C. Min. For. and Lands, Res. Br., FRDA Rep. 227, Victoria, B.C. Canada.

Stone, M. 1996. Commercial thinning of lodgepole pine: An economic analysis. p. 153. In FRDA II Working Paper No. WP-6-017, Can.-British Columbia Partnership Agreement on For. Res. Devel., Victoria, B.C.

Stone, M. 1993. An economic evaluation of commercial thinning Douglas-fir in the coastal region of British Columbia. p. 146. In FRDA II Working Paper No. WP-6-002, Can.-British Columbia Partnership Agreement on For. Res. Devel., Victoria, B.C.

Stone, M., S. Grout and C. Watmough. 1996. The TIPSY economist: An economic analysis module for WinTIPSY. p. 60. User's manual, Victoria, B.C.

Vanclay, J. K. and J. P. Skovsgaard. 1997. Evaluating forest growth models. Ecol. Modelling 98:1-12.

Warrack, G. C. 1979. Successive thinnings in a natural stand of Douglas-fir over a fifty year period. p. 11. B.C. Min. For. Res. Note No. 87, Victoria, B.C.