Modelling Framework

Summary of the Framework for Model Development

Here we summarize the Framework for Model Development based on what is practical given current knowledge. Further improvements are expected as additional research is carried out.

Conceptual model

Formally develop one or more conceptual models of the system using a comprehensive approach that considers all the relevant information and covers both the population dynamics and the observation processes.

Population dynamics model

Develop all reasonable alternative hypotheses about the population dynamics, including spatial and/or population structure, and observation processes, based on the conceptual model. Represent hypotheses as estimable parameters where possible. Use a method, such as a hierarchical approach, that avoids duplication. Use an existing general software package, if applicable, to implement the hypotheses based on the good practices of the field.

Process variation

Model all the population dynamics and observation processes that typically vary over time as random effects (or in a state-space framework), particularly those that have trends. Include demographic variation for small populations. Consider modeling temporal variation in other population dynamics and observation processes that is evident in the data or supported by other information.

Data weighting

Estimate the sampling variation outside the model, based on the sampling design, including dealing with pseudo replication and use this to define the variance parameter used in the likelihood function. Check to ensure that this sampling variance is consistent with the variance of the fit to the data. If not, reassess the model assumptions and as a last resort estimate a variance adjustment to account for the lack of fit.

Diagnostics

Apply all the appropriate diagnostics to each of the models and attempt to fix those that fail the diagnostics. Retain only the models that pass all the diagnostics that are considered essential for determining a good model. 

Apply self-testing to identify inherently biased parameters, correlation among estimated parameters, and inestimability and where present include models with a range of reasonable values of the parameter and include model “prior” weights based on reliable external information.

Model weighting

When the variance terms are well defined for all the data components, use standard model weighting approaches, otherwise use equal weighting.

Management advice from ensembles

Combine the results from all the accepted models taking into consideration both the estimation uncertainty and the model weights.

Discussion and Future Research

There has been substantial progress in fitting population dynamics models to data that has greatly improved management advice in a variety of situations from exploitation to conservation. One of the major developments has been integrated analysis where multiple diverse data sets are fit simultaneously in the same model (Maunder and Punt, 2013;Punt et al., 2013; Schaub and Abadi, 2011; Schaub and Kery, 2022). Historically, different data sets were analyzed in separate analyses, and the results compared or combined in a subsequent analysis. It is not surprising that the obvious approach of combination of multiple data sets has been independently developed in many fields (Schaub et al., 2024). However, issues such as model misspecification, unmodelled process variation, and data weighting make integrated analysis prob-lematic (Maunder and Piner, 2017). Therefore, there are several areas that need to be further improved including:

Conceptual models: 

Explicit and more comprehensive use of conceptual models to identify the alternative hypotheses (e.g., Minte-Vera et al., 2024). This would be facilitated by the development of a good practices guide for developing conceptual models.

Hierarchical hypotheses: 

Development of hierarchical (e.g., Maunder et al., 2020b) or other techniques to facilitate the implementation of hypotheses identified in the conceptual models and to avoid duplication and over weighting of specific hypotheses.

Diagnostics: 

Development of diagnostics that identify and fix model misspecification based on comprehensive simulation analysis (e.g.,

Kapur et al., submitted). This includes identification of unmodelled process variation.

Modeling process variation:

 Development of efficient methods to implement process variation in multiple processes and the estimation of their variances (e.g., Kristensen et al., 2016).

Representing uncertainty: Development of methods that efficiently and accurately combine parameter and model uncertainty (e.g., Mon-nahan, 2024).

The obvious benefits of integrating multiple diverse data sets into a single analysis has resulted in integrated analysis being developed independently in multiple fields (Schaub et al., 2024). In fact, Good Practices for modeling should not differ among fields, but often approaches differ due to different routes towards integrated analysis, dif-ferences in application characteristics, different types of data, and different management questions. It is also often difficult to compare approaches due to differences in terminology used among fields. Sharing ideas among fields and harmonizing methods and terminology would benefit everyone. This special issue makes great steps towards that by outlining good practices in a variety of fields.

An Open Science approach will also facilitate sharing of information among fields. This includes making all data and code available so that analyses can be repeated. In addition, Open Science facilitates peer re-view of analyses and more buy in from stakeholders. For example, the Inter-American Tropical Tuna Commission (IATTC) posts the data and model files for all their assessments on the web (e.g., https://www.iattc.org/StockAssessments/2024/BETWebsite/BET2024.htm). The R based package (R4ss; Taylor et al., 2021) available for viewing the results from the stock assessment computer program Stock Synthesis (Methot and Wetzel, 2013), which is used by the IATTC, produces an html file that is easily posted on the web. Since a general stock assessment program is used, stakeholders with the appropriate skills can easily take the files and modify them to investigate alternative assumptions (Hoyle et al., 2022). The data used in these assessments are substantially processed and summarized and therefore there are no issues with proprietary or other confidentiality. However, the detailed data used in the external analyses to create the summarized data may be confidential and it is not possible to provide this data to the public. In some cases, data agreements can be made to share the information with interested parties. 

The modelling framework I outline has many similarities to that outlined by Jakeman et al. (2024), but also some differences. In particular, the framework I present lacks the comprehensiveness included in the first phase of Jakeman et al.’s modeling process that scopes the problem. It also lacks the comprehensiveness of the final phase that ensures that the model is usable in the future and by a wider user group. These are valuable additions to my framework. On the other hand, the framework I outline focusses more on good practices in developing the model and diagnosing and improving the model that are part of Jakeman et al.’s phase 3, which is a common focus of other Good Modeling Practice structures (Jakeman et al., 2024).