Trusting the Numbers: An Overview of the Testing Framework for the Fisheries Integrated Modeling System (FIMS)

class: center, middle, inverse, title-slide

.title[
# Trusting the Numbers: An <br> Overview of the Testing <br> Framework for the Fisheries <br> Integrated Modeling System <br> (FIMS)
]
.author[
### Bai Li <br><br> Contractor with ECS Federal LLC in support of <br> NOAA Fisheries Office of Science and Technology <br> Email: <a href="mailto:bai.li@noaa.gov" class="email">bai.li@noaa.gov</a>
]
.institute[
### FIMS CIE Review
]
.date[
### 2026/04/22
]

---

layout: true

.left-wide {
  width: 65%;
  float: left;
}

.right-narrow {
  width: 30%;
  float: right;
}
</style>

.footnote[U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service]

---
# Roadmap for Today

.pull-left[
- Why is rigorous testing vital for scientific software?
- How we catch errors before they reach your desk?
- How FIMS stacks up against the established models?
- How can we improve the testing framework?
]

.pull-right[
![](https://renewep.com/wp-content/uploads/2023/09/road-map-1030x722.jpg)
]

---
# What is FIMS? (Quick Refresher)

- A flexible suite of software tools to support sustainable fishery management
- FIMS modules are written in C++ and linked to R using Rcpp*<sup>1</sup>*
- Template Model Builder*<sup>2</sup>* serves as the engine for statistical inference in FIMS

.footnote[
[1] .hyperlink-style[[Rcpp:](https://cran.r-project.org/web/packages/Rcpp/index.html)] An R package that provides R functions as well as C++ classes which offer a seamless<br> integration of R and C++.<br>
[2] .hyperlink-style[[Template Model Builder:](https://kaskr.github.io/adcomp/Introduction.html)] An R package for fitting statistical latent variable models to data.
]

???
This R/C++ design shapes how we organize and automate testing.

---
# Why Testing Matters

.pull-left[
<img src="static/Hook_and_Kelly_scientific_software_errors.png" width="80%" height="80%" />

(Hook and Kelly, 2009)
]

.pull-right[
- Scientific software output can accumulate many sources of error
- Testing ensures the code implementation matches the scientific theory
- Reproducibility guarantees consistent results across systems
- Comprehensive test suites allow for refactoring without unintentionally altering research outcomes
- Well-tested code builds trust and strengthens peer review
]

.footnote[
[1] .hyperlink-style[[Hook and Kelly, 2009:](https://se4science.org/workshops/secse09/Presentations/09_Hook.pdf)] Testing for (code) trustworthiness in scientific software.<br>
]

???
In science, a small typo in code can lead to a big error in a quota recommendation. Testing is our 'Peer Review' for code—ensuring our implementation actually matches our equations.
---
# Overview of FIMS Testing

- We adopt key best practices from software engineering

.footnote[
Figure source: .hyperlink-style[[GeeksforGeeks](https://www.geeksforgeeks.org/types-software-testing/)]
]

---
# Testing Layers in FIMS

<div class="DiagrammeR html-widget html-fill-item" id="htmlwidget-b04d44c9937fbc556863" style="width:100%;height:504px;"></div>
<script type="application/json" data-for="htmlwidget-b04d44c9937fbc556863">{"x":{"diagram":"\n   graph TB\n   a[<b>Types of FIMS Testing<\/b>]-->A[<b>Manual Testing<\/b>]\n   a-->B[<b>Automation Testing<\/b>]\n   A-->C[<b>Functional Testing<\/b>]\n   A-->D[<b>Non-Functional Testing<\/b>]\n   C-->E[<b>Unit Testing<\/b>]\n   C-->F[<b>Integration Testing<\/b>]\n   C-->G[<b>System Testing<\/b>]\n   D-->H[<b>Performance Testing<\/b>]\n   D-->I[<b>Usability Testing<\/b>]\n   D-->J[<b>Compatibility Testing<\/b>]\n   style a fill:#FFFFFF, stroke:#003B5C\n   style A fill:#C6E6F0\n   style B fill:#C6E6F0\n   style C fill:#90DFE3\n   style D fill:#90DFE3\n   style E fill:#5BC0DE\n   style F fill:#5BC0DE\n   style G fill:#5BC0DE\n   style H fill:#5BC0DE\n   style I fill:#5BC0DE\n   style J fill:#5BC0DE\n"},"evals":[],"jsHooks":[]}</script>

---
# Types of Tests Implemented
<table class="table table-striped table-hover" style="font-size: 12px; width: auto !important; margin-left: auto; margin-right: auto;">
 <thead>
<tr>
<th style="border-bottom:hidden;padding-bottom:0; padding-left:3px;padding-right:3px;text-align: center; " colspan="2"><div style="border-bottom: 1px solid #ddd; padding-bottom: 5px; ">Tests</div></th>
<th style="border-bottom:hidden;padding-bottom:0; padding-left:3px;padding-right:3px;text-align: center; " colspan="3"><div style="border-bottom: 1px solid #ddd; padding-bottom: 5px; ">Testing Frameworks and Tools</div></th>
<th style="border-bottom:hidden;padding-bottom:0; padding-left:3px;padding-right:3px;text-align: center; " colspan="4"><div style="border-bottom: 1px solid #ddd; padding-bottom: 5px; ">Roles and Responsibilities</div></th>
</tr>
  <tr>
   <th style="text-align:left;"> Test Type </th>
   <th style="text-align:left;"> Description </th>
   <th style="text-align:left;"> GoogleTest (C++) </th>
   <th style="text-align:left;"> testthat (R) </th>
   <th style="text-align:left;"> Case Studies / Research Project </th>
   <th style="text-align:left;"> Developers </th>
   <th style="text-align:left;"> Testers </th>
   <th style="text-align:left;"> Users </th>
   <th style="text-align:left;"> GitHub Actions </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Unit Testing </td>
   <td style="text-align:left;"> Test individual components </td>
   <td style="text-align:left;"> ✔️ </td>
   <td style="text-align:left;"> ✔️ </td>
   <td style="text-align:left;"> ❌ </td>
   <td style="text-align:left;"> ✔️ </td>
   <td style="text-align:left;"> ❌ </td>
   <td style="text-align:left;"> ❌ </td>
   <td style="text-align:left;"> ✔️ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Integration Testing </td>
   <td style="text-align:left;"> Test components working together </td>
   <td style="text-align:left;"> ✔️ </td>
   <td style="text-align:left;"> ✔️ </td>
   <td style="text-align:left;"> ❌ </td>
   <td style="text-align:left;"> ✔️ </td>
   <td style="text-align:left;"> ✔️ </td>
   <td style="text-align:left;"> ❌ </td>
   <td style="text-align:left;"> ✔️ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> System Testing </td>
   <td style="text-align:left;"> Evaluate the overall functionality of a complete FIMS model </td>
   <td style="text-align:left;"> ❌ </td>
   <td style="text-align:left;"> ✔️ </td>
   <td style="text-align:left;"> ✔️ </td>
   <td style="text-align:left;"> ❌ </td>
   <td style="text-align:left;"> ✔️ </td>
   <td style="text-align:left;"> ✔️ </td>
   <td style="text-align:left;"> ✔️ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Performance Testing </td>
   <td style="text-align:left;"> Ensure the system performs properly under its expected workload </td>
   <td style="text-align:left;"> ❌ </td>
   <td style="text-align:left;"> ✔️ </td>
   <td style="text-align:left;"> ✔️ </td>
   <td style="text-align:left;"> ❌ </td>
   <td style="text-align:left;"> ✔️ </td>
   <td style="text-align:left;"> ✔️ </td>
   <td style="text-align:left;"> ✔️ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Usability Testing </td>
   <td style="text-align:left;"> Assess user interface effectiveness </td>
   <td style="text-align:left;"> ❌ </td>
   <td style="text-align:left;"> ❌ </td>
   <td style="text-align:left;"> ✔️ </td>
   <td style="text-align:left;"> ❌ </td>
   <td style="text-align:left;"> ✔️ </td>
   <td style="text-align:left;"> ✔️ </td>
   <td style="text-align:left;"> ✔️ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Compatibility Testing </td>
   <td style="text-align:left;"> Check FIMS compatibility across different operating systems </td>
   <td style="text-align:left;"> ✔️ </td>
   <td style="text-align:left;"> ✔️ </td>
   <td style="text-align:left;"> ❌ </td>
   <td style="text-align:left;"> ❌ </td>
   <td style="text-align:left;"> ❌ </td>
   <td style="text-align:left;"> ❌ </td>
   <td style="text-align:left;"> ✔️ </td>
  </tr>
</tbody>
</table>

---

# Unit Testing: Testing Individual Components

- Test each scientific component **in isolation** before combining them 
- Ensures each mathematical piece behaves exactly as expected
- **Example:** If a logistic selectivity module has an inflection point of 10 and a slope of 0.2, verify that selectivity at age 10 equals 0.5 as expected from the logistic function

.footnote[
Figure source: .hyperlink-style[[FIMS selectivity Rcpp tests](https://noaa-fims.github.io/FIMS/testdown/test-rcpp-selectivity.html)]
]

???
- Testing frameworks and tools
  - Use **GoogleTest** to test C++ code
  - Use **testthat** to test R code
- Roles and responsibilities
  - Module developers write tests and execute them locally
  - GitHub Actions (GHA) automatically runs all tests before each Pull Request (PR) as a safeguard 
- Why this matters:
  - Prevents small mathematical errors from propagating into full model results
  
---
# Integration Testing: Testing Components Working Together

- Verify that different model components **interact correctly**
- Ensures information flows properly between components
- **Example:** Create a selectivity module, assign it to a fleet module, and verify that the fleet correctly stores and uses the associated selectivity IDs without errors

.footnote[
Figure source: .hyperlink-style[[FIMS selectivity Rcpp tests](https://noaa-fims.github.io/FIMS/testdown/test-rcpp-fleet-interface.html)]
]

???
- Testing frameworks and tools
  - Use **GoogleTest** to test C++ code
  - Use **testthat** to test R code
- Roles and responsibilities
  - Developers or testers write tests and execute them locally
  - GHA automatically runs all tests before each PR as a safeguard 
- Why this matters:
  - Even correct components can produce wrong results if they interact incorrectly
---
# System Testing: Testing the Full Model Behavior

- Evaluate the **entire FIMS model** as a complete system
- Run full model scenarios and compare results to known expectations
- **Example:** Run a full FIMS model using simulated data and verify results match known "true" population trends

<table class="table table-striped table-hover" style="font-size: 12px; width: auto !important; margin-left: auto; margin-right: auto;">
 <thead>
  <tr>
   <th style="text-align:left;"> Test Type </th>
   <th style="text-align:left;"> Description </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Deterministic Test </td>
   <td style="text-align:left;"> Fix parameters at "true" values from the operating model. </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Estimation Test 1 </td>
   <td style="text-align:left;"> Estimate using age composition input only. </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Estimation Test 2 </td>
   <td style="text-align:left;"> Estimate using length composition input only. </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Estimation Test 3 </td>
   <td style="text-align:left;"> Estimate using both age and length composition input. </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Estimation Test 4 </td>
   <td style="text-align:left;"> Estimate with NAs in the input data. </td>
  </tr>
</tbody>
</table>

.footnote[
Figure source: .hyperlink-style[[FIMS tests](https://noaa-fims.github.io/FIMS/testdown/test-integration-caa-mle-wrappers.html)]
]

???
- Testing frameworks and tools
  - Use case studies to test FIMS in a separate repo
- Roles and responsibilities
  - Tester/users write tests and execute them locally
  - GHA automatically runs all test cases

---
# Performance Testing
.pull-left[
- Ensure the system performs properly under its expected workload
- Evaluate speed, load capability, and consistency of results
- **Example:** Run multiple FIMS models in parallel and confirm results match serial runs while maintaining fast execution 
]

.pull-right[
![](slides_20260422_BaiLi_files/figure-html/unnamed-chunk-10-1.png)
]

.footnote[
Figure source: .hyperlink-style[[FIMS parallel tests](https://github.com/NOAA-FIMS/FIMS/blob/main/tests/testthat/test-slow-parallel-caa-mle-wrappers.R)]
]

???
- Testing frameworks and tools
  - Use **testthat** to check if FIMS can be run in parallel
  - Use case studies to test FIMS in a separate repo
- Roles and responsibilities
  - Tester/users write tests and execute them locally
  - GHA automatically runs all test cases 
  
---
# Usability Testing

- Evaluate how easily end users can run and interpret FIMS models
- Identify workflow challenges and user pain points
- **Example:** A stock assessment scientist runs a case study and identifies bottlenecks or confusing steps
  
<iframe src="https://noaa-fims.github.io/case-studies/content/NEFSC-yellowtail.html" width="100%" height="400px" data-external="1"></iframe>

---
# Compatibility Testing

- Ensure FIMS runs on different operating systems
- Verify results are reproducible on Windows, macOS, and Linux
- **Example:** Run the same model across systems and confirm results are the same

.footnote[
Figure source: .hyperlink-style[[FIMS R Universe builds](https://noaa-fisheries-integrated-toolbox.r-universe.dev/builds)]
]

???
- Testing frameworks and tools
  - Run C++ and R tests on Windows, macOS, and Ubuntu
- Roles and responsibilities
  - GHA automatically runs all tests

---
# Automation Testing
- Use appropriate automation tools (e.g., GitHub Actions) to run tests for FIMS
- C++ and R tests are automatically executed on every commit
- Code coverage results are reported for every pull request

![](slides_20260422_BaiLi_files/figure-html/unnamed-chunk-13-1.png)

.footnote[
Figure source: .hyperlink-style[[FIMS GitHub Actions](https://github.com/NOAA-FIMS/FIMS/actions/workflows/run-googletest.yml)]
]

---
# Test Reporting
- Use {testdown} to generate human readable test reports
  - 132 tests with 542 checks 
  - 470 passing checks, 72 skipped checks, and 2 warnings

.footnote[
Figure source: .hyperlink-style[[FIMS testing report](https://noaa-fims.github.io/FIMS/testdown/global-results-for-package-fims.html)]
]

---
# Code Coverage
- Use Codecov to track and report code coverage
  - Current code coverage is 83%, meaning 83% of the codebase is exercised by tests
  - Coverage changes are automatically reported on each pull request to ensure coverage does not decrease significantly

![](static/code_coverage_20260415.png)

.footnote[
Figure source: .hyperlink-style[[FIMS code coverage](https://app.codecov.io/gh/NOAA-FIMS/FIMS?search=&trend=all%20time)]
]

---
# Statistical Validation
- Statistical validation ensures the science is correct
- We use an operating model with known truth and compare FIMS estimates to that truth

.metric-box {
  padding: 15px;
  border-radius: 10px;
  color: #003B5C;
  font-size: 0.9em;
}

.m1 { background-color: #E6F2FF; }
.m2 { background-color: #E6FFF2; }
.m3 { background-color: #FFF4E6; }
.m4 { background-color: #F3E6FF; }
</style>

<div class="metric-box m1">
<b>Metric 1: Magnitude Accuracy</b><br>
$$
\mathrm{median}\left(\left|\frac{\hat{y}-y}{y}\right|\right) \leq 0.2
$$
</div>

<div class="metric-box m2">
<b>Metric 2: Directional Bias</b><br>
$$
\left|\mathrm{mean}\left(\frac{\hat{y}-y}{y}\right)\right| \leq 0.1
$$
</div>

<div class="metric-box m3">
<b>Metric 3: Coefficient of Determination</b><br>
$$
\mathrm{cor}(\hat{y}, y)^2 \geq 0.9
$$
</div>

<div class="metric-box m4">
<b>Metric 4: Outlier Detection</b><br>
$$
\max\left(\left|\frac{\hat{y}-y}{\hat{\sigma}}\right|\right) \leq 4
$$
</div>

</div>

???
- metric 1: Validates the overall scale and magnitude of the outputs
- metric 2: Ensures the model does not systematically over- or under-estimate "true" values
- metric 3: Verifies that estimated trends move in tandem with expected trends
- metric 4: A deliberately lenient threshold to accommodate derived quantities across large matrices (n_years × n_ages)
Together, these provide complementary views of model performance

---
# Benchmarking FIMS Against Established Assessment Tools
<style>
.flow-grid {
  display: grid;
  grid-template-columns: 1fr 1fr 1fr;
  grid-gap: 15px;
  align-items: stretch;
}

.flow-box {
  padding: 15px;
  border-radius: 10px;
  color: #003B5C;
  font-size: 0.9em;
}

.om { background-color: #E6F2FF; grid-column: span 1;}
.em { background-color: #FFF4E6; grid-column: span 2;}
.analysis { background-color: #F3E6FF; grid-column: span 3; }
.arrow {
  text-align: center;
  font-size: 1.5em;
}
</style>

<div class="flow-box om">
<b>Operating Model (OM)</b><br>
- Age-structured (grouper-like)<br>
- 30-year simulation<br>  
- One fishing fleet<br>
- One survey fleet<br>

(Li et al., 2021)
</div>

<div class="flow-box em">
<b>Estimation Models (EMs)</b><br>
- Age Structured Assessment Program (ASAP)<br>
- Beaufort Assessment Model (BAM)<br>
- Stock Synthesis (SS3)<br>
- Woods Hole Assessment Model (WHAM)<br>
- FIMS <br><br>
- WHAM and FIMS:<br>
&nbsp;&nbsp;- Fixed-effect recruitment deviations<br>
&nbsp;&nbsp;- Random-effect recruitment deviations 
</div>

<div class="flow-box analysis">
<b>Analysis</b><br>
- 500 simulations generated<br>
- 100 converged runs retained<br>  
- Performance measure metrics: bias, accuracy, convergence rate 
</div>

</div>

.footnote[
[1] .hyperlink-style[[Li et al., 2021:](https://doi.org/10.7755/FB.119.2-3.5)] A comparison of 4 primary age-structured stock assessment models used in the United States.<br>
]
---
# Benchmarking FIMS Against Established Assessment Tools
**Time series of recruitment estimates from each estimation model**

---
# Benchmarking FIMS Against Established Assessment Tools
**Relative error in fishing mortality over time from each estimation model**
<img src="https://github.com/NOAA-FIMS/model-comparison-project/blob/main/C0/figure/re_boxplot_f.png?raw=true" width="65%" style="display: block; margin: auto;" />

---
# Benchmarking FIMS Against Established Assessment Tools
**Distribution of relative error in spawning biomass across all years and 100 simulation replicates for each estimation model**
<img src="https://github.com/NOAA-FIMS/model-comparison-project/blob/main/C0/figure/re_sb.png?raw=true" width="65%" style="display: block; margin: auto;" />

---
# Benchmarking FIMS Against Established Assessment Tools
** Mean of median relative error (%)**
<table class="table table-striped table-hover table-condensed" style="font-size: 11px; width: auto !important; margin-left: auto; margin-right: auto;">
 <thead>
  <tr>
   <th style="text-align:left;font-weight: bold;padding: 1px;"> case </th>
   <th style="text-align:left;font-weight: bold;padding: 1px;"> model </th>
   <th style="text-align:left;font-weight: bold;padding: 1px;"> metric </th>
   <th style="text-align:right;font-weight: bold;padding: 1px;"> mre </th>
   <th style="text-align:right;font-weight: bold;padding: 1px;"> mre_lower </th>
   <th style="text-align:right;font-weight: bold;padding: 1px;"> mre_upper </th>
   <th style="text-align:right;font-weight: bold;padding: 1px;"> sd_mre </th>
   <th style="text-align:right;font-weight: bold;padding: 1px;"> n_sims </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;color: black !important;padding: 1px;"> ASAP </td>
   <td style="text-align:left;padding: 1px;"> fishing_mortality </td>
   <td style="text-align:right;padding: 1px;"> 0.58 </td>
   <td style="text-align:right;padding: 1px;"> -0.38 </td>
   <td style="text-align:right;padding: 1px;"> 1.19 </td>
   <td style="text-align:right;padding: 1px;"> 3.13 </td>
   <td style="text-align:right;padding: 1px;"> 100 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;color: black !important;padding: 1px;"> BAM </td>
   <td style="text-align:left;padding: 1px;"> fishing_mortality </td>
   <td style="text-align:right;padding: 1px;"> 0.57 </td>
   <td style="text-align:right;padding: 1px;"> -0.38 </td>
   <td style="text-align:right;padding: 1px;"> 1.20 </td>
   <td style="text-align:right;padding: 1px;"> 3.14 </td>
   <td style="text-align:right;padding: 1px;"> 100 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;color: black !important;padding: 1px;"> SS3 </td>
   <td style="text-align:left;padding: 1px;"> fishing_mortality </td>
   <td style="text-align:right;padding: 1px;"> 0.44 </td>
   <td style="text-align:right;padding: 1px;"> -0.50 </td>
   <td style="text-align:right;padding: 1px;"> 1.05 </td>
   <td style="text-align:right;padding: 1px;"> 3.08 </td>
   <td style="text-align:right;padding: 1px;"> 100 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;color: black !important;padding: 1px;"> WHAM_fixed_effects </td>
   <td style="text-align:left;padding: 1px;"> fishing_mortality </td>
   <td style="text-align:right;padding: 1px;"> 0.58 </td>
   <td style="text-align:right;padding: 1px;"> -0.37 </td>
   <td style="text-align:right;padding: 1px;"> 1.23 </td>
   <td style="text-align:right;padding: 1px;"> 3.14 </td>
   <td style="text-align:right;padding: 1px;"> 100 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;color: black !important;padding: 1px;"> WHAM_random_effects </td>
   <td style="text-align:left;padding: 1px;"> fishing_mortality </td>
   <td style="text-align:right;padding: 1px;"> 0.41 </td>
   <td style="text-align:right;padding: 1px;"> -0.54 </td>
   <td style="text-align:right;padding: 1px;"> 1.09 </td>
   <td style="text-align:right;padding: 1px;"> 3.13 </td>
   <td style="text-align:right;padding: 1px;"> 100 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;color: black !important;padding: 1px;"> FIMS_fixed_effects </td>
   <td style="text-align:left;padding: 1px;"> fishing_mortality </td>
   <td style="text-align:right;padding: 1px;"> 0.62 </td>
   <td style="text-align:right;padding: 1px;"> -0.23 </td>
   <td style="text-align:right;padding: 1px;"> 1.30 </td>
   <td style="text-align:right;padding: 1px;"> 3.19 </td>
   <td style="text-align:right;padding: 1px;"> 100 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;color: black !important;padding: 1px;"> FIMS_random_effects </td>
   <td style="text-align:left;padding: 1px;"> fishing_mortality </td>
   <td style="text-align:right;padding: 1px;"> 0.45 </td>
   <td style="text-align:right;padding: 1px;"> -0.37 </td>
   <td style="text-align:right;padding: 1px;"> 1.11 </td>
   <td style="text-align:right;padding: 1px;"> 3.19 </td>
   <td style="text-align:right;padding: 1px;"> 100 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;color: black !important;padding: 1px;"> ASAP </td>
   <td style="text-align:left;padding: 1px;"> recruitment </td>
   <td style="text-align:right;padding: 1px;"> 0.12 </td>
   <td style="text-align:right;padding: 1px;"> -0.42 </td>
   <td style="text-align:right;padding: 1px;"> 0.35 </td>
   <td style="text-align:right;padding: 1px;"> 1.60 </td>
   <td style="text-align:right;padding: 1px;"> 100 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;color: black !important;padding: 1px;"> BAM </td>
   <td style="text-align:left;padding: 1px;"> recruitment </td>
   <td style="text-align:right;padding: 1px;"> 0.12 </td>
   <td style="text-align:right;padding: 1px;"> -0.41 </td>
   <td style="text-align:right;padding: 1px;"> 0.36 </td>
   <td style="text-align:right;padding: 1px;"> 1.60 </td>
   <td style="text-align:right;padding: 1px;"> 100 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;color: black !important;padding: 1px;"> SS3 </td>
   <td style="text-align:left;padding: 1px;"> recruitment </td>
   <td style="text-align:right;padding: 1px;"> 0.17 </td>
   <td style="text-align:right;padding: 1px;"> -0.26 </td>
   <td style="text-align:right;padding: 1px;"> 0.46 </td>
   <td style="text-align:right;padding: 1px;"> 1.57 </td>
   <td style="text-align:right;padding: 1px;"> 100 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;color: black !important;padding: 1px;"> WHAM_fixed_effects </td>
   <td style="text-align:left;padding: 1px;"> recruitment </td>
   <td style="text-align:right;padding: 1px;"> 0.12 </td>
   <td style="text-align:right;padding: 1px;"> -0.41 </td>
   <td style="text-align:right;padding: 1px;"> 0.38 </td>
   <td style="text-align:right;padding: 1px;"> 1.60 </td>
   <td style="text-align:right;padding: 1px;"> 100 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;color: black !important;padding: 1px;"> WHAM_random_effects </td>
   <td style="text-align:left;padding: 1px;"> recruitment </td>
   <td style="text-align:right;padding: 1px;"> 0.17 </td>
   <td style="text-align:right;padding: 1px;"> -0.34 </td>
   <td style="text-align:right;padding: 1px;"> 0.43 </td>
   <td style="text-align:right;padding: 1px;"> 1.60 </td>
   <td style="text-align:right;padding: 1px;"> 100 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;color: black !important;padding: 1px;"> FIMS_fixed_effects </td>
   <td style="text-align:left;padding: 1px;"> recruitment </td>
   <td style="text-align:right;padding: 1px;"> 0.14 </td>
   <td style="text-align:right;padding: 1px;"> -0.43 </td>
   <td style="text-align:right;padding: 1px;"> 0.34 </td>
   <td style="text-align:right;padding: 1px;"> 1.58 </td>
   <td style="text-align:right;padding: 1px;"> 100 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;color: black !important;padding: 1px;"> FIMS_random_effects </td>
   <td style="text-align:left;padding: 1px;"> recruitment </td>
   <td style="text-align:right;padding: 1px;"> 0.18 </td>
   <td style="text-align:right;padding: 1px;"> -0.37 </td>
   <td style="text-align:right;padding: 1px;"> 0.41 </td>
   <td style="text-align:right;padding: 1px;"> 1.58 </td>
   <td style="text-align:right;padding: 1px;"> 100 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;color: black !important;padding: 1px;"> ASAP </td>
   <td style="text-align:left;padding: 1px;"> spawning_biomass </td>
   <td style="text-align:right;padding: 1px;"> -0.18 </td>
   <td style="text-align:right;padding: 1px;"> -0.60 </td>
   <td style="text-align:right;padding: 1px;"> 0.42 </td>
   <td style="text-align:right;padding: 1px;"> 2.23 </td>
   <td style="text-align:right;padding: 1px;"> 100 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;color: black !important;padding: 1px;"> BAM </td>
   <td style="text-align:left;padding: 1px;"> spawning_biomass </td>
   <td style="text-align:right;padding: 1px;"> -0.18 </td>
   <td style="text-align:right;padding: 1px;"> -0.61 </td>
   <td style="text-align:right;padding: 1px;"> 0.41 </td>
   <td style="text-align:right;padding: 1px;"> 2.23 </td>
   <td style="text-align:right;padding: 1px;"> 100 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;color: black !important;padding: 1px;"> SS3 </td>
   <td style="text-align:left;padding: 1px;"> spawning_biomass </td>
   <td style="text-align:right;padding: 1px;"> -0.09 </td>
   <td style="text-align:right;padding: 1px;"> -0.65 </td>
   <td style="text-align:right;padding: 1px;"> 0.44 </td>
   <td style="text-align:right;padding: 1px;"> 2.22 </td>
   <td style="text-align:right;padding: 1px;"> 100 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;color: black !important;padding: 1px;"> WHAM_fixed_effects </td>
   <td style="text-align:left;padding: 1px;"> spawning_biomass </td>
   <td style="text-align:right;padding: 1px;"> -0.18 </td>
   <td style="text-align:right;padding: 1px;"> -0.60 </td>
   <td style="text-align:right;padding: 1px;"> 0.37 </td>
   <td style="text-align:right;padding: 1px;"> 2.23 </td>
   <td style="text-align:right;padding: 1px;"> 100 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;color: black !important;padding: 1px;"> WHAM_random_effects </td>
   <td style="text-align:left;padding: 1px;"> spawning_biomass </td>
   <td style="text-align:right;padding: 1px;"> -0.02 </td>
   <td style="text-align:right;padding: 1px;"> -0.47 </td>
   <td style="text-align:right;padding: 1px;"> 0.51 </td>
   <td style="text-align:right;padding: 1px;"> 2.24 </td>
   <td style="text-align:right;padding: 1px;"> 100 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;color: black !important;padding: 1px;"> FIMS_fixed_effects </td>
   <td style="text-align:left;padding: 1px;"> spawning_biomass </td>
   <td style="text-align:right;padding: 1px;"> -0.21 </td>
   <td style="text-align:right;padding: 1px;"> -0.93 </td>
   <td style="text-align:right;padding: 1px;"> 0.48 </td>
   <td style="text-align:right;padding: 1px;"> 2.30 </td>
   <td style="text-align:right;padding: 1px;"> 100 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;color: black !important;padding: 1px;"> FIMS_random_effects </td>
   <td style="text-align:left;padding: 1px;"> spawning_biomass </td>
   <td style="text-align:right;padding: 1px;"> -0.05 </td>
   <td style="text-align:right;padding: 1px;"> -0.79 </td>
   <td style="text-align:right;padding: 1px;"> 0.62 </td>
   <td style="text-align:right;padding: 1px;"> 2.31 </td>
   <td style="text-align:right;padding: 1px;"> 100 </td>
  </tr>
</tbody>
</table>

---
# Benchmarking FIMS Against Established Assessment Tools
**Median absolute relative error (%)**

<table class="table table-striped table-hover table-condensed" style="font-size: 11px; width: auto !important; margin-left: auto; margin-right: auto;">
 <thead>
  <tr>
   <th style="text-align:left;font-weight: bold;padding: 1px;"> case </th>
   <th style="text-align:left;font-weight: bold;padding: 1px;"> model </th>
   <th style="text-align:left;font-weight: bold;padding: 1px;"> metric </th>
   <th style="text-align:right;font-weight: bold;padding: 1px;"> mare </th>
   <th style="text-align:right;font-weight: bold;padding: 1px;"> mare_lower </th>
   <th style="text-align:right;font-weight: bold;padding: 1px;"> mare_upper </th>
   <th style="text-align:right;font-weight: bold;padding: 1px;"> mean_are </th>
   <th style="text-align:right;font-weight: bold;padding: 1px;"> sd_are </th>
   <th style="text-align:right;font-weight: bold;padding: 1px;"> n_obs </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;padding: 1px;"> ASAP </td>
   <td style="text-align:left;padding: 1px;"> fishing_mortality </td>
   <td style="text-align:right;padding: 1px;"> 2.67 </td>
   <td style="text-align:right;padding: 1px;"> 2.56 </td>
   <td style="text-align:right;padding: 1px;"> 2.79 </td>
   <td style="text-align:right;padding: 1px;"> 3.42 </td>
   <td style="text-align:right;padding: 1px;"> 3.17 </td>
   <td style="text-align:right;padding: 1px;"> 3000 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;padding: 1px;"> BAM </td>
   <td style="text-align:left;padding: 1px;"> fishing_mortality </td>
   <td style="text-align:right;padding: 1px;"> 2.67 </td>
   <td style="text-align:right;padding: 1px;"> 2.57 </td>
   <td style="text-align:right;padding: 1px;"> 2.80 </td>
   <td style="text-align:right;padding: 1px;"> 3.42 </td>
   <td style="text-align:right;padding: 1px;"> 3.17 </td>
   <td style="text-align:right;padding: 1px;"> 3000 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;padding: 1px;"> SS3 </td>
   <td style="text-align:left;padding: 1px;"> fishing_mortality </td>
   <td style="text-align:right;padding: 1px;"> 2.68 </td>
   <td style="text-align:right;padding: 1px;"> 2.56 </td>
   <td style="text-align:right;padding: 1px;"> 2.78 </td>
   <td style="text-align:right;padding: 1px;"> 3.37 </td>
   <td style="text-align:right;padding: 1px;"> 3.13 </td>
   <td style="text-align:right;padding: 1px;"> 3000 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;padding: 1px;"> WHAM_fixed_effects </td>
   <td style="text-align:left;padding: 1px;"> fishing_mortality </td>
   <td style="text-align:right;padding: 1px;"> 2.66 </td>
   <td style="text-align:right;padding: 1px;"> 2.55 </td>
   <td style="text-align:right;padding: 1px;"> 2.80 </td>
   <td style="text-align:right;padding: 1px;"> 3.42 </td>
   <td style="text-align:right;padding: 1px;"> 3.17 </td>
   <td style="text-align:right;padding: 1px;"> 3000 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;padding: 1px;"> WHAM_random_effects </td>
   <td style="text-align:left;padding: 1px;"> fishing_mortality </td>
   <td style="text-align:right;padding: 1px;"> 2.69 </td>
   <td style="text-align:right;padding: 1px;"> 2.54 </td>
   <td style="text-align:right;padding: 1px;"> 2.80 </td>
   <td style="text-align:right;padding: 1px;"> 3.40 </td>
   <td style="text-align:right;padding: 1px;"> 3.14 </td>
   <td style="text-align:right;padding: 1px;"> 3000 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;padding: 1px;"> FIMS_fixed_effects </td>
   <td style="text-align:left;padding: 1px;"> fishing_mortality </td>
   <td style="text-align:right;padding: 1px;"> 2.63 </td>
   <td style="text-align:right;padding: 1px;"> 2.51 </td>
   <td style="text-align:right;padding: 1px;"> 2.79 </td>
   <td style="text-align:right;padding: 1px;"> 3.37 </td>
   <td style="text-align:right;padding: 1px;"> 3.09 </td>
   <td style="text-align:right;padding: 1px;"> 3000 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;padding: 1px;"> FIMS_random_effects </td>
   <td style="text-align:left;padding: 1px;"> fishing_mortality </td>
   <td style="text-align:right;padding: 1px;"> 2.63 </td>
   <td style="text-align:right;padding: 1px;"> 2.49 </td>
   <td style="text-align:right;padding: 1px;"> 2.74 </td>
   <td style="text-align:right;padding: 1px;"> 3.35 </td>
   <td style="text-align:right;padding: 1px;"> 3.06 </td>
   <td style="text-align:right;padding: 1px;"> 3000 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;padding: 1px;"> ASAP </td>
   <td style="text-align:left;padding: 1px;"> recruitment </td>
   <td style="text-align:right;padding: 1px;"> 3.87 </td>
   <td style="text-align:right;padding: 1px;"> 3.70 </td>
   <td style="text-align:right;padding: 1px;"> 4.03 </td>
   <td style="text-align:right;padding: 1px;"> 4.98 </td>
   <td style="text-align:right;padding: 1px;"> 4.43 </td>
   <td style="text-align:right;padding: 1px;"> 3000 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;padding: 1px;"> BAM </td>
   <td style="text-align:left;padding: 1px;"> recruitment </td>
   <td style="text-align:right;padding: 1px;"> 3.87 </td>
   <td style="text-align:right;padding: 1px;"> 3.71 </td>
   <td style="text-align:right;padding: 1px;"> 4.03 </td>
   <td style="text-align:right;padding: 1px;"> 4.98 </td>
   <td style="text-align:right;padding: 1px;"> 4.43 </td>
   <td style="text-align:right;padding: 1px;"> 3000 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;padding: 1px;"> SS3 </td>
   <td style="text-align:left;padding: 1px;"> recruitment </td>
   <td style="text-align:right;padding: 1px;"> 3.83 </td>
   <td style="text-align:right;padding: 1px;"> 3.67 </td>
   <td style="text-align:right;padding: 1px;"> 4.03 </td>
   <td style="text-align:right;padding: 1px;"> 4.98 </td>
   <td style="text-align:right;padding: 1px;"> 4.44 </td>
   <td style="text-align:right;padding: 1px;"> 3000 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;padding: 1px;"> WHAM_fixed_effects </td>
   <td style="text-align:left;padding: 1px;"> recruitment </td>
   <td style="text-align:right;padding: 1px;"> 3.86 </td>
   <td style="text-align:right;padding: 1px;"> 3.71 </td>
   <td style="text-align:right;padding: 1px;"> 4.02 </td>
   <td style="text-align:right;padding: 1px;"> 4.98 </td>
   <td style="text-align:right;padding: 1px;"> 4.44 </td>
   <td style="text-align:right;padding: 1px;"> 3000 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;padding: 1px;"> WHAM_random_effects </td>
   <td style="text-align:left;padding: 1px;"> recruitment </td>
   <td style="text-align:right;padding: 1px;"> 3.85 </td>
   <td style="text-align:right;padding: 1px;"> 3.72 </td>
   <td style="text-align:right;padding: 1px;"> 4.01 </td>
   <td style="text-align:right;padding: 1px;"> 4.99 </td>
   <td style="text-align:right;padding: 1px;"> 4.45 </td>
   <td style="text-align:right;padding: 1px;"> 3000 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;padding: 1px;"> FIMS_fixed_effects </td>
   <td style="text-align:left;padding: 1px;"> recruitment </td>
   <td style="text-align:right;padding: 1px;"> 3.83 </td>
   <td style="text-align:right;padding: 1px;"> 3.69 </td>
   <td style="text-align:right;padding: 1px;"> 4.04 </td>
   <td style="text-align:right;padding: 1px;"> 5.03 </td>
   <td style="text-align:right;padding: 1px;"> 4.54 </td>
   <td style="text-align:right;padding: 1px;"> 3000 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;padding: 1px;"> FIMS_random_effects </td>
   <td style="text-align:left;padding: 1px;"> recruitment </td>
   <td style="text-align:right;padding: 1px;"> 3.82 </td>
   <td style="text-align:right;padding: 1px;"> 3.69 </td>
   <td style="text-align:right;padding: 1px;"> 4.03 </td>
   <td style="text-align:right;padding: 1px;"> 5.03 </td>
   <td style="text-align:right;padding: 1px;"> 4.55 </td>
   <td style="text-align:right;padding: 1px;"> 3000 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;padding: 1px;"> ASAP </td>
   <td style="text-align:left;padding: 1px;"> spawning_biomass </td>
   <td style="text-align:right;padding: 1px;"> 2.15 </td>
   <td style="text-align:right;padding: 1px;"> 2.06 </td>
   <td style="text-align:right;padding: 1px;"> 2.25 </td>
   <td style="text-align:right;padding: 1px;"> 2.68 </td>
   <td style="text-align:right;padding: 1px;"> 2.52 </td>
   <td style="text-align:right;padding: 1px;"> 3000 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;padding: 1px;"> BAM </td>
   <td style="text-align:left;padding: 1px;"> spawning_biomass </td>
   <td style="text-align:right;padding: 1px;"> 2.15 </td>
   <td style="text-align:right;padding: 1px;"> 2.06 </td>
   <td style="text-align:right;padding: 1px;"> 2.26 </td>
   <td style="text-align:right;padding: 1px;"> 2.69 </td>
   <td style="text-align:right;padding: 1px;"> 2.52 </td>
   <td style="text-align:right;padding: 1px;"> 3000 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;padding: 1px;"> SS3 </td>
   <td style="text-align:left;padding: 1px;"> spawning_biomass </td>
   <td style="text-align:right;padding: 1px;"> 2.12 </td>
   <td style="text-align:right;padding: 1px;"> 2.02 </td>
   <td style="text-align:right;padding: 1px;"> 2.23 </td>
   <td style="text-align:right;padding: 1px;"> 2.66 </td>
   <td style="text-align:right;padding: 1px;"> 2.51 </td>
   <td style="text-align:right;padding: 1px;"> 3000 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;padding: 1px;"> WHAM_fixed_effects </td>
   <td style="text-align:left;padding: 1px;"> spawning_biomass </td>
   <td style="text-align:right;padding: 1px;"> 2.15 </td>
   <td style="text-align:right;padding: 1px;"> 2.06 </td>
   <td style="text-align:right;padding: 1px;"> 2.26 </td>
   <td style="text-align:right;padding: 1px;"> 2.69 </td>
   <td style="text-align:right;padding: 1px;"> 2.52 </td>
   <td style="text-align:right;padding: 1px;"> 3000 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;padding: 1px;"> WHAM_random_effects </td>
   <td style="text-align:left;padding: 1px;"> spawning_biomass </td>
   <td style="text-align:right;padding: 1px;"> 2.14 </td>
   <td style="text-align:right;padding: 1px;"> 2.04 </td>
   <td style="text-align:right;padding: 1px;"> 2.26 </td>
   <td style="text-align:right;padding: 1px;"> 2.68 </td>
   <td style="text-align:right;padding: 1px;"> 2.53 </td>
   <td style="text-align:right;padding: 1px;"> 3000 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;padding: 1px;"> FIMS_fixed_effects </td>
   <td style="text-align:left;padding: 1px;"> spawning_biomass </td>
   <td style="text-align:right;padding: 1px;"> 2.11 </td>
   <td style="text-align:right;padding: 1px;"> 2.01 </td>
   <td style="text-align:right;padding: 1px;"> 2.21 </td>
   <td style="text-align:right;padding: 1px;"> 2.64 </td>
   <td style="text-align:right;padding: 1px;"> 2.44 </td>
   <td style="text-align:right;padding: 1px;"> 3000 </td>
  </tr>
  <tr>
   <td style="text-align:left;padding: 1px;"> C0 </td>
   <td style="text-align:left;font-weight: bold;padding: 1px;"> FIMS_random_effects </td>
   <td style="text-align:left;padding: 1px;"> spawning_biomass </td>
   <td style="text-align:right;padding: 1px;"> 2.08 </td>
   <td style="text-align:right;padding: 1px;"> 1.99 </td>
   <td style="text-align:right;padding: 1px;"> 2.19 </td>
   <td style="text-align:right;padding: 1px;"> 2.63 </td>
   <td style="text-align:right;padding: 1px;"> 2.46 </td>
   <td style="text-align:right;padding: 1px;"> 3000 </td>
  </tr>
</tbody>
</table>

---
# Benchmarking FIMS Against Established Assessment Tools
**Convergence success for 500 simulations across seven estimation models configurations**

---
# Benchmarking FIMS Against Established Assessment Tools

- FIMS estimates are consistent with established models
- Software differences (ADMB vs TMB) have minimal impact on results when configured equivalently
- Small differences arise from implementation details (e.g., recruitment constraints)
- Random-effects models reduce bias compared to fixed-effects models

**These findings support the use of FIMS as a reliable and flexible framework for next-generation fisheries stock assessments while maintaining consistency with existing modeling approaches**

---
# Challenges

.pull-left[
- Defining appropriate tolerance for statistical model outputs
- Maintaining consistent reference outputs across FIMS releases
- Benchmarking FIMS against other assessment models
- Managing execution time without losing coverage
- Is the {testdown} report useful and interpretable?
]

.pull-right[
![](https://static7.depositphotos.com/1026550/780/i/600/depositphotos_7809833-stock-photo-challenges-ahead-road-sign.jpg)
]

???
- How can we manage tolerance testing across both maximum likelihood and Bayesian estimation frameworks within the same suite?

---
# Summary

.left-wide[
- We don’t just test code. We verify that our implementation stays true to the underlying biological and statistical theory.
- Multiple automated "safety nets" catch errors at every stage, from individual formulas to full-scale model runs.
- FIMS is benchmarked against established models to ensure consistent and reliable results.
- Every update is automatically tested, documented, and openly available for peer review, building a "glass box" you can trust.
]

.right-narrow[
<img src="static/Hook_and_Kelly_scientific_software_errors.png" width="100%" />
]

.footnote[
[1] .hyperlink-style[[Hook and Kelly, 2009:](https://se4science.org/workshops/secse09/Presentations/09_Hook.pdf)] Testing for (code) trustworthiness in scientific software.<br>
]

slideshow.on('afterShowSlide', function(slide) {
    var count = slideshow.getSlideCount();
    var index = slide.getSlideIndex();
    bar.style.width = (index / (count - 1) * 100) + '%';
  });
})();
</script>