-
Notifications
You must be signed in to change notification settings - Fork 0
/
README.Rmd
77 lines (59 loc) · 4.27 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
---
output:
rmarkdown::github_document:
html_preview: no
---
<!--- README.md is generated from README.Rmd. Please edit that file -->
# Benchmark solvers
[](https://www.tidyverse.org/lifecycle/)
```{r, eval = !file.exists("data/final/results.rda"), include = FALSE}
knitr::knit_exit()
```
```{r, include = FALSE}
# load results
load("data/final/results.rda")
```
## Summary
This repository contains code for benchmarking the performance of exact algorithm solvers for conservation planning problems (see [Mittelmann 2021](http://plato.asu.edu/bench.html) for general purpose benchmarks). Specifically, these benchmarks examine the performance of the [_Gurobi_](https://www.gurobi.com/) and [_IBM CPLEX_](https://www.ibm.com/analytics/cplex-optimizer) commercial solvers. They also examine the [_CBC_ (COIN-OR branch and cut)](https://projects.coin-or.org/Cbc), [_HiGHS_](https://highs.dev/), and [_SYMPHONY_](https://prioritizr.net/reference/add_rsymphony_solver.html) (via the _Rsymphony_ and _lpsymphony R_ packages) open source solvers. The benchmarks are performed using a realistic conservation planning dataset, involving `r nrow(spp_data)` species and costs derived from land valuation data (see [Schuster *et al.* 2020](https://doi.org/10.7717/peerj.9258)). To investigate how well different solvers scale with different sized datasets, these benchmarks examine different versions of the dataset wherein planning units are resampled to different resolutions. The [results of benchmark analysis](https://prioritizr.net/) are reported in the [Benchmark vignette](https://prioritizr.net/articles/benchmark.html) of the [_prioritizr R_ package](https://prioritizr.net/).
## Usage
After downloading (cloning) this repository, you can rerun the analysis on your own computer using the system command `make clean all`. **Please ensure that the `MODE` variable in the _Makefile_ is set to `release` to obtain correct results.** Project maintainers can then use the `make export` command to upload the results files (`results/results.rda` and `results/solutions.tif`) to the [Results storage release](https://github.com/prioritizr/benchmark/releases).
## Repository structure
This repository is organized as follows:
* _data_:
+ _raw_: Raw data used to run the analysis (see [Data storage release](https://github.com/prioritizr/benchmark/releases/tag/v0.0.1) for large files).
+ _intermediate_: Intermediate data generated during analysis.
+ _final_: Final data created after completing the analysis.
* _code_:
+ _parameters_: Settings for running the analysis ([TOML format](https://github.com/toml-lang/toml)).
+ [_R_](www.r-project.org): Code used to run the analysis.
* _results_:
+ `results.rda`: Rdata file containing the results.
+ `solutions.tif` GeoTIFF files containing solutions form benchmark runs.
* _renv_: [_R_ package management](https://rstudio.github.io/renv/).
Note that the files in the _results_ directory are created after completing benchmark analysis. They are not present in the online repository. If you wish to access the results completed on our systems, please see files attached the [Results storage release](https://github.com/prioritizr/benchmark/releases).
### Software required
* Software
+ _GNU make_ (version 4.1+)
+ _GDAL_ (version 2.2+)
+ [_Gurobi_ (academic licenses are available for no cost)](http://www.gurobi.com/)
+ [_IBM CPLEX_ (academic licenses are available for no cost)](https://www.ibm.com/analytics/cplex-optimizer)
+ [R (version 4.3.1+)](https://www.r-project.org)
* System packages (Ubuntu operating system):
+ gdal-bin
+ libjq-dev
+ libprotobuf-dev
+ protobuf-compiler
+ libudunits2-dev
+ libgdal-dev
+ libgeos-dev
+ libproj-dev
+ libv8-dev
+ libxml2-dev
+ libssl-dev
+ libgit2-dev
+ coinor-libcbc-dev
+ coinor-libclp-dev
+ coinor-libsymphony-dev
## References
[Schuster R, Hanson JO, Strimas-Mackey M and Bennett JR (2020) Exact integer linear programming solvers outperform simulated annealing for solving conservation planning problems. _PeerJ_, **8**: e9258.](https://doi.org/10.7717/peerj.9258)
Mittelmann H (2021) Benchmarks for optimization software. <http://plato.asu.edu/bench.html> [accessed February 2021].