(dis)agreement between solvers

[mihai-sysbio]

I've been running a knockout screen of 600+ genes across 50 tissue models. Using COBREXA, I've leveraged its use of the JuMP framework to test over 20 solvers. The main take-away is that HiGHS (free, open-source) gave identical results to Gurobi and Mosek (commercial, academic licenses), and almost identical with Tulip (which was slightly worse and about 50% longer).
There is one puzzling bit though: OSQP, a different solver, has managed to find flux vectors with 1.5x higher flux carried by the objective function than what the 3 other robust solvers have. Does anyone have any experience/thoughts on this?

[exaexa]

Hi,
yes this is common. I'm currently finding workforce for making a rigorous benchmark of all this stuff (actually I ran quite some already with FROG models, which is a nice curated dataset, but we need someone to actually interpret it...).

The current findings:

• HiGHS is pretty good, the solver that I generally liked most was SCIP
• Gurobi sometimes overshoots optima by tolerance so much that it then fails at 99%-optimum FVA (this is a pretty good test for tolerance handling btw)
• CPLEX is mostly OKish but not open
• OSQP has terrible problems with tolerances sometimes reporting success despite having unsatisfied constraints (I assume that might be your case)
• Tulip.jl is super cool for small problems, Clarabel.jl is surprisingly cool for quadratic stuff (sometimes you need to give it a bit of a poke by initializing it by another linear solver's output, but then it optimizes QPs very nicely)
• GLPK is super conservative w.r.t. tolerances, thus sometimes not finding a solution.

Linear solving is a wild business.

I'm attaching a picture with performances (completely uncompensated/uncurated) from the mentioned benchmark. HiGHS ain't bad. I kicked out some solvers (including OSQP) for producing errors. Also, I unfortunately didn't know SCIP is a thing by then.

Hope this helps. :D

fba-all.pdf

BTW this list is quite useful because it's usually the most updated :D https://jump.dev/JuMP.jl/stable/installation/#Supported-solvers

[exaexa]

re "model or the solver" -- depends, we got models which solve reliably with solver A but not B, and others which go better with B. I guess that avoidance of numeric issues is kinda model responsibility, but can you reliably avoid all numeric issues in all future solver versions? (no)

[cdanielmachado]

I see, that's an interesting validation to have. Although i would say the job of the model is to represent the biology. Dealing with numerical issues should be the responsibility of the simulation methods. I'd say we don't really "solve a model", we solve a particular formulation of a particular simulation method. :)

[exaexa]

Yeah, "validation" would say it best.

The point with the model interpretation is valid but the scale is huge, in extremes you can go for "interpret as pure data" (and just publish the genome untainted by COBRA view, letting everyone build the model themselves) or "interpret as program" (which gets you perfect reproducibility for the predictions etc, but good luck guessing what all the magic constants and reactions do there). We hover somewhere in the middle in the high utility area... :)

[mihai-sysbio]

Interesting how this turned into an exploration of what a model is. Representation of biology indeed, but does it work only for this one dataset? or this one research question? or only when I run it on my machine? In my view the general aim is for the model to perform "reliably"/"consistently": for more users (including in different sofware environments) and for more research questions. Every bit of improvement counts 💪🏻

[exaexa]

@mihai-sysbio the "reliably/consistently" is basically in favor of "model as tested software"; you can't really do any reliability or consistency test on "base data".

OTOH since software is basically a function (or many functions) that transforms tons of parameters into some kind of result, this view for me kinda puts a limit on the re-usability of the model data (what if you want to do something that the authors didn't ever think of).

So maybe (to avoid the "reliable blackbox" view of models), we could go with a software where all code has some attached biological annotation? e.g. "we multiply by 1.23456 here because that's what was measured by Guy et al [123]"

...or maybe data with precisely standardized ways of interpretation? (Think of SBML, but where they actually standardized the interpretation. :D )