Add CMake-based build support

[Mizux]

FYI Travis-ci build: https://travis-ci.com/Mizux/CoinUtils

note: I have also fork for Osi, Clp, Cgl and Cbc since we need it for google/or-tools (to avoid #87 since CMake support space in path contrary to autotools/GNU Make)
note: There is also a simpler version already living in the vcpkg repo https://github.com/Microsoft/vcpkg/tree/master/ports/coinutils but not working so much
see: microsoft/vcpkg#5166 (comment)

[Mizux]

False call on Travis-CI see https://travis-ci.com/Mizux/CoinUtils/builds/106899978

[tkralphs]

Thanks for the PR. I re-built on Travis and it looks good now. We've had quite a few requests for CMake support over time. We'll have to look carefully at what you've done here and decide how to integrate it. Are you in a position to support it going forward? None of the core developers are very expert in CMake, so I hesitate to try to start supporting it without help.

[Mizux]

Hi,

Sorry for the delay...

• I'm currently working, as a Google LLC Contractor to maintain Google OR-Tools which is using CBC/CLP as backend solver so it is more/less in my scope to do this CMake support of our dependencies.
• Providing your own CMake support will avoid poor integration like in vcpkg and help all the CMake user to have one single source of truth (i.e. your repository)
• You can still add a disclaimer CMake support is Community supported...

[christoph-cullmann]

Is there any ongoing work on this?
CMake support would be awesome.
At the moment AbsInt uses some internal CMake hack to get all things properly compiling in our CI on all operating systems ;)

[tkralphs]

We are in the process of re-working the entire build system and CMake support is one of the items to look at. However, we don't have any experts in-house, so it is tending to sit on the back burner. A group of us are working quite intensively on Cbc this week and I will see if we can manage to take a look at this.

[Mizux]

I've bumped my PR from the Apr 15, 2019 to the actual master head...

It may still need few fix to work...
EDIT: should be ok now..

[tkralphs]

Thanks for the bump and the update. We just finished the updating of the auto-tools based build system that has been underway for a (very) long time. We still have a few things on the TODO list before I'll be ready to take a closer look at this, but hopefully soon. Thanks for your patience!

[CLAassistant]

All committers have signed the CLA.

[christoph-cullmann]

Hi, had some chance to look at this?
Having a CMake based build system would help me a lot to get current master to build properly on Windows ;=)

[Mizux]

Since all others PR for Cbc have been silently rejected I don't think this one will be accepted...
IIRC, I don't think Coin-OR team want to allow a community supported CMake based build system (like Protobuf or GLog did)...

The problem is now you have weird patch outhere (e.g. in vcpkg), instead of having one source of truth here, which can help to consolidate/federate the FOSS community effort...
ref: https://github.com/microsoft/vcpkg/tree/master/ports/coinutils

see for master branch:

• Add CMake based build support on master Cbc#217 (rejected)
• Add CMake based build on master Clp#105
• Add CMake support Cgl#46
• Add CMake support Osi#122
• Add CMake-based build support #101

for stable branch:

• Add CMake based build support on stable/2.10 Cbc#287 (rejected)
• Add CMake based build on Stable/1.17 Clp#129
• Add CMake based build support on stable/0.60 Cgl#51
• Add CMake support Stable/0.108 Osi#128
• Add CMake-based build support on stable/2.11 #116

[christoph-cullmann]

Hi, any update on this?
At the moment, given the build system got more complex with the generated config headers it is really not that nice to build it at all on e.g. Windows whereas the CMake build variant will allow exactly the same build process on all major operating systems.

[tkralphs]

I know it may look like PRs are being silently rejected, but I can assure you it's not the case. I was probably far too optimistic in my comment above. Although things are slowly getting done, there is still a quite sizable TODO list. I can fill you in on the many things we still need to do as part of our overall move to Github, a revamp of our build system ,etc. if you're interested. It's more than meets the eye. And of course, that comment above was made just before the guano really hit the fan :(.

Keep in mind that the entire COIN-OR Optimization Suite stack (20+ projects) is being maintained by a (very) small band of unpaid volunteers in their spare time. I completely agree with you about having a single source of truth, but there has been a proliferation of people trying to package different COIN-OR projects and most are doing it without coordinating with us, so the fracturing is happening all over the place. I try to follow up on the ones I see and most are packaging in ways that are autotools-friendly, so it's easier to deal with them. I have nothing against CMake, but I initially thought it would be a good idea to know a little about it before incorporating it into the main development branch after which we would inevitably start getting questions about it.

I'm admittedly a little sad to hear that our build system seems difficult, since I have put a huge effort into ensuring that building is easy on all popular platforms with all popular compilers. I don't know of any platform and compiler where this is not the case and we have automated builds going across all platforms that show how to do it, as well as pretty good documentation. I guess it would help to know exactly what the blockers are for just using the build system as it is (aside from just simply preferring CMake). COIN-OR projects can be installed in an ever-expanding number of ways that includes all platforms.

As for these PRs, since I haven't gotten to it them as quickly as I would like, I'll entertain any proposals you might have. If the PR is going to go into master, I would prefer to have at least some minimal documentation of what options are available for building with CMake in different configurations, etc. and a committed maintainer. A better option might be to maintain a branch that is kept in sync with master (or the current stable), but where the CMake harness lives. While I don't mind if there are people who really need CMake coming here as a source of truth, I still want it to be clear that the autotools build is probably always going to be far more flexible and well-supported, it's what coinbrew is based on, yada, yada. So unless there is a really good reason, one should default to the autotools version, which is supported on homebrew, Linuxbrew, conda, across all package managers for Linux, etc. We also autogenerate binaries for every commit on all platforms. I couldn't tell if vcpkg is actually autotools friendly, since there's no real developer documentation that I could see.

We are of course an open community and help is always welcome with a wide range of tasks that would move things a long more quickly! This is just a slow-moving train due to the lack of manpower.

[christoph-cullmann]

Hi,

just to make sure: I am VERY happy with the effort you all put in maintaining + improving the coin-or projects.

It's the best MIP solver solution I found so far that is open source, I love it.

For the building:

A major issue for me is, that with the current system, building & configuring is completely different between Unices & Windows.

With CMake it will be 1:1 the same, which makes it a lot easier to use if you can't rely on pre-compiled binaries.

[tkralphs]

@christoph-cullmann Thanks for the support! What do you mean exactly that building and configuring is completely different between Unices and Windows? In Msys2, I'm building with exactly the same recipe, which is what I personally like about our current build system. Being able to build with the Visual Studio compiler, but using the same recipe as with Gcc and clang is convenient. And there is also WSL in Windows.

[christoph-cullmann]

Hi, sorry, I wasn't clear.

With on Windows, I mean on a Windows without any Unix-like extra like Msys2 (or WSL, which is no real solution to integrate stuff into your build either).

With CMake, it works only with a Visual Studio installed and no foreign tooling. e.g. at company we don't use any msys/cygwin/... whatever but only the native stuff. We did always run into issues with mixing msys/.. and Co. and other projects that rely on plain Windows setups.

I can understand, that if you have enough external stuff installed, you can use the configure scripts on Windows, too.

But integrating CBC into stuff that only uses the native tooling isn't that easy with that.

(given now Visual Studio even supports CMake directly and you have package managers like vcpkg mentioned above that rely on it)

[tkralphs]

But this isn't a fundamental technological hurdle, it's just a preference, right? I understand that if you already use CMake for other purposes and don't use MSys2, then installing MSys2 is an extra step. But it's a very lightweight install, not like Cygwin, you don't even need to use a GUI installer, you can just unzip a small archive somewhere and can delete anytime. It's useful to have around anyway if you are already used to using bash in Linux.

It would seem to solve all of your problems to just use coinbrew in Msys2 to build libraries with the Visual Studio compiler that you can then easily link to directly from other Visual Studio solutions or CMake builds. The steps could even be easily put into a Windows batch script or Powershell script. If you really wanted to keep Msys2 out of the way, you could temporarily unpack a very minimal MSys2 install and then delete is afterwards.

Another alternative that I haven't personally tried but should be possible is to use Git bash, which you probably already have on your Windows machine. Or the bash that can be installed with conda. If nothing else, we will soon have a conda build working on Windows (this can't be done with 2.10.3, but should work with master).

[christoph-cullmann]

We need to version all our stuff on our own to be able to rebuild old stuff.
Any extra component is extra work, needs to be updated, etc.

Besides, if you work on Windows (or Unices) with CMake for close to all other things, it is not done with just unpacking some extra tool.
You must ensure that e.g. the compiler flags you want are propagated 1:1 like for your CMake stuff.
(e.g. complex rpath settings and such are even ugly on Unices to get right for CMake and a configure thingy)

Don't get me wrong, I just seem to fail to see a bit the issue with just moving to CMake, if somebody already provides the needed steps.

With that it "just works" on all systems you support at the moment and the CMakeLists.txt are actually a lot easier to maintain than what configure/autoconf/automake/... does.

There is a reason many C++ things switched to CMake, KDE years ago, Qt last year and so on.

To avoid the hassle with the configure stuff, I did now write some local CMake files myself again (that are completely useless to share, as they have zero configurability and just build coinutils/clp/cgl/cbc at once).

Still, it's not me that will need to maintain the stuff and if you feel it is easier to keep all things as is, this is fine for me, the people that do the work decide. I hope my comment here doesn't offend you, that is not the intention, I value the work that is done on this project a lot, the build system isn't the crucial thing, would just be nice to have.

[tkralphs]

I understand everything you're saying as far as versioning and tracking, etc. I still don't really see exactly why you couldn't build the COIN-OR libraries in a completely independent way and just treat them as if they were provided to you pre-built by a package manager, as I guess this is essentially what you're doing on Linux. But anyway, I get that it's a bit complicated and it is certainly preferable not to deal with more than one build system. This makes sense.

As far as the arguments for CMake, I am familiar with them and have participated in my share of lively debates on this topic over 🍺 :). If we were starting fresh, I guess the choice would be easy. Also, if we were a big project with a large number of developers and even some financial backing (as the others you mentioned are), then we would also have an easier time deciding to switch. But there is a big picture that makes it highly non-trivial from a purely pragmatic perspective to make this move. Since this comes up on a regular basis, it might be useful exercise to just try to lay out the full list of steps that would be needed, which go well beyond what others have already provided so far. This list is certainly non-exhaustive.

1. Most importantly, there are 20+ inter-linked projects (plus an array of third-party projects that we also support building, all of which themselves use the autotools natively) in the "Optimization Suite" and it would be pointless to convert some without converting all. Only the Cbc stack has been converted to CMake so far.
2. Our current automake setup supports a (very) wide array of build options that I can't imagine are all captured in the current CMake setup and would all need to moved/translated for all 20+ projects.
   • The knowledge of how to link to a variety of optional packages, what workarounds are needed, etc., has been accumulated
     over 20+ years and would all need to be transitioned.
   • All the logic of what symbols need to be defined in order to enable optional components would have to moved.
   • All the custom arguments to configure would need to be replicated, however that works in CMake.
   • Some projects have very complex builds that involve building different sets of artifacts for parallel execution using MPI, etc.
     We would need to replicate all of this logic.
   • Many projects have examples which themselves consume the libraries of the project and essentially have their own
     independent build harnesses that would need to be replicated. Currently, the configure scripts automatically generate and
     install additional human-readable Makefiles that the user can then use to build the examples later by hand.
   • A variety of other artifacts are produced and installed at build time that would also need to produced by CMake somehow,
     such as automatically generated parameter files for test cases and examples, pkgconfig files, doxygen configuration files,
     doxygen documentation itself (as well as other documentation).
3. We have automated scripts for updating the build harness of all 20+ projects in an automated way, which would need to be fully replaced.
4. We have scripts for automatically creating new stable versions and releases that do things like automatically generating the version number, replacing in various files where needed, re-building the configuration scripts, templates, etc., and automatically building and testing everything. All of that would need to be replaced.
5. We have scripts for auto-generating all the configuration files for CI (Travis and Appveyor), which would all need to be replaced.
6. We would need to re-work Docker images, homebrew recipes, conda recipes, Debian packages, Fedora packages, etc.
7. coinbrew itself would need to be completely re-worked, as many of our processes depend on it and users also depend on it to fetch dependencies and automatically build projects monlithically.
8. Build documentaton would need to be updated in many places.
9. Finally, everyone who currently consumes COIN-OR projects would need to update their build processes. This would be a huge impact on users.

If some benevolent person came along and could provide all of the manpower needed to do all of the above, it would still need to be maintained. So we would need the on-going efforts of someone with CMake knowledge to help with that, at least until we accumulated the knowledge to do it ourselves. We have already accumulated that knowledge for the autotools over the last decade or two.

I'm pretty sure these are not the only complications that would be involved, but the list above is already one that would take years to achieve at the rate things are currently happening, given the available developer bandwidth. I hope that helps explain why it doesn't seem like even a remote possibility that we could do this unless something dramatic changes.

Still, I am open to the possibility of maintaining some basic CMake harness for some subset of the full Optimization Suite stack, but we would need to work out how to do that in a way that would not open up too many potential cans of worms. If someone has a proposal, let's discuss it. My best idea is to maintain CMake in one or more separate branches that some interest person could just keep in sync by pull request. I would be happy to just accept such pull requests quickly as long as they are living in a separate branch and we could point people who really want CMake there. Anyone would be free to sync it up and submit a pull request at any time.

[tosttost]

At work I'm in a similar situation to @christoph-cullmann : developing an application uses cmake and calls CBC as a library.
Our solution is to call coinbrew from within cmake instead of hacking some basic cmake support into coin-or. Works well enough for our use case (hard wire some of coinbrew's parameters...). We can crosscompile for windows on a linux host using mingw, so we have one build system for all platforms and some hope that our build system does not break every time we update CBC.

Given the limited developer capacity I'd rather vote to make CBC thread-safe by default, improving setting parameters and clean up that >100 different #ifdef, especially does that are undocumented and fix some bugs - I know they exist because I reported one of the bugs fixed that way.
I know @tkralphs is already refactoring a lot of stuff and I greatly appreciate that.

task: how do you make CbcMain0/CbcMain1 print into a file instead of stdout?

Hint: solver->messageHandler()->setFilePointer(FILE*) does not work.

solution: you pass in your on message handler.
Calling setFilePointer on the default one does not really work because there are a few places where a new MessageHandler is created except you've passed in your own. In that case no new message handler is created.
Luckily coin-or is free software and we can debug it ;)

[tkralphs]

@tosttost Thanks for the suggestion on using coinbrew from within CMake! Indeed, I'm doing some refactoring and I will soon be at a point where I would welcome help on some things. The topics you point out (thread safety, parameter mechanism, and getting rid of #ifdefs) are top on the list. I have tried to make a small amount of headway on better user control of output. Cbc has a multiple different mechanisms that affect output and also a mix of just printing to stdout versus using the message handler.