CMake Performance with Open Babel
Recently, Luca made a post comparing the speed of CMake and autotools in which some timings were posted. I have to say that I am not sure I agreed with the conclusion and have had a very different experience with the projects I am involved in. As with anything your mileage may vary, and I have not looked at Wesnoth.
I think it is questionable at best to include the time it takes to build CMake, but not autotools. Seems like this is a one time cost and the build time is not that high for either. All tests were performed on my quad core Gentoo box at work. Each step is for the first cold run as would normally be the case when compiling Open Babel from source. The make step used `time make -j5` and I have listed the real time in each case.
The timings are shown in the table below. They seem somewhat similar to the experiences of the QGIS developers who made this move quite some time ago. All times shown are in seconds and are the real time reported by the time command.
| Autotools | CMake | |
|---|---|---|
| Configure time | 20.39 | 4.79 |
| Compile time | 116.45 | 102.23 |
| Install time | 28.39 | 4.19 |
| Total time | 165.23 | 111.21 |
For those interested, on this system the total CMake compilation/installation time (cmake-gui disabled) was 1 minute and 54 seconds. The compilation/installation time for automake, autoconf, libtool, m4 was 2 minute and 14 seconds. I am not sure how relevant either of those times are, other than to show neither of them take that long to compile and install. Gentoo users/developers may or may not have CMake installed, most other developers will install the binary packages for either one and are likely to be much more interested in how well it integrates with their development environment, compile and install times.
As a developer I prefer CMake, and have been using/maintaining the CMake based build system for Open Babel for over a year now. It was originally contributed by the KDE Windows porting team, but I found that I spent less time waiting for it to do things when I was working on code too. Add to that the extras CMake comes with, such as CTest, CDash and CPack I think it makes a very attractive option for many projects. I am also hoping that it will allow Open Babel to drop maintaining a totally separate build system for MSVC.
I am sure the Open Babel autotools build system could be optimized (I never tried), but when you add in the additional benefits mentioned above, support for multiple targets such as makefiles, XCode, MSVC, Eclipse etc, one shared language/syntax for all build files and an increasingly polished competitor to autotools, I honestly think it is a sensible choice for projects to move to CMake. There are a few less well known features such as Fortran module dependency parsing that I think are fairly unique and valuable, in scientific coding at least (and I have used the Fortran module dependency parsing at least once and was pleasantly surprised).
Full disclosure: I recently accepted a job offer with Kitware, and will start in the Fall assuming all the visa paperwork falls into place. The opinions expressed here are my own. I think it is great to discuss issues like this objectively, and hope to be a part of making CMake a better build system. As with most software - there are areas that need improving.
Trackbacks
No Trackbacks
Comments
Display comments as Linear | Threaded
lu_zero on :
Marcus D. Hanwell on :
Ultimately, having some competition in this area is likely to cause both projects to improve. The Boost project is also seriously considering a move to CMake from what I have seen.
Geoffrey Hutchison on :
The configure step is usually the biggest improvement for CMake over autotools for me. As a maintainer, autoconf re-runs frequently and doesn't cache as much data than CMake. So the performance improvement is a big productivity boost.
Marcus D. Hanwell on :
There are so many other benfits to this move for many projects, that the build performance becomes a bonus. For my Gentoo centric readers the build timings are of a greater concern when projects move to CMake.
Ian Monroe on :
I just ported TelepathyQt4 to cmake. Quite a complex thing since there was so many generated files, and I was porting to Windows at the same time.
CMake really just wins by default since automake doesn't even try to work on windows. But I also just find it a lot nicer to work with. That said its still a build system, so it still sucks.
What a troll of a blog you linked to. If I was an automake-supporter I would cringe in embarrassment.
Marcus D. Hanwell on :
I will certainly do what I can to help. I agree with you, when working on a project that aims to support Linux, Mac OS X and Windows - build performance becomes secondary to OS support (which is very good) and that makes CMake the obvious choice. At the end of the day most of us are developers, and do not want to spend our time working on build systems. I think this is another area where CMake really wins.
You don't read many reviews raving about how nice the build system a project uses is. It is all about the program/library - what features it has, reliability, cross platform performance, speed, etc. That is what I alluded to at the end of the post - CMake can actually help with some of these when you start using CTest, CDash and CPack. I think it is good to make some objective comparisons of different solutions, assess where improvements could be made and hear about what other people in the community think.
James Pike on :
One major disappointment is the clunky old fashioned language cmake provides. It feels like using visual basic.
I was very disappointed to see that Kitware did not show any interest in the lua bindings which allowed you to write cmakefiles with all the power of lua.
This is one sore point for me.. something that is great could be absolutely fantastic, but without kitware accepting that the cmake "language" is inadequate for many tasks nothing will happen.
As an example.. I rewrote a few of my common cmake macros in lua in about a quarter of the space. Cmake already saves me time and space, and it could save so much more.
Marcus D. Hanwell on :
I will do some reading. I will say that one attractive thing about CMake is that there is just one language to learn for the whole build system management task. The more dependencies added, the longer it takes to build and I would think this would also add to the complexity of porting to different architectures. I don't know enough about lua to comment on the size of the dependency or the cross platform support. I would imagine the situation is much better than Python for example.
David Valentim Dias on :
I think you should create a method for compare the timing using a n number of runs and test the difference of them using some statistic method (t-test?).
Sorry if my English is not good.
Marcus D. Hanwell on :