Marcus D. Hanwell's Blog

Reflections on 2011: Open Source, Open Science and Open Chemistry

2012-01-01T20:40:41Z

It has been so long since I uttered a word here, 2011 was certainly a busy year for me and I hope to ensure I dedicate more time telling people about what I am up to in 2012. In preparation for that I have spent some time moving my blog to some new hardware, in the cloud. I also got to the bottom of the poor performance of page loads, and things should be much snappier now. After all that I figured it was time for a cosmetic refresh, so after upgrading the Serendipity I selected a new and hopefully cleaner theme.

As an extra special treat I updated the photo to something a little more recent to - me emerging from my TARDIS with a mug of espresso in hand! Kitware grew a lot last year, so much so that we had to take space in a new building across the road. It was decided that our scientific computing team would move, along with the communications team. In the move I got my own office, with a view of the old office across the street.

VTK was accepted as a mentoring organization in the Google Summer of Code program. We were lucky enough to get two very talented and tenacious students who produced some great work over the summer. We also continued improving and extending Gerrit, and thanks to the Google Summer of Code I had the opportunity to attend both the Mentor Summit and the Git Together (held the day after). Not satisfied with two meetings in one trip I also attended a small portion of the Open Science Summit, and hope to be able to attend the whole thing if it happens again in 2012.

Speaking of Open Science, 2011 was a big year for the area both on a personal level and in the wider community. I talked about our work in open science at several conferences, and more specifically the work we have been doing in Open Chemistry. I wrote a Source article introducing the work that we have done in Open Chemistry since I joined Kitware, and we recently acquired the openchemistry.org domain and have begun populating it. We were also awarded a Phase II SBIR which gives us two years of funding to develop many of the applications and libraries that I mentioned in the article.

The Science Code Manifesto was conceived in 2011, and Michael Nielsen released his new book Reinventing Discovery. There was also Open Access Week which highlighted the need for open access to scientific journals and data. I was very pleased to be a coauthor on two open access articles in 2011, the Quixote project and the Blue Obelisk five years on. I was also honored to receive my own Blue Obelisk award in 2011 from Peter Murrary-Rust!

There is so much else, but I am out of time for now and this post is already very long. Google+ was released to the masses, you can see a little more of me there, and I have created an Open Chemistry page that I will try to keep updated over the coming year. I was invited to a workshop on Semantic Physical Science in Cambridge, England and so I will be starting my traveling very much earlier than usual - leaving next Thursday. Here is to a great new year, one in which I hope we as a community can make significant progress in opening science for the world, and creating a truly shared set of tools for all!

I hope that 2012 is the year more of us start sharing in meaningful ways.

Conferences: Talking Open Science at OSCON, Desktop Summit and Chemical Databases Meeting

2011-09-04T00:37:00Z

Over the last two months I have had one of my most hectic travel schedules ever. It started withOSCON, and a panel discussion about "Practicing Open Science". This one was a bit of a surprise, as Bill Hoffman was originally presenting with Will Schroeder and Brian Wylie, from Sandia National Laboratories. As Bill couldn't make it we decided to change the content of my section, and talk about the new open chemistry area that I have been working on for about four years now. Will went first, followed by me and a wrap up from Brian, with a nice flow between Kitware working on open science for over a decade, me growing a new area of open science (now at Kitware) and Brian giving a government perspective on open source and open science. The slides are below and on slideshare if you would like to take a look.

Oscon 2011 Practicing Open Science

View more presentations from Marcus Hanwell

I thoroughly enjoyed OSCON, and would love to attend future events. The toughest thing was deciding which talks to attend as there were often multiple tracks with talks of interest to me. This was also by far the largest and most commercialized open source event I have attended so far, in the beautiful city of Portland, OR. I couldn't stick around for long after the conference as I was flying out to England on the following Tuesday, and on to Berlin, Germany Friday to attend the Desktop Summit. This was my first time in Germany, and I was looking forward to exploring Berlin a little, along with some time to catch up with a few family and friends in England before and after the conference. I talked about "Open Source Visualization of Scientific Data" on the final day of the main conference, and was very pleased to have a large and interested audience. Here I also discussed my work in open chemistry, along with a lot of the other work we do at Kitware in the Scientific Computing group.

Open Source Visualization of Scientific Data

View more presentations from Marcus Hanwell

I stayed for the remainder of the conference, attending my first KDE e.V. meeting, and was joined by Bill Hoffman towards the end of the week. Bill gave a workshop on using CMake, and I helped out with that, along with taking part in several BoF sessions and meetings. It was a very hectic week, very different feel to OSCON with a lot of great presentations, BoFs and hacking sessions. I also had the opportunity to meet up with Alexander Neundorf who was an intern at Kitware for half a year, and several other KDE developers interested in build systems, software process, testing, coverage and related areas.

Then I was back home for just over a week before braving the elements and heading straight for the path of hurricane Irene. I was invited to the 5th Meeting on U.S. Government Chemical Databases and Open Chemistry where I talked about "Chemical Databases and Open Chemistry on the Desktop". This meeting was very focused on chemical databases and the open chemistry I have been working on so hard for the last few years. It was a great experience to be able to see what others are working on, and discuss possible points for future collaboration. There is some amazing work happening in this area, and this meeting helped me gain greater clarity on how my work at Kitware can fit into the larger picture to significantly improve the landscape in open chemistry.

Chemical Databases and Open Chemistry on the Desktop

View more presentations from Marcus Hanwell

Thanks to Kitware for allowing me to attend, and funding my travel/other expenses, and to my wife and son for tolerating my long absences over the last couple of months. An even bigger thank you to my wife, Louise, for letting me off the hook on my first missed wedding anniversary so that I could present at OSCON! I had some great news about funding for the continued development of many of the ideas discussed in the slides, and so hope to have much more to talk about over the coming months (and years). This post is already pretty long, I hope to continue developing this work and promoting open science, especially in chemistry, materials science, physics and the bio areas. There are lots of other amazing people working in these areas too, and I feel like we are getting to a point where we can create real change to improve the outlook in scientific research.

Talking About Open Source Visualization of Scientific Data at the Desktop Summit

2011-08-05T00:20:00Z

I have begun my journey to the Desktop Summit, making the flight over from the US to Manchester yesterday. A short stay in Sheffield, and catch up with family before heading out to get my flight to Berlin tomorrow. I will be talking about the work I have done both at and before joining Kitware with the title "Open Source Visualization of Scientific Data". I plan to talk about a range of work from my Google Summer of Code project on Kalzium back in 2007, through to some of the exciting work at Kitware in VTK, ParaView and Titan looking at the challenges of large data, remote visualization and how to integrate the web and smartphones/tablets into the scientific data visualization workflow.

Bill Hoffman is also planning to attend, and we will be running a workshop introducing CMake on Thursday. This is my first Desktop Summit, although Bill and I have both attended previous aKademy and Camp KDE meetings. I should be in on time to attend the pre-registration event, and will not be leaving until Saturday. Looking forward to a great summit, catching up with some old friends and making some new ones. Now, I think I should try to get some sleep before my flight tomorrow!

Talking at OSCON 2011 about Open Science

2011-07-26T18:30:00Z

I am currently on a plane bound for Portland, Oregon enjoying the in-plane wi-fi. Will Schroeder, Brian Wylie and I will be talking about "Practicing Open Science" on Friday in the government track. I am standing in for Bill Hoffman who unfortunately could not make it, and will be discussing the work I have been doing to grow open chemistry both at Kitware and outside of Kitware with many amazing collaborators scattered around the world. I am really excited to have the opportunity to talk at OSCON, and would be happy to meet up and discuss this work if you are at OSCON. Will and Brian are both very passionate about open science too, they will both give their unique perspectives on practicing open science. I will be there from this evening and don't fly out until early Saturday morning.

I am very much looking forward to OSCON, and the major difficulty I have had is choosing between the talks that are all happening at the same time. In some cases there are two or three I would like to see in any given slot. I am hoping to attend the KDE release party tomorrow too, please join us there if you would like to celebrate with us.

Avogadro 1.0.3 Released

2011-04-25T16:58:39Z

I am pleased to be able to announce the availability of Avogadro 1.0.3! What happened to Avogadro 1.0.2 I hear you ask...shortly after tagging Michael reported an issue with i18n building/installations. So 1.0.3 contains a couple of very small build system fixes, but see the 1.0.2 release notes for details of most of the fixes.

As always, we appreciate your feedback. There are still a few issues outstanding, but many things were fixed. These binaries are also built against much newer versions of Qt and Open Babel where significant improvements have also been made. There may be one or two more releases of the 1.0 line if necessary (I have streamlined the release process with a view to making more releases), but I would like to focus our efforts on an unstable release for 1.1. Once 1.1 is stable, a 1.2.0 release will be cut and branched. There are lots of new features in master that we would love more feedback on.

Blue Obelisk Award

2011-04-13T17:22:45Z

At the recent ACS Spring meeting I attended the Blue Obelisk dinner, where I was honored to receive a Blue Obelisk award, pictured below, for my contributions to Open Data, Open Standards and Open Source. This is largely due to the work I have done on Avogadro, Open Babel and other open source chemistry tools.

This was one of the biggest dinners I have had the opportunity to attend, and I got to meet many of the people I have worked with (or used their work), along with several people I had not had the opportunity to work with yet, but hope to in the future. We presented the work we had been doing on Quixote project at the chemical information symposium on chemistry and the internet, after attending the first Quixote meeting the previous week (thank you to Hartree Centre for inviting me to speak there, and sponsoring the event).

These are exciting times, thank you very much to Peter Murray-Rust for presenting me with the award, and all of the support he has shown, along with his relentless passion for open science. I have only been a part of this for a few years, but Peter has been working on opening up chemistry for decades now.

Visualization Toolkit (VTK) in the Google Summer of Code

2011-03-27T16:38:32Z

As I already mentioned on the Kitware blog, the Visualization Toolkit (VTK) has been accepted as a mentoring organization for Google Summer of Code this year. You can see the VTK entry in Melange, and browse through our project ideas. I have taken part in the Google Summer of Code program since 2007 (first as a student, and later as a mentor) as part of the KDE project. I still maintain close ties to KDE, and work on several related projects such as Avogadro, CMake and VTK. VTK has Qt integration, and ParaView builds on both VTK and Qt for the visualization of large scientific data sets.

If you are a student, and would like to work on an exciting open source project, processing and visualizing some of the largest scientific data sets in the world, take a look at the Visualization Toolkit. There are a wide range of ideas, and if you have an idea you think would fit then please feel free to discuss it with me. I will let you know if it would be a good fit, and whether we have available mentors for the proposed project. We have mentors available who are experts in visualization, large data, parallel algorithms and related technologies. The core of VTK is written in portable C++, with new changes being tested daily. Our API is automatically wrapped in Python, TCL and Java.

I am very excited about VTK's first year in the Google Summer of Code, this represents a unique way for students to get involved in a large, well tested open source project. We have started using Gerrit for code review, and you can view build and test results on many platforms for VTK both continuously and nightly. We have a well established software process which will serve you well in any project where software quality is important, with nearly 1400 unit and regression tests. This is a large, collaborative project with more than 100 contributors last year (as measured by Ohloh).

CMake External Projects: Building Project Dependencies

2011-03-05T21:41:21Z

Historically projects have attempted to minimize their dependency list, and often bundle in small third party libraries in an attempt to make things easier for new developers/users to compile their code. In the Avogadro project we have bundled a few really small libraries, but on the whole have maintained a dependency list and tried to keep it smaller. As I work on new code, I see opportunities to break off bits of functionality, such as with OpenQube, but don't want to add yet another thing a new user or developer must download, compile and install somewhere.

Linux packagers, myself included, dislike the practice of bundling in libraries. It means that instead of patching one libxml2, we get to patch one plus the three or four in our tree that have been bundled (often with different version, some local patches). The problem is less pronounced on Linux where package managers are ubiquitous and we are able to provide a list of packages to install, but even there we might be developing against versions not yet in the main distribution repository. This is one of the reasons I have always favored rolling release distributions over the periodic.

CMake's external project module helps us to deal with this issue in quite an elegant fashion. Coupled with meta repositories to bring several source trees together, CMake is able to direct the build of several projects, passing locations between projects and expressing dependencies between the projects being built. This means that something like Open Babel can build zlib and libxml2 before building the main Open Babel library. External projects and CMake allow us to download the source, create the build trees and even direct the build of non-CMake based projects like libxml2.

I have a prototype of this that I just put up to build the core of Avogadro, its working name is Avogadro Squared as I was feeling geeky that day and had no good names. One thing you should note is that everything in there is an external project, and Avogadro is the last one to be built (it depends on all of the other projects). It requires minimal changes to the projects it contains, it uses git submodules for some of the source, and CMake's download and tar functionality for zlib and libxml2. I will be adding options to simply use system versions of the libraries it can build, but Linux distributions etc can continue using the Avogadro repository directly.

As a new developer or user I can checkout the meta repository, have git download the submodules and CMake download the source tarballs. I can then build the entire project, and then continue to work in the Avogadro subdirectory of the build tree after that. That build tree is almost identical to the one I would have ended up with had I not used the meta repository, except it points to the dependencies I just built. I can then use vim, and IDE or whatever I choose to work on the inner projects. This works across Linux, Mac and Windows to get new users and developers up and running very quickly while only loosely coupling the dependencies to the Avogadro project.

I have worked on other larger projects, such as Titan and ParaView that are using this approach to a greater or lesser extent. Titan can actually built Qt, Boost, VTK, protobuf, Trilinos and a host of other dependencies before building the Titan libraries and applications. I think Avogadro Squared is an example of just how minimal a meta repository can be, although I will be extending it with more dependencies it really is just a glue repository.

Volume Rendering in Avogadro

2011-01-16T03:23:43Z

Since joining Kitware I have had limited spare time to work on Avogadro, and for various reasons my spare time has been more limited than usual too. Since the new year I have been able to start spending more time working on Avogadro, and open source chemistry in general, thanks to an SBIR phase I proposal that was funded last year with the US Army Corps of Engineers. This is exciting for a number of reasons, including the fact that I have the opportunity to prototype exciting new features for chemistry visualization, workflow and data management.

One of the new bits of work I have been doing is to use some of the advanced visualization techniques in VTK such as GPU accelerated volume rendering. Now the code is still pretty rough, and is more a proof of concept. I wrote a simple external Avogadro extension that links to and uses VTK to render the first volume found in the current Avogadro molecule. All of the parameters are currently fixed, I am hoping to get the time to add in more options along with some integration of the Avogadro rendered molecule in the VTK render window. You can view the code here, please bear in mind it is at a very early stage.

I have also been working on several other things such as splitting out the quantum calculation code from the Avogadro plugins, and putting it in a small library. I have called the library OpenQube, right now it only has the base functionality that was in Avogadro but I will be extending it with more features, regression tests and I am hoping due to the decoupled nature and liberal BSD license it will encourage wider collaboration in this field.

There is also the Quixote project which I am very excited about. Meaningfully storing the results of quantum calculations, annotating them and retrieving them within an open framework. This is a growing problem in todays world, and I am working on extensions to Avogadro to allow it to fully exploit the semantic chemical web. This includes some of the previous work to access the PDB and other public resources as well as private databases within groups and organizations.

I think this is going to be a very exciting year for Avogadro, and open source chemistry in general.

One Year at Kitware...Already?!?

2010-10-31T19:23:00Z

It is hard to believe, but I have been at Kitware for just over a year now. How are things going? I would say very well...I am very pleased I made the move, and that Bill Hoffman pushed me into applying after meeting him at the first Camp KDE. Kitware is growing fast, we are always on the lookout for new talent and I am already starting to feel like an old timer with all of the new employees joining.

I had my first ever annual review, which went well. We received word in September that my first SBIR proposal had been accepted, and we are working on getting the contract in place for that. So watch this space - a great collaboration coming up working on open source chemistry visualization, editing, database integration, and computational chemistry input file generation along with analysis of the outputs. I think this is a great opportunity to extend VTK, and Avogadro.

I took a very active role in our migration to Git, and I am pleased to say that it has been going well. I also more recently got Gerrit up and running, introducing tightly integrated code review to some of our open source projects at Kitware. I played a large role in setting up one of our most complex build systems to date for Titan, where it can build Qt, Boost and VTK (among other dependencies) using CMake's external project features. I have also had the opportunity to work with some of the Boost developers, and am helping with their build system work.

I have mostly concentrated on 2D rendering in VTK, using OpenGL. I replaced the existing 2D charts in VTK and ParaView with new charts using a new 2D rendering abstraction. So we now have a selection of chart types, with interactivity, that can be used on both client and server side. More recently I have been going even lower level, and working on FreeType font rendering in VTK, and seeing what I can do to improve the capabilities there.

There is lots of other stuff, some of it I have talked about here, and other bits I will when I find time. It has been a great first year, and looks like it is shaping up to be an even better second year. I feel very lucky to be getting paid to work on open source, open science and I get to work on some very interesting problems that help real scientists. Going forward I hope to introduce more scientists to open source, open data, open standards and collaboration platforms. I am very privileged to have worked with so many forward looking scientists over the last few years, and am a proud unmember of the Blue Obelisk.

I think Kitware is the perfect place for me to push forward open source in science, and am refreshed that I rarely need to push anyone here in that direction. I have been driven to learn a lot of new things in the past year, and it has been tough at times, but I have thoroughly enjoyed it. There are some really amazing projects coming up in the next year - so watch this space!

England: Open Source Chemistry and Real Ale

2010-10-17T13:42:00Z

In September Louise, William and I went to England for a little over three weeks. It was a great opportunity to catch up with friends and family, and because we needed to stay for an extended period (in order to obtain new visa stamps), it also seemed like it would be a good opportunity to meet with researchers around the country and talk about open source, software development, chemistry, physics and the role that I and Kitware might play in improving the field.

As we landed at London Heathrow on Monday, we parted ways hours later at London St Pancras station. Louise and William headed up to Sheffield to catch up with family, while I headed up to Chester for the first Open Babel developer meeting. Chris Morley kindly hosted the event, and it was the reason I came out as early as I did. It was great to see Chris and Noel again, as well as meeting Tim who has been a long time Open Babel and Avogadro contributor. In addition to all of the development discussion Chris gave us a great tour of Chester.

After that it was our third visit to the US embassy, to get new visa stamps. After a little drama, and having to keep a toddler entertained during the hours of waiting, all was well and we went for a little pub food before doing some shopping on Oxford Street. We spoilt William by buying him a Paddington bear from Hamleys, and eventually got the tube to our friend's house. This was the first time I had ever had afternoon tea, with the little cucumber sandwiches and everything. We tried it at a little cafe in Harrow on the Hill, and it was very nice. William loved the chocolate cake from the bottom tier.

On the way back up to Sheffield we took the opportunity to visit another friend in Harpenden, and then I spent a whole day in Sheffield before heading out again. This time I was off to visit a friend in Lancaster University, who had invited me to give a talk after another researcher had told them about all the cool things they had done with Avogadro. The talk went well, and we veered off into how Kitware can make profit giving its software away, version control and software process.

I had an even shorter stay in Sheffield before heading back out to Manchester, and Daresbury Laboratory, developers of GAMESS-UK and DL Poly among other codes. It was great to go back out to Daresbury Lab, and I was very graciously hosted by Jens. I had previously been invited to a workshop he organized to talk about tools and visualization in chemistry in early 2008. I had the opportunity to present what had been happening with Avogadro development, and many of the new things I am doing at Kitware in both VTK and ParaView development. We also discussed licensing, open source and how scientific research can benefit from this approach to software development.

I had the opportunity to meet quite a large array of the developers at Daresbury in the two days I spent there, as well as help Jens with some Open Babel and Avogadro development. We finished off the week with a visit to the University of Liverpool, starting off with a tour of the centre for materials discovery. After that I gave a very brief talk, and a fuller live demo to a group that does mostly computational chemistry research. We rounded off the day with a walk around Liverpool, and a great meal before a long train ride back to Sheffield.

After a brief morning with my family I was off again, to Nottingham this time for a friends wedding. I hitched a ride with another good friend who was also at the wedding and now lives in Bristol, which luck would have it very close to Cardiff. I spent a really fun Sunday in Bristol checking out the city, and the historic village of Clifton before getting a train over to Cardiff (Wales) to meet with more chemists.

This was my first trip to Cardiff, and it was very fun. I visited the chemistry department at Cardiff University, where I gave another talk about Avogadro and some of the new things I am doing at Kitware (and planning to do). The talk was well attended by quite a cross-section of people (even a few ParaView users in the audience). I gave another live demo (I should do them more often) to show off some of the features of Avogadro. Peter Knowles (one of the principal Molpro developers) kindly hosted me while I was in Cardiff.

After a morning working on some code with Peter, I headed to the train station on Tuesday afternoon to get back up to Sheffield. This was almost my final visit, and I have to say I thoroughly enjoyed visiting so many people, but it was exhausting and I still need to sort through a backlog of email. After seeing so many people I see a real need for Avogadro, and improved visualization and software process in computational chemistry. I am also very encouraged by some of the codes going open source (NWChem after I got back), and some that will likely to go open source.

Tuesday evening to Sunday was my Sheffield time. I went into my old department to present some of the work I do in scientific visualization (Avogadro, VTK and ParaView) to people at the University of Sheffield, largely with a focus on materials science. A fire alarm went off as I was being introduced, and we weren't allowed in for twenty minutes...that was disruptive. Most people were able to return after the unplanned break though, and I got a lot of interesting questions from the audience, along with several going away to download what I had just shown off.

William met lots of family while I was traveling the country. I also managed to get out for some real ale, and visited Kelham Island on the Thursday evening. I also took the opportunity to get in some full English breakfasts, "real" bacon and the classic fish and chips. We went out for a few curries too, I miss the curries you can get in England. It was great to catch up with old friends and family, although I didn't get chance to see as many as I would have liked.

On Sunday we got a train down to London, and stayed in a nice hotel during our final night in England. We took William to the palace, but the queen was unavailable to host us...so we headed back to our hotel, grabbed our bags and made our way to the airport. Travel went quite smoothly, once we got back into the US we picked up our rental car and drove back up to Clifton Park. Not sure I want to do that again, but it was great to be home.

Since then work has been keeping me very busy...I have been working on this post for about two weeks and only just found time to finish it off. Hope you enjoy reading about my adventures. It was great to get back after almost two years away. I am very excited about the work I am doing in computational chemistry, and feel that we are on the verge of accomplishing great things that will have wide impact. More to come on this soon, I am certainly very happy to be where I am and to have the opportunity to work with such a great community.