Centralized Build System
Compile build is most time consuming process in software development life cycle. When I have first started my current company the situation was same. We have more than 2 million lines of Java code and we are using Clearcase as source control and ant as build tool. We have two main ant targets to create end product.
- Compile – Compiles source code
- Build – Generates client code from wsdl, compiles jsp and creates war files, creates jar files etc.
Compile takes nearly 120 minute and build also takes 120 minute on a developer’s machine so totally 240 minutes. In my first year in company they gave me a fairly slow desktop PC. I think it was a dell computer. And a compile+build was taking more than 450 minutes. Imagine that I had to do that two or three time in week before delivering my changes to ensure that my changes were not going to broke main branch. Another bad thing is running compile/build nearly freezes computer. Your PC become unusable, opening a menu in eclipse or editing and saving code takes too long during compile/build. It was totally waste of time.
Sixth month later I learnt that we have 12 retired servers. They were totally idle. These servers are Sun Netra 240 servers. These machines have 4GB of memory. I logged in one of these machines with ssh and played a bit. I figured out how to open a Clearcase view and how to run our compile/build process on this machine. These machines are very fast if we compare it with our desktop computers. It can complete compile/build in 75 minutes. We have (240 – 75) 165 minute gain in every compile/build and another advantage is running compile/build on another machine. Developer’s desktop does not freeze anymore because of compile build. But somehow it did not worked as I thought. Developers found it hard to open a view via ssh and running ant scripts from command line. Another problem was assignment of servers. We have nearly 100 developers and 12 machines. Compile/build process slows down if three or more developer wants to run compile/build at same machine. So everyone continued to use their own desktop computer. Because it was easy to use Clearcase user interface and they were able to run compile/build just clicking one bat file.
Then I decided to build easy to use build system by using these machines. Following was my requirements.
- It must be web based system. No body wants to install extra software. It is an overhead for users(developers). If you can make it web based, make it web based.
- System must make load balancing. Compile/build request must be evenly distributed among servers.
I did not have root access to these machines also so I need to select tools which are I can run it without root access.
Java
I prefer Java because Java is the language which I feel most comfortable. And none of the other language interpreters was installed on system and as I said I did not have root access anyway Java was okay for me 
Tomcat
I decided to use tomcat as web server because I was familiar with it. And you can run it without install just extract from zip and run.
HSQLDB
I think it is best solution for java side if you want to embed a database in your application it is small and it covers all your needs for small applications.
Here is a small figure of complete system. Master server (also a build server) serve web pages and distribute build request among servers evenly. Server communicate each other via web services.
Populate user and stream data
Master server runs following command every night to populate user and stream data.
cleartool lsstream -invob <vob-name>
This command lists all streams with user name and its creation date.
2005-02-18T15:19:11+02:00 <stream name> <stream owner>
Database
Now we are going to put these user name and stream to database. Here is my humble database tables.
Stream: Stream name and creation date. This table is extracted from Clearcase data.
User: Owner of a stream.
My application runs “cleartool lsstream -invob <vob-name>” nightly and updates stream and user tables.
build_request: This table contains compile/builds requests which come from users. This request is sent to build server and is deleted after a message is received from build server which indicates that build succeeded/failed.
Downloading Output Files
When application is initialized it also initialize Apache FTP Server. Apache FTP Server is just a jar file and you can embed it into your application easily. FTP server’s root directory maps main build directory. When build is completed application send email to user. This email has ftp links and user can download output files from there. We have one big zip file at the end of build (end product) and it is about 350MB. I prefer a FTP server here instead of letting users download output files from http to decrease load on tomcat server.
Maintaining Disk Usage
Nearly 10GB of new files generated every day by this application (classes, war files etc.) To keep disk usage at reasonable levels application check build folders every day and if there is any unused build folder (folders which are older than 5 days), it is deleted. If any user make a lot of build in a day only newest three output files (which I was stated previous, 350MB zip file) are kept and rest is deleted.
How It Looks Like
When user logged in, all his streams are listed. See following figure. You can select one of your streams from table or you can just type stream name in “Stream Name” text box. Actually this text box have auto complete feature so if you want to build someone else’s stream just type some characters there and it shows you matching streams. Compile, recompile, build and zip check boxes correspond our ant targets. Zip classes/work check boxes are used to zip compiled classes and other output files. These files are useful if you do not want to make full system compile/build and micro compile/builds like package compile or single war file build are enough for you. We have other product specific options under advanced tab.
Users can see status of their builds in following screens. Stream column shows which stream is being built now. When user clicks a stream under stream column, a new tab is opened which points to ftp://server-ip/user/stream. User can see all generated files like logs, compiled classes etc. Server column shows that which server runs your build. State column shows current operation, they are “Creating View’,'Compiling’,'Building’,'Deleting View’ and more. Elapsed column shows how much time elapsed during build. User column indicates user who has started build. Cancel column to cancel build. It kills ant process at remote machine. Only build owner can cancel a build. If he is not owner of a ongoing build ‘Cancel’ link does not appear for that user. If someone else hijacks your account and cancels your build, you receive an email which says “Your build request cancelled by <ip-adress>”. If ip address does not belong to you, you can easily find guilty by running ‘tracert <ip-address>’. It works well at corporate environment
This screen is used to monitor all servers in system. Actually it is not for users, it is a public administration screen. I follow server disk usage and build request distribution from this screen. First column shows server ip address and link points to ftp://server-ip/. This ftp location list all users’ build directories in server. Second column shows disk usage. Background threads runs in system periodically to keep disk usage in reasonable percentage. Build directories which are not accessed more than five days are deleted. If disk usage exceed critical levels system administrator (it is me) receive an email. (It did not happened yet ) Version column shows current software version which runs on system. Sometimes I need to upgrade some servers individually to new version to fix a bug so I can see which server runs which version from there. Last column shows server’s load.
In this post I have tried to summarize that how I have built a centralized build system which is well integrated with Clearcase. We have prevented broken code delivery to mainstream by this centralized build system. Developers started to make full compile/build on their streams frequently by using centralized build tool so untested code delivery was ended. It has also increased developer productivity because developers are never interrupted anymore because of compile/build process.
You can build your own in house centralized build system. It has advantages; every company has its own compile/build process so you can create one which well integrates your development process and development style.It is also kinda fun to create useful tools for other developers. There are also open source build systems like Cruise Control and you can also integrate it your compile/build process too.