Mastering the art of command line jujitsu is something every developer (regardless of OS) should strive for. While OS UIs are becoming more and more slick and helpful tools are abundant, possessing the ability to drop into a console and efficiently knock something out (find . -iname *.java | xargs egrep -i “todo” anyone?) has come in handy, for me at least, on more than one occasion.

For instance, every modern IDE now has a plug-in that facilitates working with Subversion– in fact, when I made the switch to Subversion from CVS a fews years back, I initially failed to even learn the Subversion commands– my favorite tools did it all for me graphically! Yet, in some cases I found myself distrusting the implementations these tools provided– for instance, one of my favorite CVS commands was cvs -nq update, which effectively tells you what’s changed in your local sandbox as compared to the tip of the CVS repository. This, of course, is handy in figuring out what you plan on committing; hence, I would run this command before a cvs commit so as to filter out unneeded changes.

This developmental cadence of making changes and checking them against the tip (and consequently committing them) is actually common among SCM systems– the cadence has three steps:

• updating a local sandbox to reflect what is in a repository
• understanding what has changed locally
• committing local changes into a repository

With Subversion, these three steps can be achieved via the command line with three commands:

• svn update
• svn diff
• svn commit

The svn update command takes any changes in a Subversion repository and pushes down to a local sandbox. For instance, if more than one person is making changes and committing those changes into a repository, when another person runs the svn update command, those changes will be literally downloaded into that person’s sandbox.

Using the update command is as easy as opening up a console and typing:

tty:~/dev/projects/easyb/ yts$ svn update

If there are changes in the repository, you should see some scrolling text like so:

U    maven-plugins/maven-easyb-plugin/src/main/java/org/easyb/maven/StoryReportMojo.java
U    maven-plugins/maven-easyb-plugin/src/site/apt/usage.apt
U    maven-plugins/maven-easyb-plugin/src/it/story-report-location-test/pom.xml
U    maven-plugins/maven-easyb-plugin/src/it/site-examples-test/pom.xml
U    intellij-plugins/easyb-plugin/easyb-plugin.iws
U    intellij-plugins/easyb-plugin/src/main/java/org/easyb/plugin/ui/EasybPresenter.java
U    intellij-plugins/easyb-plugin/pom.xml
Updated to revision 329.

Note, the U means updated– there are other characters for new files added, merged, in conflict, etc. What’s key here is that by running the update command, your local sandbox is in sync with what’s on the server. Running this command often often means you’ll probably avoid conflicts too!

After you’ve made some local changes, it’s often times helpful to capture what those changes were– if you’ve been working for any length of time or working with more than one file, you’ll most likely find the diff command indispensable.

Like the update command, diff can be run via the command line like so:

tty:~/dev/projects/easyb/ yts$ svn diff

This command will compare what’s in the repository with what’s in a local sandbox; what’s more, it’ll report on the exact changes as follows:

Index: behavior/groovy/org/disco/bdd/prototype/FixtureFeature.story
===================================================================
--- behavior/groovy/org/disco/bdd/prototype/FixtureFeature.story        (revision 322)
+++ behavior/groovy/org/disco/bdd/prototype/FixtureFeature.story        (working copy)
@@ -4,7 +4,7 @@
 description "this story is fleshing out fixture logic for scenarios"

-each "some description" , {
+every "some description" , {
     given "a value", {
         value = 12
     }
Index: src/groovy/org/disco/easyb/StoryBinding.groovy
===================================================================
--- src/groovy/org/disco/easyb/StoryBinding.groovy      (revision 322)
+++ src/groovy/org/disco/easyb/StoryBinding.groovy      (working copy)
@@ -102,7 +102,7 @@
       listener.describeStep(description)
     }

-    binding.each = {  description = "", closure = {} ->
+    binding.every = {  description = "", closure = {} ->
         beforeScenario = closure
     }
     return binding

As you can see in the text above, the local code (labeled as “working copy”) has changed the each call with every, which appears in revision 322 of the repository.

Once you’ve reviewed these changes and are happy with what you’ve got, you can then proceed to push these changes into the repository via the svn commit command– of course, this assumes you’ve run a local build and it passed successfully, right?

tty:~/dev/projects/easyb/ yts$ svn commit -m "updated dsl to use every instead of each"

Note that the commit command can take a flag (-m) that enables you to specify a comment. You should see some text indicating the changes are being pushed into the repository.

Sending        easyb/behavior/groovy/org/disco/bdd/prototype/FixtureFeature.story
Sending        easyb/src/groovy/org/disco/easyb/StoryBinding.groovy
Transmitting file data ..
Committed revision 325.

Note how these changes increment the revision number in Subversion– this is key as all commands allow you to work with various revisions (via flags); hence, by knowing a particular revision, you can logically branch if needed.

These three Subversion commands will come in handy if you aren’t already using them; hence, learn them now! Indeed, for becoming one with the command line is the path to enlightenment.

Let’s say you have a fairly complicated build structure, with multiple batch files that fire off different aspects of the build (even if, at the end of the day, they just call ant).

And let’s assume that for various historical reasons, these batch files encapsulate their environment variables using SETLOCAL and ENDLOCAL.

At first glance, this is a pretty cool way to scope out your environment variables. But then, you find out that Hudson determines if a build has succeeded or failed by looking at the %ERRORLEVEL% environment variable. If it is not 0, the build has failed.

Uh oh. Your build is failing, but the moment ENDLOCAL is called, %ERRORLEVEL% is set back to its previous value, which, most likely, will be 0.

Given this situation, how do you get the %ERRORLEVEL% back up to Hudson? Well, don’t call ENDLOCAL. Instead use:

EXIT %ERRORLEVEL%

at the end of your batch file. This works because there’s an implicit ENDLOCAL at the end of your batch file, and EXIT will return the ERRORLEVEL as the exit value.

It seems the build platform debate continues to heat up with Steven Devijver’s posting on the DZone entitled “What’s Wrong with Build Systems in Java Today?” where he previews a build platform that builds upon Ant and Maven 2 and adds, as he states:

Better-than-Maven 2 JAR and project dependency management, No XML, and Integration of Ant, Maven 2, Gant, Buildr, Rake, Grails, … projects in multi-project builds

Clearly, in the Java space, Ant is the de-facto build platform; consequently, the future of build languages will most likely need to leverage the extensive infrastructure in place to support Ant– dropping XML is key should you require a more imperative style of building things. Dependency management has always been a splintered solution ranging from Ivy to Maven’s declarative dependency framework to Ruby’s gems– all interesting and viable solutions, which makes his statement all the more interesting.

All in all, these are exciting times as dynamic languages are enabling a complete change in the way we define software builds, which will ultimately lead to tools that make building software (that is, compiling, testing, inspecting, deploying, etc) easier.

Production-ready software, on-demand

The general notion of the practice of Continuous Integration is that it’s performed during the “development cycle”. I think we need to expand this into areas typically thought of as “latter lifecycle” activities. In my experience, even the most Agile of projects aren’t considering all aspects of what it takes to make software “production ready”.

Over the past year, I’ve been advocating a much more intensive criteria for Continuous Integration with my peers and customers, which is more like “Continuous Production” meaning you can produce production-ready software on a daily or even a continuous basis. Effectively, we’re talking about extending the Continuous Integration model into all environments (Test, Stage and Production), not just the development environment. In fact, at Stelligent, our purpose is to get our customers to production-ready software on a daily basis.

A posting on Wikipedia describes Continuous Production as “a method used to manufacture, produce, or process any product without interruption. There is no discrete rate at which goods are produced, as opposed to a batch production process, or job production.” I’m suggesting we move toward a similar model for developing software.

A lot is bundled in this term, Continuous Production, though. In my experience, it means that you automate most everything it takes to create software so that it can be deployed into the user’s environment any time. It means you may automate some or all of the following:

1. Compilation of source code and tests - this is a given for CI.Tools: Ant, MsBuild, rake, NAnt or similar build tool.
2. Database Integration/Migration to target database server (e.g. Oracle) - Some shops are doing this with CI. Tools: Ant sql task or ORM-based Ant tasks, dbdeploy
3. Testing - execution of unit, component, system, functional/acceptance, security, load and performance tests - Some shops are doing this with CI. Tools: JUnit, DbUnit, Selenium, JUnitPerf, FindBugs, JMeter, NUnit, NDbUnit
4. Inspections - execution of static/dynamic analysis checkers: code coverage, code duplication checks, coding standards, cyclomatic complexity and dependency analysis - Some shops are doing this with CI. Tools: PMD, CheckStyle, FindBugs, JavaNCSS, JDepend, Cobertura, NCover, NProf, Source Monitor, NCover
5. Deployment - Deployment and repeatable configuration to servers (e.g. JBoss) - Some shops are doing this with CI. Tools: Application Server-specific Ant tasks (e.g. WebLogic)
6. Remote Deployments to target environments - For example, from a build machine to staging environment (for web containers and database servers). Tools: Ant, Java Secure Channel and SmartFrog, Capistrano
   1. You will need a way to rollback to a previous deployment (this includes database changes)
7. (Optionally) Creation of installation distribution - Few shops are doing this with CI. Tools: NSIS and antinstaller, InstallShield .
8. (Optionally) Generation of distribution media - Few shops are doing this with CI
9. Source labeling and (possibly) build labeling - Some shops are doing this with CI
10. Automated notification of build status - This is a given for CI via Email, X10 devices, etc.
11. Automating the promotion through QA, Staging and Production environments - Promote the binary and accompanying artifact through each target environment. Tools: Build Management servers such as AntHill Pro are useful for build promotion
12. Documentation Generating the readme docs, installation and user guides. Tools: JavaDoc, NDoc, Doxygen, GraphViz, Grand, etc.

Of course, the actual steps will vary depending the type of software application you are producing whether it be a rich client, web application, embedded or other…but you get the point. Continuous Production, to me, means you have the capability to create production-ready software whenever necessary .

What I am saying is that the creation of production-ready software should become a “nonevent “.

The benefits are numerous, but here are a few:

• Decisions can be made based on working software , not on a task item on a project management chart - no more “it’s 80% done”
• Eliminate the reliance on an individual with specialized knowledge to produce the software for production
• There’s “done”, there’s “done, done” (coded with acceptance tests). With Continuous Production, it’s “done, done, done” and ready for release to users (code, acceptance tests, operating in the customer’s environment)

Note: This does not mean that you will be taking up cycles producing production-ready software with every change to the version control repository. Test, Stage and Production builds are more likely to be performed on demand (by a person). However, the point is that your build environments are capable of building production-ready software on a continuous basis.

There are very, very few projects that are doing this right now. I’d have to guess it’s much less than 1%. It makes me think of the saying “it’s not the destination, it’s the journey”. If your team sets this as a goal, the closer you get to it, the less you’ll be spending time on environment-related, repetitive and error-prone issues and the more time you spend on new features and improvements to the software. Is it possible, right now, for every project to do this? Probably not. However, once you start asking questions like “What do I need to do to make this feature production ready?”, it changes your perspective on everything from coding to testing and deployment.

In reality, I think even the most automated development environments still need (or should) have a manual component when creating software that is production-ready. This may include usability testing or even a simple manual sanity check. But, I think the more we move toward this model, the less headaches we’ll have and the better our estimates.

In my book on Continuous Integration, I make a reference to something Tim O’Reilly mentioned regarding Flickr’ s ability to deliver updates to its web software as frequently as every 30 minutes. This is only possible if you’ve automated many of the deployment steps.

Let me know if you or someone you know is doing this or moving toward this type of model. I’m curious to hear your experiences with it.

Over the course of a few years a number of build automation tools have surfaced. One tool that is widely accepted as a standard in the Java community is Ant, a cross-platform build tool that uses an XML file format.

Ant’s attractiveness stems from its ease-of-use and ability to be seamlessly extended with custom capabilities. Plenty of people know how to use Ant and there is a broad ecosystem of tools and support around it. On the other hand, given the nature of the XML syntax, Ant is occasionally limited when it comes to expressiveness. As a result, build tools based on an existing scripting language have entered the scene.

One such tool, Raven, a build platform built on top of Ruby, leverages the power of a full-featured programming language with the simplicity of a build-centric Domain Specific Language. Paul Duvall describes this in the latest installment of his “Automation for the people” series,

“In particular, Raven enables dependency-based tasking with a full-featured imperative programming language (rather than a declarative one like XML).”

In addition to expanding on the benefits of Raven including demonstrating two approaches to getting Raven installed and configured, his article, “Build Java projects with Raven” also illustrates the relationship between Raven and Rake. If you’re looking for a tool that enables you to utilize the power and flexibility of the Ruby language within a build script, check out the article first, then check out Raven.

Learn about the other installments of “Automation for the people”, a series of articles dedicated to exploring the practical uses of automating software development processes and teaching you when and how to apply automation successfully.

I think one of the reasons that I have had great success with CI Factory when introducing teams to CI is polish, or ease of use.  Some of that polish comes with CI Factory out of the box, this post is about that extra that I have found myself adding of late.

I like to provide a script that will get developers all setup from the get go.  Recently I have found the best place to locate this script to be on the CCNet dashboard.

The workspace setup script should check for the needed source control client, create the directory structure needed, and get the latest from the source control repository.  Lastly it opens a Windows Explorer window to the Product directory:

From here the developer can run a build by double clicking the Build.bat.  I have gotten to including a check for required software at the beginning of the personal build.  For example if Visual Studio 2005 SP1 is required I check for it and prompt the developer about it if missing.  Where possible I have the script install missing software with the developers permission.

Once all the required software checks pass the rest of the build runs as normal.  The last bit to be polished is the cctray install.  In CI Factory the cctray download is a zip not a setup.exe.  I like to turn this into a self extracting zip and include a pre-configured settings.xml file.  This way when cctray starts up for the first time it is already configured.

In my experience these things leave developers the head room to focus on things that deserve their attention.  Communicating this through actions like polish go a long way to demonstrating how CI can have an affect on the quality of their lives.

To add download links to the main web dashboard edit the file:

C:\Projects\<project name>\Current\Build\dashboard\templatesFarmSideBar.vm

<table width="100%">

  #foreach ($link in $links)

  <tr><td><a href="$link.Url" class="$link.LinkClass">$link.Text</a></td></tr>

  #end

  <tr>

    <td>

      <a href="WorkspaceSetup/CreateWorkspace.bat.txt" >

        Workspace Setup Script

        <br/>Save As .bat

      </a>

    </td>

  </tr>

</table>

Here is an example of some NAnt script added to the personal build that checks for VS 2005 SP1:

<property name="VisualStudioServicePack" value=""/>

<readregistry hive="LocalMachine"                  

              key="SOFTWAREMicrosoftDevDivVSServicing8.0SP" 

              property="VisualStudioServicePack" failonerror="false"/>

<ifnot test="${VisualStudioServicePack == ‘1′}">

  <ask answer="Answer" 

question="It looks like Visual Studio 2005 SP1 is not installed.  Do you wish to continue?" 

      caption="Proceed Without Required Software?" 

      showdialog="true" >

    <options>

      <string value="Continue"/>

      <string value="Stop and Install SP1"/>

      <string value="Exit"/>

    </options>

  </ask>

  <ifthenelse test="${Answer == ‘Stop and Install SP1′}">

    <then>

      <asyncexec program="cmd" 

      commandline=‘/C " explorer http://msdn2.microsoft.com/en-us/vstudio/bb265237.aspx"‘ 

          createnowindow="true" 

          redirectoutput="false" 

          useshellexecute="true" 

          waitforexit="false" />

      <fail message="Installing VS 2005 SP1!" />

    </then>

    <elseif if="${Answer == ‘Exit’}">

      <fail message="Please install VS 2005 SP1!" />

    </elseif>

  </ifthenelse>

</ifnot>

The batch file for creating the workspace is a little clunky compared to NAnt script.  I have been mulling around the idea of including NAnt in a self extracting zip and executing a script instead of this batch file fun:

echo off

set ProjectName=CI Factory

set SVN.URL=https://ci-factory.googlecode.com/svn/%ProjectCodeLineName%

mkdir "%ProjectCodeLineDirectory%"

SET /P Anonymous="Do you wish to do an anonymous checkout of the source? Yes for patch creators, No for submitters:(y,n)"

IF %Anonymous%==y set SVN.URL=http://ci-factory.googlecode.com/svn/%ProjectCodeLineName%

IF EXIST "%ProgramFiles%\TortoiseSVN\bin\TortoiseProc.exe" GOTO UseTortoise

svn –version

IF NOT %ERRORLEVEL%==0 (set PATH=%PATH%;%ProgramFiles%\Subversion\bin) ELSE GOTO UseSubversion

svn –version

IF NOT %ERRORLEVEL%==0 (set PATH=%PATH%;%ProgramFiles%\CollabNet Subversion Server\bin) ELSE GOTO UseSubversion

svn –version

IF %ERRORLEVEL%==0 (GOTO UseSubversion) ELSE GOTO NoSubversion

:UseSubversion

IF %Anonymous%==n SET /P SvnUserName="What is the user name you wish to use to checkout the source?"

set Credentials=

IF DEFINED SvnUserName set Credentials=–username "%SvnUserName%"

svn checkout %Credentials% "%SVN.URL%" "%ProjectCodeLineDirectory%"

IF %ERRORLEVEL%==0 (GOTO OpenFolder) ELSE GOTO END

:NoSubversion

echo I can’t find where you have Subversion installed!

GOTO END

:UseTortoise

"%ProgramFiles%\TortoiseSVN\bin\TortoiseProc.exe" /command:checkout /url:"%SVN.URL%" /path:"%ProjectCodeLineDirectory%"

IF %ERRORLEVEL%==0 (GOTO OpenFolder) ELSE GOTO END

:OpenFolder

explorer "%ProductDirectory%"

:END

There are two general ways in which you can structure a build pipeline: one cuts across the product and the other across validation.

I was recently asked about breaking up the build by having a build hierarchy that mirrored the product’s package dependency tree. I have seen this done, and done it myself, in a very coarse way, dividing along lines strongly marked both by organization and architecture. There is a even a well known pattern on this: Named Stable Bases. It should be made clear that this is a form of deferred integration. Yes, yes, the ill consequences of deferred integration are the very things that continuous integration is working to alleviate. Just because deferred integration has been abused in the past is no good reason not to wield it wisely now. So lets take a look at how it alters the dynamics of a build system. To show this I have created a loop diagram, see this post for more details on loop diagrams and remember that the "s" means varies the same and "o" means varies the opposite.

The entire product is not built, even compiled, all at one time; it is spread out over time. Many times there can be a significant delay between when a dependency is built and when a dependant consumes that new build. This breaking up of the build has the effect of decreasing the size of the Codeline which ultimately shows the effect of increasing the rate of change to the Codeline. This is a short sighted view, remember that we are building a whole product and that we have deferred some integration. The key here is some, not all. So the question is will the things effected positively out weigh the deferred integration? I have illustrated the forces at play here in a well know pattern or archetype in Systems Thinking named Fixes That Fail.

Fixes That Fail (^)

The Fixes That Fail structure consists of a balancing loop and a reinforcing loop. These two loops interact in such a way that the desired result initially produced by the balancing loop is, after some delay, offset by the actions of the reinforcing loop.

The internal balancing loop operates in the standard balancing loop fashion. The Action that influences the migration of the Current State also influences, after some delay, some Unintended Consequences. These Unintended Consequences subsequently impede the migration of the Current State in the intended direction.

One of the dangers I see with this approach is how it shifts the focus away from the product and the overall build systems to the product components and their smaller builds. It is more difficult to see the big picture and foresee the repercussions of actions in the small on the large. This type of build pipeline shifts the delay of feedback from an individual integration duration to the push or pull between individual builds. For example there is an uncomfortable delay in getting a developer feedback when the build of an entire product takes 45 minutes. One might feel a good solution is to break up this single build into 3 builds one for the common libs, one for the server, and one for the client. Now when a developer of the client submits to the build they get feedback in 8 minutes. Here is the deception of this model, the overwhelming majority of the time the client build is working against old versions of the common libs and server. The delay is getting the build output of the common libs and server integrated into the client build.

In my experience most people start down the path of splitting a build in this fashion by first seeing a waist in recompiling packages that don’t need to be recompiled. This is a valid observation and fruitful optimization, and I think that in most cases division of build process to realize this benefit yields a net negative in consequences. I have only seen this structure work well as a means to compensate for organizational issues. For example when the web front end team has little contact with the back end service team. The web front end team may experience interruptions due to changes in the back end that impede progress. One means to compensate for this is to accept regular stable deliveries of the back end. This allows for uninterrupted progress, flow, and regular planned integrations to the new version of the back end.

I have experienced great success with division of the build across validation as apposed to across the product. In the past I have referred to this as creating developer, release, tester, and customer facing builds. Let’s generalize that a little to: role facing builds. Before I explain remember that the desired state is rapid feedback and the gap is perceived delay in feedback. When we take into account the different roles on the team we realize that they each have different desires. The developers are accepting of less comprehensive feedback in exchange for quicker execution times. The testers in contrast are accepting of longer execution times in exchange for more comprehensive feedback. The developers are not interested in creation of an installer, verses everyone else is extremely interested in an installer. If you continue down this path of inquiry you will have a set of values that will help you shape a build pipeline. I bet that you would find only the developers perceive a delay in feedback. Creating role facing builds can be quite liberating in what you can do and what you don’t need to do in a build. It is the don’t need to do that has the greatest affect on the developer facing build. You don’t need to do a clean build, an incremental or dirty build can be very quick. You don’t need to execute long running tests. You don’t need to create an installer.

This is not a perfect solution, there is no such thing. So what about the negative consequences? A successful build from a developer facing build can give a developer the illusion that all is well. They need to remain engaged and interested in the results of the cascaded builds. I have not seen any other generic issues with this approach, here is an example of an issue with how it is implemented: It is important that the developer be able to easily execute all parts of any build in the system that can cause a failure and or create build artifacts. If for example a developer can not run a test that is causing a build to fail they will not be very likely to jump right on it. There will be negatives in what ever implementation you conceive of. You need to be alert to them as you will not be able to foresee them all. There are many ways in which to mitigate and or alleviate there impact.

Creating a loop diagram of your build system can be a tremendous help in this regard. Here is a picture of a generic build system to help illustrate the value:

Here is a link to the free software that I drew this with and the diagram itself:

http://www.simtegra.com

In “Am I Alone Here?”, Tim Goodwin has a nice blog commenting on his desire to run a single command build using only assets from the version control repository. I could not agree more. As Martin Fowler mentions in his Continuous Integration article “…anyone should be able to bring in a virgin machine, check the sources out of the repository, issue a single command, and have a running system on their machine.”

Tim points out some of the challenges he experienced while working in a Microsoft environment. It is imperative that vendors give us the capability to run everything from the command line with minimal effort. GUI tools are fine as long as you still have the capability to run the same from the command line or API. Wizards or other GUI tools provide a level of automation (the first time) but not when we need to repeat a series of activities. In particular, when we need to setup an automated Continuous Integration server.

I am fond of saying if you can’t run (a tool: server, program, etc.) from the command line, it doesn’t exist (as a tool).

P.S. When is Microsoft going to eliminate the Visual Studio dependency on MSTest? This means I must install an IDE on any machine I wish to build software when using MSTest…not an acceptable option to me.

In his excellent book, Customer Centered Selling, Robert Jolles relays a poem about consequences:

A recent client lacked concern about building software regularly and successfully. In his mind, build failures were equal to the annoyance of a hobbled horse: something resolved easily when a release date approached.

But this same chain of events could be applied to the software business. My crude version of the poem – adopted for software development – lacks the poetic prowess of Mr. Franklin. Hopefully, though, it illustrates a point.

We would all be well served to remember the potential impact of small problems as we consider where to spend our resources. Little annoyances like failed builds or test cases can be very early indicators that you are “losing the war” in your development efforts.

How important are the little annoyances in your software organization? Do you get concerned if a particular build effort fails? Do you have developer test cases? Are they executed regularly? Who gets heartburn if they fail?

For his latest Automation for the people installment, “Asserting architectural soundness”, Stelligent CTO, Paul Duvall, demonstrates how using tests such as JUnit, JDepend, and Ant can play a role in enforcing architectural reliability.

Asserting architectural soundness describes a technique to build checks into your build scripts to proactively detect violations while coding. Locating potential deviations as soon as they’re introduced, instead of waiting to flag problems after the fact, ultimately leads to better quality software delivered faster.

In addition, Scott Laningham, host/editor of IBM developerWorks podcasts, interviewed Paul about the software development need that he addressed in the article. Paul kicked off the interview by indicating that,

Over time software architectures tend to become brittle. The architecture that you think you have versus the architecture that you actually have, as its manifested in the code, is often different.

Whether you read the article or listen to the podcast (hopefully both!), you’ll undoubtedly carry away the need to take charge of your architecture by using a proactive build process.