There is an interesting thread on InfoQ entitled “Opinion: Code Coverage Stats Misleading” where the author states that coverage reports do not provide adequate confidence on testing efforts. It’s a good read– I also recommend checking out my article dubbed “Don’t be fooled by the coverage report” which uses code examples to show how misleading coverage really is.

Web 2.0 components Ajax and mashups were among the key players of dozens assessed in Gartner’s annual report identifying emerging technologies and their impact on businesses over the next decade. According to the report, mashups are nearing the hype cycle peak and “provides incremental improvements to established processes that will result in increased revenue or cost savings for an enterprise.”

The mashup term, derived from the pop music scene to describe the result of combining music from one song with the vocal from another (think Christina Aguilera and the Strokes hit mashup “A Stroke of Genius”) has extended its name to the Web. Instead of a combination of songs; a mashup also describes a Web site or Web application that combines content from one or more sources (think Craigslist + GoogleMaps = HousingMaps.com). HousingMaps took Craigslist’s real estate ad content combined with GoogleMaps’ API to create a visual representation of CL’s text listings on a map, helping others better discern where exactly a listing is located.

There is far more to mashups than just maps, but the majority of mashup activity has been predominant in the consumer space; meaning they’re public and often created as a proof of concept rather than a business (think Google Maps + celebrity web data = Celebrity Maps). Very fun (and popular), but not exactly useful unless you’re in the paparazzi business. In recent months companies have begun evaluating whether or not there is a business in mashups. A new term has come into the IT vernacular – “Enterprise Mashup,” which essentially means a simpler, more cost-effective way of integrating applications (think expense reporting, tracking payments and invoices, sales force collaboration, etc).

The Gartner report is on the right track – there is a rising corporate appetite for faster and more effective ways of building applications. Much like the way blogs have transformed the idea of each person having a voice on the web, mashups are revolutionizing web development. They are everything enterprise integration is not - they’re cheap, far easier to create, more maintainable. In addition, they don’t require the technical skill sets; so theoretically the non-tech user can perform the complex task of building applications. This will potentially foster a new wave of innovation as the barrier to developing applications is dropped.

Last week, Salesforce.com, the high-profile seller of on-demand CRM solutions combined their customer management program with Google’s powerful online advertising services to create the Salesforce for Google AdWords application. Using the new program, Salesforce customers can create, manage, and measure search engine marketing campaigns from beginning to end.

Despite Gartner’s recent predictions there are still disconcerting opinions concerning the infrastructure, system and data security, as well as real tools for creating, deploying, and managing these enterprise mashups. Of course this situation will likely change as mashups and their tools mature. Still, businesses are beginning to recognize the power and cost-effective appeal. While mashups will never replace back-end IT assets in the enterprise, there is a world of innovative solutions, whether it makes your CRM software product more useful or is a yet to be created custom application.

Paul will be giving a presentation on Continuous Database Integration on Wednesday, September 14th at the Richmond Java User Group Meeting.

His presentation illustrates how performing continuous builds is essential for ensuring working software. Topics include how to leverage your CI system to rebuild your database and the associated test data when changes occur and demos for dropping and recreating a database along with its associated tables, inserting and modifying the test data. Examples will use the CruiseControl CI tool, Ant build tool and the MySQL database.

I’ll be speaking at the Mid-Atlantic Chapter of the IASA in Reston, VA on September 21st and will be giving a presentation on “How Code Quality Supports Architectural Quality”– for more information on the types of metrics I’ll be speaking about see my article entitled “In pursuit of code quality: Code quality for software architects.”

If you are in the Northern Virginia area, come on by!

Steve McConnell observed as early as 1993 that, “trying to improve software quality by increasing the amount of testing is like trying to lose weight by weighing yourself more often.” This quote from Code Complete is as true today as it was then.

I joke with my team that “testing” to improve product quality is like trying to make your car go faster by looking at your speedometer.

Why then, am I the CEO of a company that chooses “Test Early” as its motto?

Because testing – specifically, developer testing – is the most powerful way to assess the quality of code and introduce meaningful practices to improve code quality.

There remains an ongoing debate about the economic impact of software defects, the relevance of testing to address that economic impact and even the definition of “quality software”.

But, without testing – again, early developer testing – software developer teams are flying blind and relying on the implicit hunches of their most influential members.

Note that I used the phrase “most influential” rather than “best programmers.” Anecdotal evidence should assure the majority of us that these two groups are often not the same people.

My company focuses on quality of software during the manufacturing process. An analogous example would be to say if we were in the automobile industry, we would not examine a completed automobile, nor would we examine the plans for an automobile. We instead inspect and improve every step along the assembly line.

It is clearly true that a car with poor design plans will likely have poor quality – even if every step along the manufacturing process is executed perfectly. Also, a well-designed and built car could have little financial success if the final, completed product does not resonate emotionally with the car-buying public.

So, gaps in design and market assessment can have material impact on the quality of applications, independent from the effectiveness of the manufacturing process.

I believe, though, that the software industry’s biggest gap in competency actually lies in the manufacturing process itself.

Ask any VP of Engineering or Director of Software Development to provide metrics for the overall cyclomatic complexity of an application. Or the ratio of unit tests to complexity. Or the rank of modules by their dependency from other modules. Or the frequency and coverage of executing unit tests. Or the frequency and duration of a complete build of an application for final production.

My experience is that very few engineering teams give meaningful service to these critical measures of the software manufacturing process.

Does that mean that the code quality of these teams is poor? No. It just means that an assessment of code quality is based on the input of the most influential members of the team rather than independent, objective measures.

Another joke of mine is, “I’m not only the hair club president, I am also a client.”

I embraced this idea of improving software manufacturing by contracting a consulting firm a few years ago for my JNetDirect product business. I was so impressed with the way early software quality can transform the way developers engage in the creation of code that we acquired the company and launched Stelligent.

While testing does not make software better, it does give objective feedback regarding the quality of code that constitutes the product. Constructing meaningful tests that (1) become an artifact of the code repository and (2) are executed with every change to the product will provide real feedback to the engineers who author code.

Looking at your speedometer will not tell you why the car is moving slowly, but it might remind you that you have left the parking brake engaged before you start smelling the fumes from your brake pads.

As in any competitive endeavor, you cannot win unless you keep score. If improving software quality is your goal, you need the scoring mechanism of developer testing to gauge the effectiveness of your efforts.

Testing may not make better software, but it will let you know whether the software you are making is getting better or worse.

Without testing, you are playing the game with your eyes closed. You might score. Then again, you might be shooting at the wrong goal.

I created a three-minute movie that steps you through the process of downloading, installing, and configuring CruiseControl for a single project. Use this to get up and running with CruiseControl in no time!

I’ve written about three test categories (unit, component and system) for developer testing on a number of occasions- and I’ve even tried to draw a distinction between system and functional tests. Briefly, a system test verifies a software application from system end points, like web pages or web services- but they mimic the user or the end point protocol.

For example, you could write a system test for a web application with jWebUnit, which essentially mimics a browser by sending HTTP requests. Here is a jWebUnit test, written in TestNG, which logs into a web application:

package test.com.acme.web.cvg.ng;

import net.sourceforge.jwebunit.WebTester;

public class LoginTest  {
 private WebTester tester;

 /**
  * @testng.before-class = "true"
  */
 protected void init() throws Exception {
  this.tester = new WebTester();
  this.tester.getTestContext().
    setBaseUrl("http://acme.nfs.com:8080/cvg/");
 }

 /**
  * @testng.test groups = "system"
  */
 public void verifyLogIn() {
  this.tester.beginAt("/");
  this.tester.setFormElement("j_username", "lucy");
  this.tester.setFormElement("j_password", "lou");
  this.tester.submit();
  this.tester.assertTextPresent("Logged in as Lucy");
 }
}

This tests logs into an application and asserts that specialized text is present. While this may seem like a functional test, it isn’t because doesn’t act like a user- no browser is involved. This is as close as it gets, however, to a browser and accordingly, these tests are extremely valuable.

A functional test, as opposed to a system test, doesn’t mimic the end user- it is the end user. In essence, a functional test drives the application under test via a browser or even drives a GUI.

Also too, the functional test/system test line is usually divided by who can implement them. Almost always, system tests are somewhat clear box oriented- they are usually written in a development language and thus assume a fair amount of programming skills. Functional tests, on the other hand, are usually more friendly to author.

Selenium, for instance, is a functional testing tool, which provides a simple platform for verifying application functionality. Tests are authored in tabular format, much like that of the Fit framework. In fact, I think Selenium’s table format is easier to understand. Much like the xUnit paradigm, tests are organized into suites- through these suites, a test execution order can also be established.

For example, the following Selenium test is equivalent to the jWebUnit test from above.

The first row specifies a URL to open and the next two rows indicate form values to be filled out (j_username set to “lucy” and j_password set to “lou”). Next, a command is issued to hit the login button and then web page text (”Logged in as Lucy”) is verified.

Suites then can be created, which, in essence, map to use cases. For example, imagine a use case for user account creation. First, successfully login to the application and verify that worked. Next, create a new user and verify the user was created. Lastly, clean things up (i.e. delete the user so the test can be run again).

A Selenium suite simply points to tests- in this case, I have a suite that points to three logical tests:

Click on the image for a larger view

Selenium’s intuitive tabular format for declaring tests makes authoring functional tests a breeze. For more information, see Selenium’s documentation and an excellent article on IBM developerWorks.

Continuum is a tool that supports the practice of (automated) Continuous Integration (CI). In this entry, I discuss the basic setup and configuration of the tool.

Continuum has been available for about a year. It is a part of the Apache Maven project. It has built-in support for the Ant, Maven and command-line (shell) build tools. It is open source and freely available for download. Currently, Continuum fully supports five popular version control tools. They are: CVS, Subversion, Bazaar ,StarTeam, and Perforce. There is partial support for ClearCase, Visual Source Safe, and a file system.

To install, you download a file and run a command program. Continuum can be administered using a web-based application. “Out of the box”, like Luntbuild, all of the configuration and administration is performed using the web-based application.

After logging in, you can select one of the four types of projects: Ant, Maven, Maven2, and Shell.

Step One: Download, Installation, and Configuration

• Download the zip file
• Extract the zip; For example, I extracted it to C:\dev\tools which created the C:\dev\tools\continuum-1.0.3 directory

Installation and Configuration
I chose to install Continuum as a Windows service, so the following steps applied. Because I was already running a Tomcat server at port 8080, I modified the port node in the apps/continuum/conf/application.xml file, as shown below.

From the directory you downloaded Continuum:
• Select the bin directory and enter run from the command line
• Open a different command prompt and enter InstallService

This starts the Continuum service for Windows. Continuum uses a Jetty server under the hood. You may now access Continuum at http://localhost:[yourport]/continuum. When you first access it, you will see a page similar to this:

Step Two: Creating a project
Using Ant
I already had an Ant build script, so I chose to create an Ant project in Continuum. Upon selecting the Ant project, the page below was displayed. There is context-sensitive help embedded into the application. For instance, there are examples for SCM url format that Continuum requires. Furthermore, Continuum establishes a number of defaults for you. This includes the build tool location and a default scheduler - which checks for changes to your SCM every hour.

Step Three (Optional): Modifying the Scheduler
By default, Continuum sets up a schedule to check for changes to your version control repository every hour. Because I want to check for changes more often, I added a new schedule to check every five minutes as shown below. Then, I modified my project to use this new scheduler called CONTINUOUS. You will use the cron format to modify how often Continuum looks for chanes to your SCM

If you are using Windows, Continuum will always detect the build as being successful. The fix for this is described here. To quickly make the fix, you can replace your ant.bat file with this file.

Monitoring Builds
After logging in, you see the main page for Continuum. You may use this page to monitor your build status.

You can control access with Continuum using roles, based on capabilities using the Users Group Management menu item.

This was just an intro to setting up Continuum. For more information on installing, configuring, and administering Continuum, see the Getting Started Guide provided on its website.

LuntBuild is a tool that supports the practice of (automated) Continuous Integration (CI). In this entry, I discuss the basic setup and configuration of the tool.

Luntbuild has been available for a couple of years. It has built-in support for the Ant, Maven and command-line build tools. It is open source and freely available for download. Currently, Luntbuild supports eight popular version control tools such as CVS, Subversion, ClearCase and Perforce.

It is fairly simple to install and administer the tool via a web-based user interface. In fact, all of the configuration and administration is performed using this web-based application.

Luntbuild uses a tabbed interface. To configure, you step through the tabs to setup users, projects, schedules, builds, and VCS (Version Control System) adaptors.

Luntbuild supports Ant, Maven, Maven2, command-line and Rake build tools.

Step One: Download, Installation, and Configuration

1. Download the zip file
2. Extract the zip
3. Install the software

There are three ways to install the Luntbuild web application:

1. GUI, manually, and building from the source code.
2. Manually
3. Building from source code

For me, the simplest was using the GUI installer. To use, download the installer from here. From the directory you downloaded the installer, run the command:

java -jar luntbuild-xxx-installer.jar

A wizard installer application will step you through the installation.

Installation and Configuration

1. Leave the fields in the Configure the database and Configure LDAP authorization screens blank for now (unless you need to use either one, of course)
2. Make sure your application server (e.g. Tomcat) is started.
3. Use the default username/password established by Luntbuild

Once you’ve logged in successfully, you will see a page similar to this:

Step Two: Adding a User
In order to setup projects, schedule and builds, you need to create a user. Select the user tab:

Enter the username, full name, password and other fields as necessary. Select the Can Create Project checkbox. Optionally, you may select other feedback mechanisms such as instant messenger or a blog.

Step Three: Adding a Project
Once you’ve added a user, select the Project tab. You will seen a page like the figure below.

Enter the project name, description, which users will have which roles (admin, builder, viewer) on the project, and
notification methods selected for the project.

Step Four: VCS Adaptors
In setting up the project, you need to tell Luntbuild which version control system (VCS) you will be using. In the example below,
I chose Subversion. When you select a VCS from the drop down, different fields will display dynamically based on the particular
VCS options. Because, I chose Subversion, it require a Repository URL based. Furthermore, I may optionally add subdirectories for the
trunk, tags and branches.

There are eight VCS adaptors for Luntbuild. In creating a VCS adaptor,
you will need to add a module within the VCS.

Step Five: Establishing a Builder
There are five different builders that Luntbuild supports. They are Ant, Maven, Maven2, commandbuilder, and Rake.

Enter the required fields. Be sure, to enter the Ant batch/shell file. It defaults to a directory that is probably
not on your machine. The last field on this page is the Build success condition. This is how Luntbuild knows whether a build
was successful. I chose logContainsLine(”BUILD SUCCESSFUL”) for this field.

Step Six: Scheduling a Project to build

Monitoring Builds
After logging in, you see the main page for Luntbuild. You may use this page to monitor your build status.

You can choose the Refresh On option so that the information is always up to date.

This was just an intro to setting up Luntbuild. For more information on installing, configuring, and
administering Luntbuild, see the Quick Start
provided on its website.

The second Stelligent Early Quality Podcast, which focuses on Test Categorization, was recently published. Once again, disco pioneers Andrew Glover and Paul Duvall push the envelope of 70’s slang while having a technical conversation- it’s the hippest 5 minutes you’ll spend all week!