A Public Scratchpad

There’s a lot of misinformation out there regarding web application test tools, so I’ve decided to post a series of short articles comparing some of the open source options available here in Java-land, circa 2007. The first of these articles will focus on HtmlUnit and HttpUnit. Please take my criticism and praise with a grain of salt, as I’m a committer to the HtmlUnit project and thus probably biased. Nevertheless, I will do my best to be objective. I may even overcompensate in the other direction!

Confusion

The HtmlUnit and HttpUnit projects are often confused due to the similarity of their names. And the similarity doesn’t end there: they are both open source projects; they are both 100% Java frameworks, rather than drivers for native browsers like IE or Firefox; and they are both fairly mature projects.

This confusion is compounded by the fact that many test frameworks which once used HttpUnit under the covers have since switched to using HtmlUnit, mainly in order to benefit from its excellent JavaScript support. Examples include JWebUnit, whose FAQ briefly explains the switch, and Canoo WebTest, which switched in 2004 due to JavaScript support issues and an unresponsive development team [1].

HttpUnit

HttpUnit is the granddaddy web app testing framework. Started in the summer of 2000 by Russ Gold [2], it was the first project to focus on this niche area. The project has since stagnated somewhat, with nearly 40% of bugs remaining open, some of them nearly three years old. Its latest maintenance release is about a year and a half old.

The API is fairly low-level, modeling web interactions at something approaching the HTTP request and response level. The following is a slightly modified example from the HttpUnit Cookbook:

WebConversation wc = new WebConversation();
WebResponse resp = wc.getResponse("http://www.google.com/");
WebLink link = resp.getLinkWith("About Google");
link.click();
WebResponse resp2 = wc.getCurrentPage();

As you can see, things center around WebConversations, WebRequests and WebResponses. Unfortunately, any page with a decent amount of JavaScript is likely to break HttpUnit, and you can absolutely forget testing any pages which use third party JavaScript libraries.

Nevertheless, HttpUnit continues to generate 3,000 to 4,000 downloads per month. A good analogy, if I may be allowed a brief subjective comment, is that HttpUnit is to the web app testing world what Struts is to the web app framework world: there are many “better” options out there, but it just won’t go away!

HtmlUnit

HtmlUnit is itself a fairly old project, having been started by Mike Bowler in early 2002. Mike has since ceased active development, but the project currently boasts 3 or 4 active developers and a total of seven committers (whereas HttpUnit remains a one-man show). It averages about three releases per year, and has seen increased developer activity in the past six months or so, especially in the area of JavaScript support.

HtmlUnit’s API is a bit more high-level than HttpUnit’s, modeling web interaction in terms of the documents and interface elements which the user interacts with:

WebClient wc = new WebClient();
HtmlPage page = (HtmlPage) wc.getPage("http://www.google.com");
HtmlForm form = page.getFormByName("f");
HtmlSubmitInput button = (HtmlSubmitInput) form.getInputByName("btnG");
HtmlPage page2 = (HtmlPage) button.click();

As you can see, the code centers around WebClients, as well as pages, links, forms, buttons, etc. Pages with a modicum of custom JavaScript will probably work when tested with HtmlUnit. Unfortunately, pages which use third party libraries might or might not work when tested via HtmlUnit. As of the current version, Prototype, Script.aculo.us, DWR and jQuery are known to be supported fairly well, Dojo is a bit of an unknown, YUI is known to be unsupported, and GWT is known to work with fairly simple applications. Most of this compatibility has been achieved in the past two or three releases, so obviously things are fairly fluid.

Conclusion

If you’re using HttpUnit for legacy reasons, it’s a fairly solid package, but don’t expect to get much support when you need to report a bug or submit a patch for a new feature. If you’re starting a new project and are trying to decide between these two frameworks, HtmlUnit wins hands down. It has the features, the community and the momentum.

Of course, if you’re considering web application testing tools, you’re probably looking at more than just these two options. Canoo WebTest, TestMaker, JWebUnit, Selenium, WebDriver and JMeter are all likely to be on your list. Depending on your project budget and requirements, Squish and Mercury QTP may also be under consideration. If that’s the case, stay tuned, because I intend to post a series of web app testing framework comparisons in the coming months — all of them involving HtmlUnit, of course!

[1] It’s interesting to note that both Marc Guillemot and I (two HtmlUnit committers) began by using HttpUnit, submitting patches for missing features — but settled on HtmlUnit when the patches were not applied in a timely manner.

[2] The HttpUnit website states that Russ currently works for Oracle, developing the OC4J application server. Coincidentally, this is the production application server we’re using at my day job. Thanks, Russ!