HtmlUnit vs HttpUnit

There’s a lot of misinformation out there regarding web application test tools, so I’ve decided to post a series of short articles comparing some of the open source options available here in Java-land, circa 2007. The first of these articles will focus on HtmlUnit and HttpUnit. Please take my criticism and praise with a grain of salt, as I’m a committer to the HtmlUnit project and thus probably biased. Nevertheless, I will do my best to be objective. I may even overcompensate in the other direction!

Confusion

The HtmlUnit and HttpUnit projects are often confused due to the similarity of their names. And the similarity doesn’t end there: they are both open source projects; they are both 100% Java frameworks, rather than drivers for native browsers like IE or Firefox; and they are both fairly mature projects.

This confusion is compounded by the fact that many test frameworks which once used HttpUnit under the covers have since switched to using HtmlUnit, mainly in order to benefit from its excellent JavaScript support. Examples include JWebUnit, whose FAQ briefly explains the switch, and Canoo WebTest, which switched in 2004 due to JavaScript support issues and an unresponsive development team [1].

HttpUnit

HttpUnit is the granddaddy web app testing framework. Started in the summer of 2000 by Russ Gold [2], it was the first project to focus on this niche area. The project has since stagnated somewhat, with nearly 40% of bugs remaining open, some of them nearly three years old. Its latest maintenance release is about a year and a half old.

The API is fairly low-level, modeling web interactions at something approaching the HTTP request and response level. The following is a slightly modified example from the HttpUnit Cookbook:

WebConversation wc = new WebConversation();
WebResponse resp = wc.getResponse("http://www.google.com/");
WebLink link = resp.getLinkWith("About Google");
link.click();
WebResponse resp2 = wc.getCurrentPage();

As you can see, things center around WebConversations, WebRequests and WebResponses. Unfortunately, any page with a decent amount of JavaScript is likely to break HttpUnit, and you can absolutely forget testing any pages which use third party JavaScript libraries.

Nevertheless, HttpUnit continues to generate 3,000 to 4,000 downloads per month. A good analogy, if I may be allowed a brief subjective comment, is that HttpUnit is to the web app testing world what Struts is to the web app framework world: there are many “better” options out there, but it just won’t go away! ;-)

HtmlUnit

HtmlUnit is itself a fairly old project, having been started by Mike Bowler in early 2002. Mike has since ceased active development, but the project currently boasts 3 or 4 active developers and a total of seven committers (whereas HttpUnit remains a one-man show). It averages about three releases per year, and has seen increased developer activity in the past six months or so, especially in the area of JavaScript support.

HtmlUnit’s API is a bit more high-level than HttpUnit’s, modeling web interaction in terms of the documents and interface elements which the user interacts with:

WebClient wc = new WebClient();
HtmlPage page = (HtmlPage) wc.getPage("http://www.google.com");
HtmlForm form = page.getFormByName("f");
HtmlSubmitInput button = (HtmlSubmitInput) form.getInputByName("btnG");
HtmlPage page2 = (HtmlPage) button.click();

As you can see, the code centers around WebClients, as well as pages, links, forms, buttons, etc. Pages with a modicum of custom JavaScript will probably work when tested with HtmlUnit. Unfortunately, pages which use third party libraries might or might not work when tested via HtmlUnit. As of the current version, Prototype, Script.aculo.us, DWR and jQuery are known to be supported fairly well, Dojo is a bit of an unknown, YUI is known to be unsupported, and GWT is known to work with fairly simple applications. Most of this compatibility has been achieved in the past two or three releases, so obviously things are fairly fluid.

Conclusion

If you’re using HttpUnit for legacy reasons, it’s a fairly solid package, but don’t expect to get much support when you need to report a bug or submit a patch for a new feature. If you’re starting a new project and are trying to decide between these two frameworks, HtmlUnit wins hands down. It has the features, the community and the momentum.

Of course, if you’re considering web application testing tools, you’re probably looking at more than just these two options. Canoo WebTest, TestMaker, JWebUnit, Selenium, WebDriver and JMeter are all likely to be on your list. Depending on your project budget and requirements, Squish and Mercury QTP may also be under consideration. If that’s the case, stay tuned, because I intend to post a series of web app testing framework comparisons in the coming months — all of them involving HtmlUnit, of course!

[1] It’s interesting to note that both Marc Guillemot and I (two HtmlUnit committers) began by using HttpUnit, submitting patches for missing features — but settled on HtmlUnit when the patches were not applied in a timely manner.

[2] The HttpUnit website states that Russ currently works for Oracle, developing the OC4J application server. Coincidentally, this is the production application server we’re using at my day job. Thanks, Russ! :-)

About these ads

21 Comments

  1. October 11, 2007 at 4:32 am

    [...] Daniel Gredler blog post: HtmlUnit vs HttpUnit. [...]

  2. Simon R said,

    October 12, 2007 at 10:57 am

    We’re moving our tests to the new HtmlUnit, having had the same frustrations with HttpUnit – however I was frustrated that HtmlUnit didn’t provide an equivalent of HttpUnit’s in process testing capability (i.e. ServletUnit). I’d really rather not test my apps out-of-process if I can at all help it. So I’ve wired up HtmlUnit as the browser emulator and ServletUnit as the in process servlet runner – if anyone wants details, please let me know – nospamsimonrohrer@hotmail.com.

  3. October 15, 2007 at 7:30 pm

    [...] back through the pros and cons of JWebUnit as part of my research for the HtmlUnit vs Foo series of articles I’m writing. One of JWebUnit’s big draws is the set of easy-to-use assertion methods [...]

  4. December 2, 2007 at 3:42 pm

    [...] the history and other detailed information can be read about on this wonderful blog entry entitled HtmlUnit vs. HttpUnit (a must read!). That article is also very good to hear about support of external Javascript [...]

  5. January 14, 2008 at 7:30 pm

    [...] HtmlUnit vs HttpUnit « A Public Scratchpad – If you?re using HttpUnit for legacy reasons, it?s a fairly solid package, but don?t expect to get much support. If you?re starting a new project and are trying to decide between these two frameworks, HtmlUnit wins hands down. It has the features, Tags: , , ADA, advocacy, ajax, algorithms, apache, api, article, bigdecimal, browser, calendar, cms, code, collaboration, color, ComputerScience, criptaculous, curriculum, data, database, date, design, development, dhtml, ecm, education, effects, framework, gallery, geek, graphics, howto, html, htmlunit, httpunit, Humor, ie, infoq, interesting, java, javascript, JMeter, JWebUnit, layout, learning, library, lightbox, Linux, mashup, mathematics, microsoft, money, opensource, performance, php, presentation, programming, prototype, purifier, puzzles, scraping, scripting, scripts, security, Selenium, sxhtml, testing, tools, tweaks, ubuntu, web, Web2.0, webdesign, WebDriver, webservices, workflow, xml, xslt, xss [...]

  6. Senthil said,

    January 27, 2008 at 11:27 am

    Hi Daniel,

    I am a Tester, and we have decided to write automated functionals test using HtmlUnit. We are using eclipse and we have added HtmlUnit jar files to the libraries. We are testing a secure site. however, we are unable to invoke the application using :
    final WebClient webClient = new WebClient();
    final HtmlPage page = (HtmlPage) webClient.getPage(“https:/….com/”);
    assertEquals(“HtmlUnit – Welcome to HtmlUnit”, page.getTitleText());

    anyone who can help us.. ksenthil_77@yahoo.com

  7. Daniel Gredler said,

    January 27, 2008 at 2:27 pm

    Hi Senthil,

    Please post a description of your problem, including a stack trace (if you have one) to HtmlUnit’s user mailing list:

    http://sourceforge.net/mail/?group_id=47038

    or

    http://www.nabble.com/HtmlUnit—General-f2599.html

  8. Christian Hall said,

    February 6, 2008 at 3:00 pm

    Is there any plan to include what Simon suggests within HtmlUnit?

  9. Daniel Gredler said,

    February 6, 2008 at 11:16 pm

    Christian: Not at the moment. You may want to chime in on the dev list though, and see what others think: http://www.nabble.com/HtmlUnit—Dev-f2598.html

  10. Juuro said,

    March 18, 2008 at 9:16 am

    Hi!

    Thanks your article helped me very very! :)

    Best regards, Juuro

  11. WolfgangFahl said,

    April 1, 2008 at 12:50 pm

    Some of the statements here are out of date. Since mid 2007 I’m a committer on the httpunit project. So it’s not a one man show at this time.

    First I should say that I’ve been sharing your frustration about how httpunit was handled since mid 2004. I’ve been sticking to my own patched version until the middle of 2007 when I finally decided to ask Russell for committer
    rights which he kindly assigned to me.

    I’ve worked hard to get the patches of some 3 years into the project and I’m done with that. I’ve done some 200 tracker items in the last three days alone and version 1.7 and 2.0 are due soon. The javascript support of httpunit is being improved with a new DOM scripting engine. There are over 750 running test cases at this time.

    From my point of view it’s a pitty that there is a split of the developer resources around the unit testing issue of html pages. I don’t know what Russell’s opinion on this is.

    Feel free to contact me via the contact form on my webpage or the e-mail shown there.

  12. Marc Guillemot said,

    April 2, 2008 at 5:58 am

    Wolfgang,

    interesting to see that HttpUnit is not fully dead. On the other hand I don’t expect so much progress particularly in JS support, no matter how many energy you deploy: HttpUnit current JS support capabilities are approximatively comparable to what HtmlUnit could do in 2004. Looking back at its state in 2004 I can say that it represents only a small part of the current HtmlUnit JS support capabilities. 750 running tests is not really a high number: HtmlUnit has currently over 3300 running tests (some of them are large test suites of AJAX libraries) and this is surely not enough.

    I agree with you that a collaboration could allow to progress faster nevertheless I’ve already contacted you for a few days… but without response.

  13. WolfgangFahl said,

    May 13, 2008 at 5:40 am

    Thank you Daniel and Marc for getting in touch.
    I wonder what current httpunit users are saying. I did not try out htmlunit myself yet so I can’t state anything yet about how the two projects compare.
    Let’s see what happens during the upcoming months.

  14. May 22, 2008 at 8:26 pm

    Thanks for the post and great comments! In response to Wolfgang, I’m currently a BIG fan of httpunit. Its easy to add to and to build a framework around.
    My web app under test doesn’t use javascript at all so I am lucky in the regard. The one nicety is that httpunit lets me know the content type of various response pages (text/html or text/xml) and I can then do further processing there. However, httpunit can’t provide any verification features (eg getElementNames() or verifyTextPresent() ) on pages whose content type is ‘text/xml’. I wouldn’t expect it to either so I’m just using the Response.getText() and then doing some custom verification (XMLUnit and simple java regex).
    I don’t have much reason to move to a new framework; however, I have been reading about JWebUnit and looking at its 2.0 features (Selenium integration seems nice). It would be nice to have Selenium scripts using the same framework I’m using for my non GUI tests and JWebUnit has the potential to do that. JWebUnit is built on top of HtmlUnit, atleast thats what I’m reading, and so I may just make the switch over to JWebUnit.
    Is there anything that HTTPUnit can do that HtmlUnit / JWebUnit can’t do (I don’t use the ServletUnit functionality that Simon mentioned so thats a non issue for us. If Simon could elaborate on what he’s built, that would be nice too!)

  15. May 23, 2008 at 1:30 am

    I just noticed that HttpUnit 1.7 was released on May 20, 2008. I also noted that Wolfgang is a key committer and revived the HttpUnit project – kudos to you!

  16. May 23, 2008 at 7:02 am

    [...] his blog post “HtmlUnit vs HttpUnit“, Daniel Gredler wrote “HttpUnit is to the web app testing world what Struts is to the [...]

  17. Constantine Kulak said,

    January 22, 2009 at 3:10 am

    Thanks for the article! I tried to use HttpUnit in a DB2 8.1 stored procedure (don’t ask me why), and it just crashes its fenced mode. In 9.5 it works, but only if I put all supporting JAR files to DB2 JRE, because otherwise Xerces won’t work (it’s used internally by HttpUnit). I’ve spent two days struggling with that, and I wish I’d found your article before.

  18. March 7, 2009 at 9:02 pm

    Nice explanation.
    Thank you!

  19. Andrew said,

    December 9, 2009 at 4:20 pm

    Good comparison – I like what HtmlUnit offers, but I also like the simplicity of HttpUnit’s setup (1.7). HtmlUnit has way too many dependencies to manage on a host of machines simultaneously (regardless of what IBM wants us to believe about virtualization).

    I’d love to try out HtmlUnit, but could you clean up the dependency list? HttpUnit include basic classes of all necessary libraries and links to go get better ones if you want them.

    Also, documentation is huge – most of these sites/tools are aimed at developers testing their own code, but remember there are a lot of QA types that lack the developers understanding AND the developers environment. Too many things can be taken for granted in the documentation. (Just getting Junit up and running basic tests took 3 days – due to assumptions in the documentation) Remember your audience is never as smart as you think you are.

    Good article, albeit a bit old! :)

  20. Carl Karawani said,

    January 20, 2011 at 9:43 am

    Thank you very much for this article. Definitely lays out the high level differences between both frameworks. It’s also very interesting to see the different examples you mentioned.


Follow

Get every new post delivered to your Inbox.

%d bloggers like this: