SharePoint Dragons

Here’s a verdict of a list of tools we’ve tested to see if they’re useful in scenarios where you want to perform performance, load, and stress tests for SharePoint. Note that we’ve deliberately left out the Visual Studio Web Test tooling, as we’re finding lots of companies that don’t have the required licenses to use it. We do feel that every software author of each tool deserves a fair judgment based on facts and arguments, the only thing is, this verdict is not going to provide any of that. We’ve looked at a list of 100+ performance, load, and stress testing tools, and discarded any that were either commercial or not Windows oriented enough for our tastes. For the small list of remaining tools, we will just list them and tell you whether we liked it or not without getting into the in depth arguments why we reached a certain conclusion. Is that fair? Probably not. Here goes:

LoadUI at http://www.loadui.org/ is a great tool with at first, a brilliant user interface. Later on, the UI is still proves to be quite good, but not as great as the various Microsoft tooling we’re accustomed to.

WCAT (http://www.iis.net/downloads/community/2007/05/wcat-63-(x64)) is Microsoft’s community effort and is low on UI, but rich in functionality. If you’re planning on using WCAT, you’d better also download the WCAT Fiddler Extension (http://blogs.iis.net/thomad/archive/2010/05/11/using-the-wcat-fiddler-extension-for-web-server-performance-tests.aspx) which allows you to record WCAT scenarios using Fiddler.

StressStimulus (http://stresstimulus.stimulustechnology.com/) is a Fiddler add-in that looks promising but failed on our machine because it couldn’t install SQL Server Compact (we love Compact, btw). This was caused by the fact that we ran it on a 64 bit computer, and it tries to download the 32 bit version. You can solve it by downloading and installing SQL Server Compact yourself. To complicate things, the 64 bit version of Compact requires the 32 bit version of Compact (aaargh), but after that it works and you can install StressStimulus after all. That was the point when we found out that the free version of it was so limited that we had to forget about the tool.

We can forget about DieselTest (http://sourceforge.net/projects/dieseltest/), it’s as old and extinct as a dinosaur.

As far as D-ITG (the Distributed Internet Traffic Generator) goes (http://www.grid.unina.it/software/ITG/), we should have known better when we saw the animated NEW images and should have stopped there.

With the Fast Web Performance Test Tool (FWPTT at http://fwptt.sourceforge.net/), we pressed ‘Start Recording’ and that’s the last we ever heard back from the tool. That is probably caused by the fact that the tool might not believe in communicating with the end user.

The HttpRider tool (http://httprider.codeplex.com/) started out all right, but apparently isn’t able to handle authentication scenarios which renders it quite useless (unless of course, getting unauthorized HTTP responses is your thing).

After all, IxoraRMS (http://www.ixorarms.com/) turns out to be not Windows-oriented enough for our tastes. We want a setup or exe button, okay?

Load Tester Lite (http://www.webperformance.com/load-testing/free-load-tester-lite.html) would probably have been a decent tool, but since we’re still waiting for our confirmation mail after registration, we’re giving up.

Loadster (http://www.loadsterperformance.com/) throws HTTP 401 Unauthorized when hitting SharePoint. There must be better ways than this one.

OpenLoad (http://www.opendemand.com/openload/) is a command line tool that allows to specify a single URL and simulates 5 clients. Doesn’t like authentication either. It does work if you’re exactly looking for this type of functionality, but really, if this is what we can get we prefer to write our own tool instead.

OpenSTA (http://opensta.org/) begins by setting our expectations below zero by requiring a system restart. Really, for this type of tool? It’s just not fair. After that, we find out that by default it expects the IE5 browser and complains that it can’t find it (no kidding?) and again we’ve had enough.

Ostinato (http://code.google.com/p/ostinato/) looks kinda cool but managed to wreck our web server in such a way that we decided to restore a previous snapshot. Having said that though, it did look like a powerful tool that requires more studying.

TestMaker (http://www.pushtotest.com/) asked us to download a version that was 600+ MB in size. Apparently we asked for the OS version of the load and stress testing tool, such sizes are ridiculous, so we’re not that desperate yet.

XLT (http://www.xceptance.com/products/xlt/what-is-xlt.html) finds us on a not so good day. We’re not really interested in playing a game of hide and seek to find out how to start this tool.

At this point, we remember why Open Source isn’t the end all solution for all of world’s software problems. The first three tools are our top 3 choices, but certainly, when considering this type of tool, it might not be a bad idea to try to get some budget for it and check out a commercial vendor. We’d say that except for WCAT and maybe LoadUI, there’s no free lunch and the free functionality is so limited that it’s nothing more than a teaser to buy commercial versions.