Open source tools cheaper than specialist, right? Not always the case…

By Luke Barfield | August 9, 2017 | Categories: Blog | Tags: , , ,

When I first started out in testing over 10 years ago, there were few performance test tools around and certainly no open source. Since then, the test tool industry has boomed and open-source tools have become genuine contenders when selecting a tool to carry out performance testing. There is this assumption that a lack of license costs makes open source the cheapest option – but is this really the case?

At Certeco, we conduct independent tools evaluation to help clients select the right test tool for them. One thing that has become obvious when we do these tool evaluations is that whilst licencing and maintenance can be costly, it does pay to consider the whole picture before weighing up open source vs specialist. It may surprise you that in some situations, open-source tools are by no means the cheapest option.

There are a few common misconceptions about open source and commercial tools. For a start, they often think that open-source tools do the same thing as commercial tools, or that the missing functionality is not critical. They also often think that licence and support costs are the only significant cost associated with implementing the tools, and that the costs to implement and use are the same regardless of the tool. This isn’t the case. It’s also often assumed that commercial tools are expensive and involve a large capital investment. And people often think that as these tools are so in demand, skills to implement them must be in abundant supply – if in-house personnel are to be the main users, there are plenty of resources available to learn how to use them. In our experience, this isn’t the case.

Take one of our clients, a large UK university. We now provide them with a test service with an on-demand performance test, but the university had previously conducted performance testing with another supplier. This supplier had implemented a popular open source tool, which the client was keen to continue using, as it was open-source and they assumed easy on the budget. This open source tool is great for web technology, but there were a number of constraints, such as you need to be quite a technical performance tester to get the best out of it. The reporting was also quite basic and it needed to be supplemented with another tool like Excel or a backend database.

I was introduced to Neotys about 12 months ago and was impressed with what NeoLoad was capable of. I decided to try NeoLoad against a couple of the client’s applications and I immediately found that I could create initial scripts in minutes rather than hours and completed scripts in hours not days.

This increase in productivity is down to the additional features that NeoLoad provides, including:
Neoload comparison table
I believe – for this client – scripts that took our performance testers two to three days to correlate and debug could have been completed in less than a day had we been using NeoLoad.

When we do a tools evaluation, we calculate the total cost of ownership (TCO) for the organisation, which needs to factor in resourcing costs. For performance testing, the activities directly impacted by the tools are test script design, test execution, reporting and analysis. If the additional features in a commercial tool like NeoLoad allow the performance engineers to be more efficient, then there is a potential saving in effort. The makeup of the performance test team (i.e. internal team or third party) will define the financial impact of the effort saving.

Finally, when selecting a tool, we look at the longer-term picture – will our client be able to continue using the tool themselves or will they require consultancy each time? For open-source tools, there is a huge online community with many resources to learn how to use them. But this isn’t everyone’s preferred training method; it requires the right kind of tenacity to pick your way through finding the best bits that work in a specific context. As a comparison, the NeoLoad training course only takes three days to complete and you get certified expert accreditation. I also became a certified trainer, so we can train clients in using the tool too.

We understand that each client’s environment has its own specific demands and requirements – a tool that is right for one organisation, won’t be right for another. An open source tool could work perfectly in one scenario, whereas a tool like NeoLoad would be much better in another. We know the importance of conducting an independent evaluation to ensure that the most appropriate tool is selected that will deliver the right result for the client at the right price. There is no doubt that for performance testing, NeoLoad will be one of tools we will use in our evaluations – and based on my experience to date, it will be a pretty tough tool to beat.

About Certeco

Certeco is a UK based business technology change consultancy. Offering specialist consultancy services to large corporates in industries like banking, insurance, public sector, education, media and postal, Certeco helps clients with business restructuring, regulatory initiatives and technological change. Certeco helps clients through the lifecycle of change, using specialist capabilities in project and programme management, business analysis, quality assurance and testing. The firm has offices in London, Ipswich, Liverpool and Glasgow, a workforce of 200 and annual sales of over £20m.