If you’ve ever been on a highway during a peak hour, you’ve definitely experienced being stuck in an endless traffic jam. In addition to wasting time, you were probably annoyed, disappointed, and angry, just like hundreds of other drivers around you. But you didn’t have any alternatives; staying in the jam and moving at snail speed was your only option at that moment.
Why did it happen? Obviously, when the highway was constructed it wasn’t supposed to serve so many vehicles at the same time.
When building an application, you also make some predictions about the possible number of active users. But what if you underestimate? That’s great because it means that your product attracts so many users. But if the system isn’t ready for high loads, your customers literally get stuck in a traffic jam when using your application. In the case of a highway, people don’t have any other option, but when annoyed and disappointed by an app, they’ll quickly find an alternative!
This doesn’t mean that you have to limit the number of users. It would be much more efficient to ensure that the system is ready for possible high loads. How do you do this and which load testing tools should you use? This is exactly the topic of today’s article.
App performance and why it’s important
During app development, the product goes through multiple stages of testing. There are unit tests and system tests, integration and regression tests. These tests all ensure that each component of the application does what it is supposed to do and that the whole system works seamlessly.
However, it often happens that a product performs well in a testing environment, but fails when a critical number of users are online. This occurs due to lack of proper performance (load) testing.
Reasons to use load testing:
- Prove that the system is able to cope with the planned (or unplanned) load.
- Check system performance in different situations (such as varying intensities of loading, time of app usage, etc.).
- Check app behavior upon critical load.
Forewarned is forearmed. With information about possible system responses to different loads, you have a chance to improve your app’s infrastructure before it goes live. As a result, you prevent thousands of users from having potentially negative experiences with your application.
With manual testing, it is only possible to test the system response to a single user (or just a few users). But it is generally problematic to involve thousands of test engineers at the same time to ensure that the system won’t fail at peak usage. For this purpose, specialized software that imitates a large number of users was invented.
JMeter is a load testing tool distributed under the Apache license and managed by the developer community. JMeter is open source, free to use, and written in Java, making it one of the most popular load testing tools for small teams with limited budgets and startups. At the same time, it is powerful enough to satisfy the needs of big businesses and enterprises. Along with being accessible at no cost, Apache JMeter has many other advantages, proving the love of professional QA engineers.
Apache JMeter has a modular structure. This means that the core functionality is extended with free plugins. In addition, it is possible to create your own plugin for specific cases.
Advantages of Apache JMeter:
- Platform independent
Being a fully Java-based desktop application, it can run on various platforms with any Java-compatible OS (Linux, Windows, Mac OS X).
- No extra software
There is no need to install specific software. All you have to do is copy and run the executed file on your computer.
- Friendly GUI
JMeter can be run in GUI Mode, Server Mode, and Command Line Mode. All three are intuitive and don’t have a steep learning curve.
- Testing results visualization
Testing results can be displayed in tables, diagrams, graphs, log files, results tree, etc.
- Multiflow framework
JMeter provides parallel and simultaneous selection of various functions of a separate group of streams.
- High testing scalability
Test engineers have the ability to write their own test cases. In addition, the tool supports custom visualization plugins to provide additional testing scalability.
- User behavior modeling
JMeter enables you to model the behavior of multiple users with parallel flows, which helps to properly load the tested application.
- Supports multiple protocols
The tool supports all major protocols out-of-the-box, including HTTP, SOAP, LDAP, JDBC, JMS, FTP, etc.
- Script testing
JMeter can be integrated with Selenium and Bean Shell for more efficient and automated load testing.
- Integrated real-time support
Apache Tomcat collectors allow you to monitor all the data in real time.
- Simple integration with free libraries
The load testing tool supports free libraries like Jenkins, Maven, and Gradle.
Disadvantages of Apache JMeter
- High resources consumption
JMeter uses many resources, especially memory. When the computer runs out of memory, the tool can’t provide the expected load.
- Predictable user behavior imitation
The tool doesn’t support random request intervals and independent think times for different URLs.
- Java version dependency
When running a distributed test, it is important to use the same versions of Java.
Uses of Apache JMeter
Initially, Jmeter was developed for web and FTP app testing. Nowadays, it is also used for functional testing, database testing, etc. Modular architecture allows the abilities of JMeter to be extended to fit even exotic testing scenarios. Further, if none of the existing plugins meet the needs of the given application, it is also possible to write your own plugin using the provided API. JMeter enables distributed testing when several computers load one system simultaneously.
The load testing tool allows you to audit the performance of static and dynamic applications. It is able to simulate heavy loads on a server/group of servers, network or a given object. As a result, it is possible to analyze how an application performs under different loads.
Gatling is a powerful tool for load and performance testing. The framework is written in the Scala programming language and distributed under the Apache license. Gatling is an open-source tool, although the company also offers an enterprise solution called Gatling FrontLine. After a 6-week demo, the price for Gatling FrontLine varies between €5,999 and €14,999 per year.
However, the open source version of the product is powerful enough to provide extensive performance testing to all types of web applications.
Advantages of Gatling
- Cross-platform load testing tool
Gatling works with any operating system and any browser.
- Graphic reports
The tool supports graphic reports containing essential information for analysis.
- Usage of third-party cloud services
Testing scenarios can be executed in various testing clouds, which is essential for distributed testing.
- Third-party integration through plugins
The tool can be easily integrated with Jenkins, Gradle or Maven by using corresponding plugins.
- Lightweight solution
Gatling uses around 50% less memory than JMeter, which enables more exact results with system testing.
Disadvantages of Gatling
- Unable to distribute loading between multiple machines
The system randomly distributes loading between several machines and the developer has no choice in this process. As a result, sometimes a single machine performs most of the request, and other machines are barely involved.
- Horizontal scalability is impossible
Resulting from the previous drawback, when adding another machine to the testing process, you will be unable to unload the initial one.
Gatling is easy to start working with. The scripting language Scala with Gatling’s own DSL is understandable even for newbies, making it easier to review executed tests. The load testing tool enables you to discover bottlenecks in web app usage and minimize their impact.
ApacheBench is a tool that audits the performance of HTTP servers by generating a number of concurrent requests to the given URL. It was originally created to test the Apache HTTP server, but it is powerful enough to discover weaknesses in any web server.
Advantages of ApacheBench
- Easy to learn and to use
Apache Bench is easy to start using since it doesn’t require knowledge of a specific programming language.
- No need to download and install other software
The tool is already installed on computers using Linux and Mac OS. Windows users can access it as a part of the Apache package.
- Adjustability of requests
It is possible to define the general number of requests, the number of simultaneously executed requests, and their time limits.
Disadvantages of ApacheBench
- Limited functionality
It is possible to test web server performance, however different loading scenarios are not supported.
- Comprehensive performance testing is impossible
The tool is unable to imitate user loading and test app performance based on infrastructure or development.
- Lack of support for HTTPS
Although HTTPS has already become a standard, ApacheBench is still unable to test this type of protocol.
ApacheBench is a very basic tool for performance testing and doesn’t cover the needs of a comprehensive load testing process. However, it is highly popular as a simple and lightweight load testing tool that can reveal some performance problems without installing any third-party software.
LoadRunner is a product of Hewlett Packard that was recently acquired by Micro Focus. This tool can simulate thousands of users simultaneously, allowing you to discover weaknesses in app performance.
LoadRunner isn’t an open-source solution and charges a small fee for each virtual user per day. However, the product offers many benefits, and here are some of them:
Advantages of LoadRunner
- Interactive modeling of user activity
It is possible to simultaneously make multiple virtual users perform different actions in the tested applications;
- Wide set of tools support the application
Virtual User Generator (VuGen), Controller, Agent Machines, and LoadRunner Analysis allow you to set up, manage and record application tests and analyze their results.
- Optimal usage of load generator
This enables you to find out how the active use of the service by one group of users affects the response of the application for the rest of the users.
- Increased efficiency
LoadRunner can quickly and precisely define the main cause of performance issues for each application.
Disadvantages of LoadRunner
- Charges a fee for every virtual user day
LoadRunner won’t likely be a solution for projects with limited budgets since some of the already reviewed load testing tools offer comparable functionality for free.
- Console limitations
LoadRunner cannot simultaneously run several scenarios from the command line.
- Requires constant supervision
The tool requires constant technical support during test case execution.
- A heavyweight tool
Installing LoadRunner will require over 3 Gb of free space on your hard drive.
- Limited functionality for Linux and Mac OS
The Controller and VUGen work for Windows only.
LoadRunner allows you to audit website performance upon a large number of concurrent requests. This will help you understand the app usage of one group of users affects the way other users interact with the application.
App performance is often neglected during the development and testing process. However, what works for one user won’t necessarily work in the same way when thousands or millions of users are on the app simultaneously.
You should implement load testing to understand limitations of your infrastructure, to discover bottlenecks in app usage and try to maintain optimal user experience despite the number of parallel requests.
Your application is a highway. People buy trendy and expensive cars to have the freedom of moving anywhere anytime. If they get stuck in a traffic jam on your highway once, they may never return to it again. So make sure you construct a highway that is able to fit all potential drivers without sacrificing their speed.