If you're a big company with a platform that regularly supports tons of users, you have to know how incremental traffic impacts the performance of your systems. Maybe you're getting ready for a seasonal surge in traffic like Black Friday or Cyber Monday, and don't want to lose out on serious revenue in case a critical process fails.
Even startups can become a victim of their own success, if they receive an unexpected mention on TechCrunch or HackerNews and their sites collapse under the influx of new visitors.
Whether you're already supporting lots of traffic, gearing up for a seasonal spike, or become a viral success, don't get TechCrunched! Load test your APIs with real-world loads to validate their performance before it's too late.
On the Postman Engineering blog, we've already learned how to get started with test automation. Once you establish a process for functional and integration testing, the next step is to look at performance testing to gain a deeper understanding about the stability, reliability, and availability of your platform.
For example, how does your system behave when a large number of users are accessing your servers? How long does it take for a user to receive a response? As more users access your platform, does a response take significantly longer or does a request timeout? Is this effect magnified at certain times of the day or week?
Load testing falls under a broader umbrella of performance testing. During load testing, you will subject an API to a workload that approaches and possibly exceeds the limits of its specifications.
Sometimes load testing is completed along with stress testing, to determine the system's ability to handle a heavier than normal workload. Sometimes it can include endurance testing, to determine the system's ability to handle a specified work load for a prolonged period of time.
One unbelievably common approach is to throw your code into production, and be ready to roll it back if it doesn't scale well. Other developers might rely on server logs to find out when their servers are crashing.
If you're still reading, you probably want to be proactive and achieve a higher level of confidence about the stability, reliability, and availability of your platform.
Maybe you know exactly when you'll be featured on Shark Tank, or maybe you're totally unprepared for any kind of viral traffic. A single appearance on the front page of the internet can result in an unintentional distributed denial of service (DDOS) attack. Your servers might not be able to handle all of these simultaneous requests, and you'll piss off your users and lose out on revenue.
There's many ways to optimize and balance the load that hits your servers, validate user inputs, and cache static content. You should definitely do all of that in addition to load testing.
Load testing can be done as a regular part of your Continuous Integration process. At these early stages of the development lifecycle, you can identify bottlenecks and improve the stability of your application before it is released to production. It's best to catch issues and errors in these early stages when the cost of debugging is the lowest.
Load testing can also be done on an exploratory basis to validate hypotheses and gain a deeper understanding of your systems. Load testing for exploration is done similarly to regular stage-gate load testing, except on an ad hoc basis with higher than average loads. By increasing your load incrementally, you can determine the maximum throughput of your API that functions within your desired tolerances.
For example, imagine that you increase the number of active users by X% and see no significant degradation in performance. Assuming you have stable usage, you can rest assured that your current infrastructure supports next quarter's growth which is forecasted more modestly than the X% increase that you tested.
Take another example. Imagine that you run the same test, but instead you see significant degradation of a key performance metric. You can use this data to inform the acceptable bounds of a service-level agreement (SLA) for teams that rely on your services. If your system doesn't adequately support current or forecasted activity, this data can also drive the discussion for future investments in infrastructure and compute resources.
With frequent load testing, either as part of a regular process or on an ad hoc basis, we understand our architecture's performance characteristics deeper. This allows us to make performance improvements on our existing features and new implementations.
Different organizations task different members of their team for load testing. In many cases, it's the developer or tester that is responsible for measuring key performance indicators. In other cases, it's the DevOps engineer, SRE, or someone in charge of application performance monitoring.
Although CIOs and engineering managers may not be involved with the implementation of load testing, they are ultimately responsible for infrastructure and downtime costs.
Across organizations, there's a variety of functions that bear the responsibility for performance testing. Across the development community, there's a variety of opinions about testing infrastructure and best practices.
While a number of teams "do load testing", what that translates to in practice is often varied. For some teams, they're firebombing requests on non-production APIs to see what happens. That's one way of getting started, but there's still more that can be done.
Load testing can be expensive, and typically requires approval for the additional spend. No matter who is responsible for actually running a load test, demonstrating that you're doing something valuable with the results of that test is important for the whole team.
These are some common measures of API performance. Depending on your service level objectives and requirements, you may want to measure other indicators too.
In the next section, let's take a look at one way to do load testing with Postman.
During a performance test, all requests are sent from your computer where you are running the Postman desktop app. The performance test runner displays performance metrics in real time so you can observe how your API performs under load.
The development community has widely varying opinions about the performance testing of HTTP APIs. The tooling for load testing is also a fragmented landscape. There's a ton of out-of-the-box, roll-your-own, and any combination of the two solutions. To top it off, load testing is frequently handled by different functions within an organization.
With all of these different approaches, it's hard to identify any single strategy as the gold standard for load testing.
If you're just getting started, I recommend starting with your monitoring. Observing your production traffic and corresponding performance metrics may be enough to spark some questions about your system's behavior. Then, run a base case and start incrementally adding some load. Especially with complex systems that are dependent on other internal and external services, you'll be able to spot the bottlenecks and issues more clearly.