Elastic Path Commerce Sizing Guide
A high-performing cloud implementation of Elastic Path Commerce depends on a right-sized cloud configuration. Use this sizing guide to assess your business requirements, compare the requirements to the baseline performance metrics, and select the best configuration size for Amazon Web Services (AWS). As your requirements change over time, you can revisit this guide for guidance on how to maintain performance.
The baseline performance metrics are based on Elastic Path Commerce version 8.2.x deployed on an AWS Kubernetes Cluster. We do not update performance results for every release; however, performance testing continues on the current release. If the numbers change significantly, we will update this sizing guide.
Methodology
The Elastic Path Commerce platform is designed to ensure maximum performance and scalability. The Elastic Path performance team conducts rigorous testing to establish baselines for performance capacity and scalability.
The following factors can impact the performance of your Elastic Path Commerce solution:
- Correct sizing of your infrastructure
- Code extensions and customizations
- Synchronous integrations with external systems
- Extreme dimensions in catalog size or transactional data
Over time, the demands placed on the systems that support your Elastic Path Commerce solution might change. These changes can result in significant departures from the established baselines for Elastic Path Commerce.
Elastic Path recommends a three-stage performance evaluation methodology:
- Establish the initial sizing based on requirements
- Refine through testing
- Continue to monitor performance, perform maintenance, and plan for future capacity
Establish the initial sizing
Based on your business requirements, establish the initial sizing. Select a cloud configuration size that meets your requirements for the production environment and deploy your Elastic Path Commerce solution.
You can choose from the following four reference configurations:
- Extra small
- Small
- Medium
- Large
Compare these sizes against your current requirements to find a suitable match. Ensure that you adjust your configuration based on certain factors, such as busiest shopping days or redundancy needs. For example, if the small configuration size barely meets the performance requirements during peak periods, augment the configuration to ensure acceptable performance. Increasing the configuration size is suitable in case of the event that one application server fails.
Refinement through testing
Based on the performance testing during implementation, you can understand the performance impact of customizations and resolve any issues before releasing the solution to production.
Continual capacity planning and maintenance in production
After you release the system into production, ensure that you continually monitor the system. Monitoring the system determines the accuracy of the sizing and includes validation of both the system performance and system usage, such as the inputs to the initial sizing. It also enables you to monitor trends in usage, including increases, which may necessitate future increases in hardware capacity.
Reference Architecture
Cortex is a lightweight RESTful API leveraging the power of Commerce Engine to provide eCommerce services for multiple platforms.
To design and run a successful deployment, you need to understand the tiers and components within an Elastic Path Commerce system. For more information, see the Elastic Path Commerce Reference Deployment Architecture.
Elastic Path focuses on testing against Cortex APIs on the CloudOps for Kubernetes deployment architecture. Cortex APIs are the interface between the end-user clients and the Elastic Path Commerce platform. Cortex APIs process all the requests from end-user clients, such as mobile, HTML5, IoT, and more.
Completing the Initial Sizing
Translate your business requirements into one or more possible metrics. Compare your metrics against the baseline performance metrics for each reference configuration size and select the configuration that best meets your requirements. Finally, adjust the configuration based on other unique and qualitative criteria.
Business-focused metrics
You can use the following business-focused metrics for sizing:
- User Sessions per hour
- Conversions per hour
- Page views per hour
To maximize the accuracy of the sizing, estimate a value for each of the metrics. The sizing model establishes relationships between metrics, which means that the definition of one metric might suggest the values for the other metrics.
User sessions per hour
User sessions per hour is the number of visitors that you expect to visit your storefront within a one-hour period. For a new deployment, you can determine the number based on your business requirements. For an existing deployment, you can determine the number by analyzing web server logs or analytics data.
Conversions per hour
The conversion rate is the percentage of visitors to the storefront who complete a purchase. Conversions per hour is an important sizing metric because it is the easiest to determine when evaluating a system that has not been built yet. Businesses are aware of the sales targets and forecasts, and these estimates can be translated into an hourly conversion rate.
The conversions per hour test is based on a static conversion rate of 5%. For example, the sizing model assumes approximately 5% of visitors convert to purchasers, such as conversions/hour = user sessions/hour multiplied by 5%. If you expect conversion rates to differ significantly, adjust the estimated percentage of conversions and contact Elastic Path Support.
Page views per hour
Page views per hour determines capacity based on the rate at which full pages are accessed, that is, requests, not including embedded objects, such as images and CSS files. For an existing deployment, you can determine the page views per hour by analyzing web server logs or analytics data.
This metric is sometimes difficult to derive from business requirements. The page views per hour metric is the best metric to use when an existing reference system is available for analysis and the new Elastic Path Commerce solution closely matches the existing system in terms of user workflow. Attempting to estimate sizing based on page views per hour without a reference deployment may be misleading.
Assess your business-focused metrics
You can use the following sample table to record and assess your business-focused metrics:
| Metrics | Normal | Peak |
|---|---|---|
| Conversions per hour | ||
| User Sessions per hour | ||
| Page views per hour |
The table contains two columns, one for the normal load and the other for the peak load. In general, it is best to size for peak load. Sizing for normal loads may be sufficient if you have constrained resources.
You must also consider growth over the lifetime of the system. When you derive input metrics from an existing system, adjust your numbers to include capacity for growth.
Give consideration to high-availability needs, such as requirements of sizing your system to meet peak capacity, even when two application servers are failing. It can also be sufficient to support normal loads in the event of failures.
Assumptions
Consider how the business-focused metrics flow through the system and end up interacting with Cortex APIs. For example, a detailed product page might necessitate five HTTPS calls to fetch the following resources:
- Product
- Pricing
- Availability
- Description
- Product attributes
You can use the Cortex Zoom feature to fetch data from the same resources in a single HTTPS call reducing those five calls down to one.
note
If you are fetching data from multiple resources, you will still require multiple Zoom calls. For example, a product page with a mini cart in the header would require two calls.
Cortex API operates in a stateless manner. When capacity testing, the concept of sessions does not apply. Instead, Cortex API is sized by a throughput value of transactions or API calls per second, and particular workflows define the percentage of conversion throughput. To size the system with the business-focused metrics, such as user sessions per hour, you must make assumptions about the number of API calls per Cortex Zoom, Cortex Zooms per page view, and page views per user session.
Performance testing simulates interaction with the Reference Storefront, but with the functionality delivered by Cortex API and real-world data.
Based on the performance tests, Elastic Path has derived the following numbers:
| Assumption | Value |
|---|---|
| Average Resources Retrieved per Cortex Zoom | 10.36 |
| Average Cortex Zooms per Page View | 2 |
| Average Page Views per User Session | 4.65 |
| Conversion Rate | 5% |
Sizing Your Deployment
You can compare your values for the business-focused metrics with the reference configurations described in this section. The sizing recommendations for each reference configuration come from using the following combination of Commerce, CloudOps, and IaaS:
- CloudOps for Kubernetes 2.8.x
- Elastic Path Commerce 8.2.x using the Snapitup example data set
- Amazon Aurora MySQL database 5.7 v2.07
- Amazon Web Services IaaS
Compare the business-focused metrics that you have derived to those listed in the following table. Find the reference configuration that most closely matches your requirements. Ensure to include the following considerations:
- Peak load versus normal load
- Expected lifetime growth
- High availability in your choice
Performance is ultimately limited by database size. Choosing a database that can support your business-focused metrics is critical. It is also important that Cortex has sufficient replicas to support the database size you choose. Each of the following reference configurations provides both database size and a recommended number of maximum Cortex replicas. Performance varies depending on the catalog size and workload. It’s important to treat these reference configurations as a starting point. Run tests to confirm that the chosen reference configuration meets your business-focused metrics with your given catalog and workload.
| Reference Configuration | DB Size | Cortex Replicas | Zoom Transactions per second | Conversions per hour | User Sessions per hour | Pageviews per hour |
|---|---|---|---|---|---|---|
| xSmall | r5.large | 3 | 286 | 5,527 | 110,538 | 514,000 |
| Small | r5.xlarge | 6 | 596 | 11,536 | 230,710 | 1,072,800 |
| Medium | r5.2xlarge | 12 | 1230 | 23,806 | 476,129 | 2,214,000 |
| Large | r5.4xlarge | 21 | 2086 | 40,374 | 807,484 | 3,754,800 |
note
All tests were run on c5.2xLarge nodes. These nodes are the default instance type used in CloudOps for Kubernetes deployments.
important
This capacity planning guide acts as a scalability starting point. If your capacity requirements exceed the values listed in the previous table, contact Elastic Path Support. Elastic Path Commerce has been deployed and tested to much higher levels of scale with larger-sized databases and supports Horizontal Database Scaling for our largest customers.
Performance Testing Methodology
The reference configuration tables show the results of a thorough performance test called Reference Storefront Benchmark.
Reference Storefront Benchmark specifications
The Reference Storefront Benchmark consistently measures the general performance of the Cortex API framework implemented as REST Web Services. The test simulates the exact transactions executed by the React Progressive Web Application Reference Storefront framework. The Reference Storefront Benchmark aims to approximate a realistic workload from a server perspective. It measures responsiveness for a single user, non-concurrent scenario. The test repeats these measurements at increasing user or concurrency levels until the system becomes saturated, which measures the scalability of the system.
You can use the Reference Storefront Benchmark to identify and reproduce performance problems and poorly-performing functionality. Repeating the test with new versions of the application helps you to identify regression issues. You can also use the test to evaluate the impact of configuration changes. You can use the results of the test to estimate the capacity of a production deployment.
System under test for AWS
The environment under test is based on the default Kubernetes deployment in CloudOps for Kubernetes. For more information, see CloudOps for Kubernetes Architecture Overview.
For each environment size, the only change to a server instance was at the database level, where an appropriately sized Relational Database Service (RDS) database was chosen. The EP Cortex Pods are auto-scaled for each environment and are limited to the max-replicas listed in the associated reference configurations:
3for xSmall6for Small12for Medium21for Large
Auto-Scale
While Elastic Path Commerce supports auto-scale of EP Cortex pods, the pods do not scale instantly. A new Cortex pod can be created and placed into traffic in about 3 minutes. For most circumstances, this allows the cluster to dynamically scale to meet the changing needs of shopper traffic.
In cases where sudden large traffic spikes occur, it’s important to understand the limitations of our out-of-the-box auto-scale configuration. This is the maximum rate at which Elastic Path Commerce can scale up additional Cortex pods. Under these maximum scale rate conditions, testing has shown that a CloudOps for Kubernetes-Elastic Path Commerce deployment doubles the number of running Cortex Pods every three to six minutes, doubling transaction capacity. This rate of scale holds true when the database capacity is available and when the max-replicas limit for Cortex pods has not been reached yet. The time needed to double capacity varies from three minutes in cases where sufficient Kubernetes nodes already exist to six minutes in cases where new nodes need to be added via the Kubernetes node auto-scaler to support the new Cortex pods.
For example, if an Elastic Path Commerce deployment suddenly sees a 10X increase in traffic and needs to scale from two to 21 Cortex pods, it may take nearly 20 minutes for the system to scale up all 19 additional Cortex pods. This massive scale operation would occur via the following sequential steps:
- Scale from 2 Cortex pods to 4 Cortex pods: Time = 3 minutes
- Scale from 4 Cortex pods to 8 Cortex pods: Time = 3 minutes
- Scale from 8 Cortex pods to 16 Cortex pods: Time = 6 minutes
- Scale from 16 Cortex pods to 21 Cortex pods: Time = 6 minutes
The total time for the scale operation is about 18 minutes. During this period, especially during the first 10 minutes while the system is massively overloaded, response times would be negatively impacted.
To avoid these potential negative performance impacts we recommend manually pre-scaling Cortex pods in anticipation of a large traffic surge, if possible. We also recommend testing these traffic spike scenarios to better understand how any customizations might impact auto-scale capabilities.
User-system interaction model
The user-system test is built based on the following user-system interaction model.
User Transactions
A user transaction is a discrete interaction between the client and the system, where the client makes a request and waits for a response. For this system, a user transaction is one RESTful HTTP request invoking one or more Cortex resources. It requests from more than one resource which can be grouped by the Cortex Zoom functionality. The request grants the client permission to populate the page to display. For example, in a single HTTP request, the transaction UT_RetrieveRandomProduct allows the client to access a single item. It retrieves the item along with its definition, assets, pricing, availability, and a link to the add-to-cart button.
note
In this benchmark, a second transaction, UT_GetDefaultCart, supplements all user transactions. UT_GetDefaultCart simulates a client loading a page, and also the page’s mini-cart, displaying the total items in the user’s cart.
Use cases
A use case is a task performed by a user that initiates a series of user transactions. For example, the use case UC_Browse initiates the following series of user transactions:
UT_RetrieveNavUT_RetrieveNavHierarchyUT_RetrieveProductsInNavHierarchyUT_RetrieveRandomProduct
User Sessions
A user session is a continuous collection of user-system interactions that consists of one or more use cases.
User Session types
The following user session types comprise this test:
- Browse category
- Offer search
- Add to cart
- Checkout and purchase
Browse category
The BrowseCategory user session simulates an unauthenticated user or client browsing through categories and products.
| User transaction | Zoom request (URL) |
|---|---|
UT_RetrieveNav | ${application}?zoom=navigations:element,navigations:element:child |
UT_RetrieveNavHierarchy | ${application}?zoom=navigations:element,navigations:element:child |
UT_RetrieveProductsInNavHierarchy | ${application}/navigations/${scope}/lookups/form?zoom=items,items:element,items:element:code,items:element:availability,items:element:definition,items:element:definition:assets:element,items:element:price,items:element:rate,element,element:availability,element:definition,element:definition:assets:element,element:price,element:rate,element:code&followlocation |
UT_RetrieveRandomProduct | ${application}/items/${scope}/lookups/form?zoom=availability,addtocartform,addtowishlistform,price,rate,definition,definition:assets:element,definition:options:element,definition:options:element:value,definition:options:element:selector:choice,definition:options:element:selector:chosen,definition:options:element:selector:choice:description,definition:options:element:selector:chosen:description,definition:options:element:selector:choice:selector,definition:options:element:selector:chosen:selector,definition:options:element:selector:choice:selectaction,definition:options:element:selector:chosen:selectaction,recommendations,recommendations:crosssell,recommendations:recommendation,recommendations:replacement,recommendations:upsell,recommendations:warranty,recommendations:crosssell:element:code,recommendations:recommendation:element:code,recommendations:replacement:element:code,recommendations:upsell:element:code,recommendations:warranty:element:code,recommendations:crosssell:element:definition,recommendations:recommendation:element:definition,recommendations:replacement:element:definition,recommendations:upsell:element:definition,recommendations:warranty:element:definition,recommendations:crosssell:element:price,recommendations:recommendation:element:price,recommendations:replacement:element:price,recommendations:upsell:element:price,recommendations:warranty:element:price,recommendations:crosssell:element:availability,recommendations:recommendation:element:availability,recommendations:replacement:element:availability,recommendations:upsell:element:availability,recommendations:warranty:element:availability,code&followlocation |
Offer search
The OfferSearch user session simulates an unauthenticated user or client searching and loading products.
| User transaction | Zoom request (URL) |
|---|---|
UT_GetFacetedSearchForm | ${application}?zoom=searches:keywordsearchform,searches:offersearchform |
UT_SubmitFacetedSearchForm | ${application}/offersearches/${scope}/offers/form?zoom=element,element:availability,element:definition,element:definition:assets:element,element:price,element:rate,element:code,element,element:availability,element:definition,element:definition:assets:element,element:pricerange,element:items,element:items:element,element:items:element:availability,element:items:element:definition,element:items:element:definition:assets:element,element:items:element:price,element:items:element:rate,element:items:element:code,facets,facets:element,facets:element:facetselector,facets:element:facetselector:choice:description,facets:element:facetselector:choice:selector,facets:element:facetselector:choice:selectaction,facets:element:facetselector:chosen:description,facets:element:facetselector:chosen:selector,facets:element:facetselector:chosen:selectaction&followlocation |
UT_SelectFacet{1,2} | ${FacetsSelectActionHref}?followlocation&zoom=offersearchresult |
UT_FacetFilteredSearch | ${OfferSearchResultHref}?zoom=element,element:availability,element:definition,element:definition:assets:element,element:price,element:rate,element:code,element,element:availability,element:definition,element:definition:assets:element,element:pricerange,element:items,element:items:element,element:items:element:availability,element:items:element:definition,element:items:element:definition:assets:element,element:items:element:price,element:items:element:rate,element:items:element:code,facets,facets:element,facets:element:facetselector,facets:element:facetselector:choice:description,facets:element:facetselector:choice:selector,facets:element:facetselector:choice:selectaction,facets:element:facetselector:chosen:description,facets:element:facetselector:chosen:selector,facets:element:facetselector:chosen:selectaction |
UT_RetrieveRandomProduct | ${application}/items/${scope}/lookups/form?zoom=availability,addtocartform,addtowishlistform,price,rate,definition,definition:assets:element,definition:options:element,definition:options:element:value,definition:options:element:selector:choice,definition:options:element:selector:chosen,definition:options:element:selector:choice:description,definition:options:element:selector:chosen:description,definition:options:element:selector:choice:selector,definition:options:element:selector:chosen:selector,definition:options:element:selector:choice:selectaction,definition:options:element:selector:chosen:selectaction,recommendations,recommendations:crosssell,recommendations:recommendation,recommendations:replacement,recommendations:upsell,recommendations:warranty,recommendations:crosssell:element:code,recommendations:recommendation:element:code,recommendations:replacement:element:code,recommendations:upsell:element:code,recommendations:warranty:element:code,recommendations:crosssell:element:definition,recommendations:recommendation:element:definition,recommendations:replacement:element:definition,recommendations:upsell:element:definition,recommendations:warranty:element:definition,recommendations:crosssell:element:price,recommendations:recommendation:element:price,recommendations:replacement:element:price,recommendations:upsell:element:price,recommendations:warranty:element:price,recommendations:crosssell:element:availability,recommendations:recommendation:element:availability,recommendations:replacement:element:availability,recommendations:upsell:element:availability,recommendations:warranty:element:availability,code&followlocation |
Add to cart
The AddToCart user session simulates an unauthenticated user or client browsing products and adding one to cart, then viewing the cart.
| User transaction | Zoom request (URL) |
|---|---|
UC_BrowseCategory | Note: Transactions in the UC_BrowseCategory use case above are executed. |
UT_AddToCart | ${AddToCartActionHref}?followlocation |
UT_ViewFullCart | ${application}?zoom=defaultcart,defaultcart:appliedpromotions:element,defaultcart:discount,defaultcart:lineitems:element,defaultcart:lineitems:element:appliedpromotions,defaultcart:lineitems:element:appliedpromotions:element,defaultcart:lineitems:element:availability,defaultcart:lineitems:element:item,defaultcart:lineitems:element:item:code,defaultcart:lineitems:element:item:definition,defaultcart:lineitems:element:item:definition:details,defaultcart:lineitems:element:item:definition:options:element,defaultcart:lineitems:element:item:definition:options:element:selector:choice,defaultcart:lineitems:element:item:definition:options:element:selector:choice:description,defaultcart:lineitems:element:item:definition:options:element:selector:chosen,defaultcart:lineitems:element:item:definition:options:element:selector:chosen:description,defaultcart:lineitems:element:item:definition:options:element:value,defaultcart:lineitems:element:price,defaultcart:lineitems:element:total,defaultcart:order:couponinfo:coupon,defaultcart:order:couponinfo:couponform,defaultcart:total |
Checkout and purchase
The Checkout user session simulates a new user or client performing the following transactions:
- Adding to cart
- Creating a new profile
- Updating address
- Adding shipping and payment information
- Viewing the order
- Completing a purchase
- Logging out
| User transaction | Zoom request (URL) |
|---|---|
UC_AddToCart | Note: Transactions in UC_AddToCart use case above are executed. |
UT_GetRegistrationForm | ${application}/registrations/\${scope}/newaccount/form |
UT_SubmitRegistrationForm | ${registrationFormActionHref} |
UT_AuthenticationWithCreatedProfile | ${application} |
UT_ViewProfile | ${application}?zoom=defaultprofile,defaultprofile:purchases,defaultprofile:purchases:element,defaultprofile:addresses,defaultprofile:addresses:element,defaultprofile:addresses:billingaddresses:default,defaultprofile:paymentmethods,defaultprofile:paymentmethods:element |
UT_GetCountries | ${application}/geographies/${scope}/countries?zoom=element,element:regions,element:regions:element,countries:element,countries:element:regions,countries:element:regions:element |
UT_GetAddressForm | ${application}/addresses/${scope}/form |
UT_CreateDefaultAddress | ${createProfileLocationURL}/addresses |
UT_ViewProfile | ${application}?zoom=defaultprofile,defaultprofile:purchases,defaultprofile:purchases:element,defaultprofile:addresses,defaultprofile:addresses:element,defaultprofile:addresses:billingaddresses:default,defaultprofile:paymentmethods,defaultprofile:paymentmethods:element |
UT_GetPaymentTokenForm | ${application}?zoom=defaultcart:order:paymentmethodinfo:paymenttokenform,defaultprofile:paymentmethods:paymenttokenform |
UT_CreatePaymentToken | ${application}/paymenttokens/${scope} |
UT_ViewProfile | ${application}?zoom=defaultprofile,defaultprofile:purchases,defaultprofile:purchases:element,defaultprofile:addresses,defaultprofile:addresses:element,defaultprofile:addresses:billingaddresses:default,defaultprofile:paymentmethods,defaultprofile:paymentmethods:element |
UT_ViewFullCart | ${application}?zoom=defaultcart,defaultcart:appliedpromotions:element,defaultcart:discount,defaultcart:lineitems:element,defaultcart:lineitems:element:appliedpromotions,defaultcart:lineitems:element:appliedpromotions:element,defaultcart:lineitems:element:availability,defaultcart:lineitems:element:item,defaultcart:lineitems:element:item:code,defaultcart:lineitems:element:item:definition,defaultcart:lineitems:element:item:definition:details,defaultcart:lineitems:element:item:definition:options:element,defaultcart:lineitems:element:item:definition:options:element:selector:choice,defaultcart:lineitems:element:item:definition:options:element:selector:choice:description,defaultcart:lineitems:element:item:definition:options:element:selector:chosen,defaultcart:lineitems:element:item:definition:options:element:selector:chosen:description,defaultcart:lineitems:element:item:definition:options:element:value,defaultcart:lineitems:element:price,defaultcart:lineitems:element:total,defaultcart:order:couponinfo:coupon,defaultcart:order:couponinfo:couponform,defaultcart:total |
UT_ProceedToCheckout | ${application}?zoom=defaultcart,defaultcart:appliedpromotions:element,defaultcart:discount,defaultcart:order,defaultcart:order:billingaddressinfo:billingaddress,defaultcart:order:billingaddressinfo:selector:choice,defaultcart:order:billingaddressinfo:selector:choice:description,defaultcart:order:couponinfo:coupon,defaultcart:order:couponinfo:couponform,defaultcart:order:deliveries:element:destinationinfo:destination,defaultcart:order:deliveries:element:destinationinfo:selector:choice,defaultcart:order:deliveries:element:destinationinfo:selector:choice:description,defaultcart:order:deliveries:element:shippingoptioninfo:selector:choice,defaultcart:order:deliveries:element:shippingoptioninfo:selector:choice:description,defaultcart:order:deliveries:element:shippingoptioninfo:shippingoption,defaultcart:order:paymentmethodinfo:paymentmethod,defaultcart:order:paymentmethodinfo:selector:choice,defaultcart:order:paymentmethodinfo:selector:choice:description,defaultcart:order:tax,defaultcart:order:total,defaultcart:total |
UT_CompleteOrder | ${application}?zoom=defaultcart,defaultcart:appliedpromotions:element,defaultcart:discount,defaultcart:lineitems:element,defaultcart:lineitems:element:item,defaultcart:lineitems:element:item:code,defaultcart:lineitems:element:item:definition,defaultcart:lineitems:element:item:definition:options:element,defaultcart:lineitems:element:item:definition:options:element:value,defaultcart:lineitems:element:item:price,defaultcart:lineitems:element:total,defaultcart:order,defaultcart:order:billingaddressinfo:billingaddress,defaultcart:order:couponinfo:coupon,defaultcart:order:couponinfo:couponform,defaultcart:order:deliveries:element:destinationinfo:destination,defaultcart:order:deliveries:element:shippingoptioninfo:shippingoption,defaultcart:order:paymentmethodinfo:paymentmethod,defaultcart:order:purchaseform,defaultcart:order:tax,defaultcart:order:total,defaultcart:total |
UT_CompletePurchase | ${SubmitOrderHref}?followlocation&zoom=appliedpromotions:element,billingaddress,discount,lineitems:element,lineitems:element:options:element,lineitems:element:options:element:value,paymentmeans:element,shipments:element:destination,shipments:element:shippingoption |
Testing model
The testing model test uses a steady state model. For a specific period of time, each virtual user continuously simulates a series of user sessions. The time period is broken up into the following phases:
- Warm-up phase (no measurements)
- Measurement phase
The test repeats with increasing virtual user and concurrency levels, and each data point has measurements recorded.
The virtual users work continuously. There is no allowance for think time between user transactions.
User Session distribution
The following table shows the user session distribution:
| User Session | Representation |
|---|---|
| Browse | 40% |
| Search Offer | 45% |
| Add-to-Cart | 10% |
| Purchase | 5% |
Concurrency levels and simulation times
The range concurrency levels must be adjusted based on the configuration under test. The test must involve a single virtual user test at the beginning and end of the run. It should increase concurrency levels to the point where the system begins to perform unacceptably and does not meet the success criteria.
The warm-up phase and measurement phase must be adjusted based on the test configuration. The times for each phase must be long enough that each user session type is measured at least 10 times.
The following table is a sample of the expected test configuration:
| Virtual Users | Warm-up phase (minutes) | Measurement phase (minutes) | Total cumulative time (minutes) |
|---|---|---|---|
| 1 | 1 | 5 | 6 |
| 5 | 1 | 5 | 12 |
| 15 | 1 | 5 | 18 |
| 30 | 1 | 5 | 24 |
| 45 | 1 | 5 | 30 |
| 60 | 1 | 5 | 36 |
| 80 | 1 | 5 | 42 |
| 100 | 1 | 5 | 48 |
Key metrics
Overall Apdex
For each execution, the Apdex calculates all measured user transactions. The T value key metric is the smallest Apdex value measured for any specific user transaction. All user transactions use a value of T = 500 milliseconds, yielding an F of 2000 milliseconds.
Minimum Apdex
For each execution, the Apdex calculates each user transactions independently, using a T value as defined in the Overall Apdex. The metric indicates how bad the user experience was for the worst user transaction.
Transactional throughput
The transaction throughput is the average number of user transactions per second that the system is able to perform over a measurement period. To measure this transaction, divide the number of measured user transactions by the measurement time in seconds.
Average response time
The average response time for all measured user transactions.
Maximum response time
The longest response time for a measured user transaction in seconds.
Sessions throughput per hour
The average rate of user session throughput that the system was able to deliver, converted into an hourly metric. You can find the throughput per hour by using the following formula: number of user sessions completed during the measurement time divided by the measurement time (in minutes), multiplied by 60 minutes.
User errors
The number of errors the virtual end user encounters and reports for each execution.
Success criteria
A test execution is deemed to be successful when the following criteria are met:
- Overall Apdex rating = Good: If the rating for a given data point is Good or better, then the system is deemed to have successfully supported that level of load, subject to other criteria. If the rating for a given data point is Fair or lower, then the system cannot support that load level
- Minimum Apdex >= Fair: If the rating for any user transaction falls above Fair, such as Good, the system is deemed to have successfully supported that level of load. If the rating for any user transaction falls below Fair, such as Poor or Unacceptable, the system cannot support that load level
- User errors < 1 %: To be successful, less than 1% of measured user transactions can result in errors
Test results
Review all available data to ensure that the results are valid. The following situations can invalidate test results:
- Load generators are underpowered and become the bottleneck
- A scripting error causes certain user transactions to not include all HTTP requests, reporting better results