Elastic Path Commerce Sizing Guide
A high-performing cloud implementation of Elastic Path Commerce depends on a right-sized cloud configuration. Use this sizing guide to assess your business requirements, compare the requirements to the baseline performance metrics, and select the best configuration size in either Amazon Web Services (AWS) or Microsoft Azure. As your requirements change over time, you can revisit this guide for guidance on how to maintain performance.
The baseline performance metrics are based on Elastic Path Commerce version 7.5.1 deployed on AWS and on Microsoft Azure. Performance results are not necessarily updated for every release. Performance testing continues on the current release. If the numbers change significantly, the sizing guide will be updated.
Methodology
The Elastic Path Commerce platform is designed to ensure maximum performance and scalability. The Elastic Path performance team conducts rigorous testing to establish baselines for performance capacity and scalability.
The following factors can impact the performance of your Elastic Path Commerce solution:
- Code extensions and customizations
- Synchronous integrations with external systems
- Extreme dimensions in catalog size or transactional data
- IaaS (Infrastructure as a Service) provider
Over time, the demands placed on the systems that support your Elastic Path Commerce solution might change. These changes can result in significant departures from the established baselines for Elastic Path Commerce.
Elastic Path recommends a three-stage performance evaluation methodology:
- Establish the initial sizing based on requirements
- Refine through testing
- Continue to monitor performance, perform maintenance, and plan for future capacity
Establish the initial sizing based on requirements
Based on your business requirements, establish the initial sizing. Select a cloud configuration size that meets your requirements for the production environment and deploy your Elastic Path Commerce solution.
There are four reference configurations: small, medium, large, and xlarge. These sizes can be compared against your requirements to find a suitable match. Adjust your configuration based on factors, such as your busiest shopping days or redundancy needs. For example, if the small configuration just barely meets your performance requirements during peak periods, you may want to augment the configuration to ensure sufficient performance in the event that one application server fails.
Refinement through testing
Based on the performance testing during implementation, you can understand the performance impact of customizations and resolve any issues before releasing the solution to production.
Continual capacity planning and maintenance in production
After you release the system into production, ensure that you continually monitor the system. Monitoring the system determines the accuracy of the sizing and includes validation of both the system performance and system usage, such as the inputs to the initial sizing. It also enables you to monitor trends in usage, including increases, which may necessitate future increases in hardware capacity.
Reference Architecture
Cortex is a lightweight RESTful API leveraging the power of Commerce Engine to provide eCommerce services for multiple platforms.
To design and run a successful deployment, you need to understand the tiers and components within an Elastic Path Commerce system. For more information, see the Elastic Path Commerce Reference Deployment Architecture.
To improve time to market, use an Elastic Path CloudOps option to deploy your Commerce solution and enable team development on either Amazon Web Services (AWS) or Microsoft Azure:
Elastic Path focuses on testing against Cortex APIs on the AWS and Azure clouds. Cortex APIs are the interface between the end-user clients and the Elastic Path Commerce platform. Cortex APIs process all the requests from end-user clients, such as mobile, HTML5, IoT, and more.
Completing the Initial Sizing
Translate your business requirements into one or more possible metrics. Compare your metrics against the baseline performance metrics for each reference configuration size and select the configuration that best meets your requirements. Finally, adjust the configuration based on other unique and qualitative criteria.
Business-focused metrics
You can use the following business-focused metrics for sizing:
- Sessions per hour
- Conversions per hour
- Page views per hour
To maximize the accuracy of the sizing, estimate a value for each of the metrics. The sizing model establishes relationships between metrics, which means that the definition of one metric might suggest the values for the other metrics.
Sessions per hour
Sessions per hour is the number of visitors that you expect to visit your storefront within a one-hour period. For a new deployment, you can determine the number based on your business requirements. For an existing deployment, you can determine the number by analyzing web server logs or analytics data.
Conversions per hour
The conversion rate is the percentage of visitors to the storefront who complete a purchase. Conversions per hour is an important sizing metric because it is the easiest to determine when evaluating a system that has not been built yet. Businesses are aware of the sales targets and forecasts, and these estimates can be translated into an hourly conversion rate.
The conversions per hour test is based on a static conversion rate of 5%. For example, the sizing model assumes approximately 5% of visitors convert to purchasers, such as conversions/hour = sessions/hour multiplied by 5%. If you expect conversion rates to differ significantly, adjust the estimated percentage of conversions and contact Elastic Path Support.
Page views per hour
Page views per hour determines capacity based on the rate at which full pages are accessed, that is, requests, not including embedded objects, such as images and CSS files. For an existing deployment, you can determine the page views per hour by analyzing web server logs or analytics data.
This metric is sometimes difficult to derive from business requirements. The page views per hour metric is the best metric to use when an existing reference system is available for analysis and the new Elastic Path Commerce solution closely matches the existing system in terms of user workflow. Attempting to estimate sizing based on page views per hour without a reference deployment may be misleading.
Assess your business-focused metrics
You can use the following sample table to record and assess your business-focused metrics:
| Metrics | Normal | Peak |
|---|---|---|
| Conversions per hour | ||
| Sessions per hour | ||
| Page views per hour |
The table contains two columns, one for the normal load and the other for the peak load. In general, it is best to size for a peak load. Sizing for normal loads may be sufficient if you have constrained resources.
You must also consider growth over the lifetime of the system. When you derive input metrics from an existing system, adjust your numbers to include capacity for growth.
Give consideration to high-availability needs, such as requirements of sizing your system to meet peak capacity, even when two application servers are failing. It can also be sufficient to support normal loads in the event of failures.
Assumptions
Consider how the business-focused metrics flow through the system and end up interacting with Cortex APIs. For example, a detailed product page on a mobile device in HTML5 can make five HTTPS calls resulting in four additional Cortex API calls to fetch the following resources:
- Product
- Pricing
- Availability
- Description
- Product attributes
You can use the Cortex Zoom feature to fetch data from the same resources in a single HTTP call. If you are calling multiple resources, you still would require multiple Zoom calls, such as a product page with a mini cart in the header.
Cortex API operates in a stateless manner. Therefore, when capacity testing, the concept of sessions does not apply. Instead, Cortex API is sized by a throughput value of transactions or API calls per second. Particular workflows define the percentage of conversion throughput. To size the system with the business-focused metrics, such as sessions per hour, you must make some assumptions about the number of API calls per Cortex Zoom, Cortex Zooms per page view, and page views per session.
Performance testing simulates interaction with the Reference Storefront, but with the functionality delivered by Cortex API and real-world data.
Based on the performance tests, Elastic Path has derived the following numbers:
| Assumption | Value |
|---|---|
| Average Resources Retrieved per Cortex Zoom | 10.36 |
| Average Cortex Zooms per Page View | 2 |
| Average Page Views per Session | 4.65 |
| Conversion Rate | 5% |
Size Your System on Amazon Web Services (AWS)
You can compare your values for the business-focused metrics with the reference configurations described in this section. The sizing recommendations for each reference configuration come from using the following combination of Commerce, CloudOps, and IaaS:
- CloudOps for AWS 3.x
- Elastic Path Commerce 7.5.1 using the Snapitup example data set
- Amazon Web Services IaaS
- Amazon Aurora MySQL database
Compare the business-focused metrics that you have derived to those listed for the following configurations. Find the one that most closely matches your requirements. Ensure to include considerations of peak versus normal, expected lifetime growth, and high availability in your choice.
AWS small environment
AWS small environment configuration
| Elastic Path component | Infrastructure |
|---|---|
| Elastic Path Cortex Nodes | 3 x 4-core vCPU, 8 GB RAM (3 x c5.xlarge) |
| Elastic Path Integration Node | 1 x 4-core vCPU, 8 GB RAM (1 x c5.xlarge) |
| Elastic Path Admin Node | 1 x 4-core vCPU, 8 GB RAM (1 x c5.xlarge) |
| Elastic Path CM Node | 1 x 4-core vCPU, 8 GB RAM (1 x c5.xlarge) |
| Database Node | 1 x 2-core vCPU, 16 GB RAM (1 x db.r5.large) |
AWS small environment figures
| Key sizing metrics | Tomcat / Aurora |
|---|---|
| Max Zoom Transactions per second | 327 |
| Max Conversions per hour | 6,334 |
| Max Sessions per hour | 126,677 |
| Max Pageviews per hour | 589,050 |
AWS medium environment
AWS medium environment configuration
| Elastic Path component | Infrastructure |
|---|---|
| Elastic Path Cortex Nodes | 6 x 4-core vCPU, 8 GB RAM (6 x c5.xlarge) |
| Elastic Path Integration Node | 1 x 4-core vCPU, 8 GB RAM (1 x c5.xlarge) |
| Elastic Path Admin Node | 1 x 4-core vCPU, 8 GB RAM (1 x c5.xlarge) |
| Elastic Path CM Node | 1 x 4-core vCPU, 8 GB RAM (1 x c5.xlarge) |
| Database Node | 1 x 4-core vCPU, 32 GB RAM (1 x db.r5.xlarge) |
AWS medium environment figures
| Key sizing metrics | Tomcat / Aurora |
|---|---|
| Max Zoom transactions per second | 687 |
| Max conversions per hour | 13,304 |
| Max sessions per hour | 266,079 |
| Max page views per hour | 1,237,266 |
AWS large environment
AWS large environment configuration
| Elastic Path Component | Infrastructure |
|---|---|
| Elastic Path Cortex Nodes | 10 x 4-core vCPU, 8 GB RAM (10 x c5.xlarge) |
| Elastic Path Integration Node | 1 x 4-core vCPU, 8 GB RAM (1 x c5.xlarge) |
| Elastic Path Admin Node | 1 x 4-core vCPU, 8 GB RAM (1 x c5.xlarge) |
| Elastic Path CM Node | 1 x 4-core vCPU, 8 GB RAM (1 x c5.xlarge) |
| Database Node | 1 x 8-core vCPU, 64 GB RAM (1 x db.r5.2xlarge) |
AWS large environment figures
| Key sizing metrics | Tomcat / Aurora |
|---|---|
| Max Zoom transactions per second | 1,164 |
| Max conversions per hour | 22,533 |
| Max sessions per hour | 450,670 |
| Max page views per hour | 2,095,614 |
AWS xlarge environment
AWS xlarge environment configuration
| Elastic Path Component | Infrastructure |
|---|---|
| Elastic Path Cortex Nodes | 18 x 4-core vCPU, 8 GB RAM (18 x c5.xlarge) |
| Elastic Path Integration Node | 1 x 4-core vCPU, 8 GB RAM (1 x c5.xlarge) |
| Elastic Path Admin Node | 1 x 4-core vCPU, 8 GB RAM (1 x c5.xlarge) |
| Elastic Path CM Node | 1 x 4-core vCPU, 8 GB RAM (1 x c5.xlarge) |
| Database Node | 1 x 16-core vCPU, 128 GB RAM (1 x db.r5.4xlarge) |
AWS xlarge environment figures
| Key sizing metrics | Tomcat / Aurora |
|---|---|
| Max Zoom transactions per second | 1,846 |
| Max conversions per hour | 35,738 |
| Max sessions per hour | 714,762 |
| Max page views per hour | 3,323,646 |
Size Your System on Microsoft Azure
You can compare your values for the business-focused metrics with the reference configurations described in this section. The sizing recommendations for each reference configuration come from using the following combination of Commerce, CloudOps, and IaaS:
- CloudOps for Azure 1.2
- Elastic Path Commerce 7.5.1 using Snapitup example data set
- Amazon Web Services IaaS
- Azure Database for MySql
Compare the business-focused metrics that you have derived to those listed for the following configurations. Find the one that most closely matches your requirements. Ensure to include considerations of peak versus normal, expected lifetime growth, and high availability in your choice.
Microsoft Azure small environment configuration
| Elastic Path component | Infrastructure |
|---|---|
| Kubernetes Nodes | 2 x 8-core vCPU, 16 GB RAM (2 x Standard_F8s_v2) |
| Database Node | 1 x 2-core vCPU, 10.2 GB RAM (1 x Gen 5 - 2 VCore) |
| Elastic Path Cortex Pods | 2 |
| Elastic Path Search Slave Pods | 2 |
Microsoft Azure small environment figures
| Key sizing metrics | Tomcat / Azure DB for MySQL |
|---|---|
| Max Zoom transactions per second | 109 |
| Max conversions per hour | 2,119 |
| Max sessions per hour | 42,372 |
| Max page views per hour | 197,028 |
Microsoft Azure medium environment
Microsoft Azure medium environment configuration
| Elastic Path component | Infrastructure |
|---|---|
| Kubernetes Nodes | 3 x 8-core vCPU, 16 GB RAM (4 x Standard_F8s_v2) |
| Database Node | 1 x 4-core vCPU, 20.4 GB RAM (1 x Gen 5 - 4 VCore) |
| Elastic Path Cortex Pods | 3 |
| Elastic Path Search Slave Pods | 2 |
Microsoft Azure medium environment figures
| Key sizing metrics | Tomcat / Azure DB for MySQL |
|---|---|
| Max Zoom transactions per second | 244 |
| Max conversions per hour | 4,724 |
| Max sessions per hour | 94,475 |
| Max page views per hour | 439,308 |
Microsoft Azure large environment
Microsoft Azure large environment configuration
| Elastic Path component | Infrastructure |
|---|---|
| Kubernetes Nodes | 6 x 8-core vCPU, 16 GB RAM (6 x Standard_F8s_v2) |
| Database Node | 1 x 8-core vCPU, 40.8 GB RAM (1 x Gen 5 - 8 VCore) |
| Elastic Path Cortex Pods | 6 |
| Elastic Path Search Slave Pods | 2 |
Microsoft Azure large environment figures
| Key sizing metrics | Tomcat / Azure DB for MySQL |
|---|---|
| Max Zoom transactions per second | 412 |
| Max conversions per hour | 7,976 |
| Max sessions per hour | 159,519 |
| Max page views per hour | 741,762 |
Microsoft Azure xlarge environment
Microsoft Azure xlarge environment configuration
| Elastic Path component | Infrastructure |
|---|---|
| Kubernetes Nodes | 10 x 8-core vCPU, 16 GB RAM (10 x Standard_F8s_v2) |
| Database Node | 1 x 16-core vCPU, 81.6 GB RAM (1 x Gen 5 - 16 VCore) |
| Elastic Path Cortex Pods | 10 |
| Elastic Path Search Slave Pods | 2 |
Microsoft Azure xlarge environment figures
| Key sizing metrics | Tomcat / Azure DB for MySQL |
|---|---|
| Max Zoom transactions per second | 762 |
| Max conversions per hour | 14,758 |
| Max sessions per hour | 295,165 |
| Max page views per hour | 1,372,518 |
Performance Testing Methodology
The reference configuration tables show the results of a thorough performance test called Reference Storefront Benchmark.
Reference Storefront Benchmark specifications
The Reference Storefront Benchmark consistently measures the general performance of the Cortex API framework implemented as REST Web Services. The test stimulates the exact transactions executed by the React Progressive Web Application Reference Storefront framework. The Reference Storefront Benchmark aims to approximate a realistic workload from a server perspective. It measures responsiveness for a single user, non-concurrent scenario. The test repeats these measurements at increasing user or concurrency levels until the system becomes saturated, which measures the scalability of the system.
You can use the Reference Storefront Benchmark to identify and reproduce performance problems and poorly-performing functionality. Repeating the test with new versions of the application helps you to identify regression issues. You can also use the test to evaluate the impact of configuration changes. You can use the results of the test to estimate the capacity of a production deployment.
System under test for AWS
The environment under test is based on the default Author and Live deployment in CloudOps for AWS. For more information, see Authoring and Live Instance Deployment.
For each environment size, the only change to a server instance was at the database level, where an appropriately sized Relational Database Service (RDS) database was chosen. The EP Cortex Nodes are auto-scaled for each environment and are limited to the max-nodes listed in the environment size:
-3for small
-6for medium
-10 for large
-18 for xlarge
System under test for Azure
The environment under test is based on the default deployment in CloudOps for Azure. For more information, see the Architecture Overview. As with AWS, the database was sized appropriately for environment size and Kubernetes nodes and pods autoscaled to matched database thoughput capacity.
User-system interaction model
The user-system test is built based on the following user-system interaction model.
User Transactions
A user transaction is a discrete interaction between the client and the system, where the client makes a request and waits for a response. For this system, a user transaction is one RESTful HTTP request invoking one or more Cortex resources. It requests from more than one resource which can be grouped by the Cortex Zoom functionality. The request grants the client permission to populate the page to display. For example, in a single HTTP request, the transaction UT_RetrieveRandomProduct allows the client to access a single item. It retrieves the item along with its definition, assets, pricing, availability, and a link to the add-to-cart.
note
In this benchmark, a second transaction, UT_GetDefaultCart, supplements all user transactions. UT_GetDefaultCart simulates a client loading a page, and also the page’s mini-cart, displaying the total items in the user’s cart.
Use cases
A use case is a task performed by a user that initiates a series of user transactions. For example, the use case UC_Browse initiates the following series of user transactions:
UT_RetrieveNavUT_RetrieveNavHierarchyUT_RetrieveProductsInNavHierarchyUT_RetrieveRandomProduct
Sessions
A session is a continuous collection of user-system interactions that consists of one or more use cases.
Session types
The following session types comprise this test:
- Browse category
- Offer search
- Add to cart
- Checkout and purchase
Browse category
The BrowseCategory session simulates an unauthenticated user or client browsing through categories and products.
| User transaction | Zoom request (URL) |
|---|---|
UT_RetrieveNav | ${application}?zoom=navigations:element,navigations:element:child |
UT_RetrieveNavHierarchy | ${application}?zoom=navigations:element,navigations:element:child |
UT_RetrieveProductsInNavHierarchy | ${application}/navigations/${scope}/lookups/form?zoom=items,items:element,items:element:code,items:element:availability,items:element:definition,items:element:definition:assets:element,items:element:price,items:element:rate,element,element:availability,element:definition,element:definition:assets:element,element:price,element:rate,element:code&followlocation |
UT_RetrieveRandomProduct | ${application}/items/${scope}/lookups/form?zoom=availability,addtocartform,addtowishlistform,price,rate,definition,definition:assets:element,definition:options:element,definition:options:element:value,definition:options:element:selector:choice,definition:options:element:selector:chosen,definition:options:element:selector:choice:description,definition:options:element:selector:chosen:description,definition:options:element:selector:choice:selector,definition:options:element:selector:chosen:selector,definition:options:element:selector:choice:selectaction,definition:options:element:selector:chosen:selectaction,recommendations,recommendations:crosssell,recommendations:recommendation,recommendations:replacement,recommendations:upsell,recommendations:warranty,recommendations:crosssell:element:code,recommendations:recommendation:element:code,recommendations:replacement:element:code,recommendations:upsell:element:code,recommendations:warranty:element:code,recommendations:crosssell:element:definition,recommendations:recommendation:element:definition,recommendations:replacement:element:definition,recommendations:upsell:element:definition,recommendations:warranty:element:definition,recommendations:crosssell:element:price,recommendations:recommendation:element:price,recommendations:replacement:element:price,recommendations:upsell:element:price,recommendations:warranty:element:price,recommendations:crosssell:element:availability,recommendations:recommendation:element:availability,recommendations:replacement:element:availability,recommendations:upsell:element:availability,recommendations:warranty:element:availability,code&followlocation |
Offer search
The OfferSearch session simulates an unauthenticated user or client searching and loading products.
| User transaction | Zoom request (URL) |
|---|---|
UT_GetFacetedSearchForm | ${application}?zoom=searches:keywordsearchform,searches:offersearchform |
UT_SubmitFacetedSearchForm | ${application}/offersearches/${scope}/offers/form?zoom=element,element:availability,element:definition,element:definition:assets:element,element:price,element:rate,element:code,element,element:availability,element:definition,element:definition:assets:element,element:pricerange,element:items,element:items:element,element:items:element:availability,element:items:element:definition,element:items:element:definition:assets:element,element:items:element:price,element:items:element:rate,element:items:element:code,facets,facets:element,facets:element:facetselector,facets:element:facetselector:choice:description,facets:element:facetselector:choice:selector,facets:element:facetselector:choice:selectaction,facets:element:facetselector:chosen:description,facets:element:facetselector:chosen:selector,facets:element:facetselector:chosen:selectaction&followlocation |
UT_SelectFacet{1,2} | ${FacetsSelectActionHref}?followlocation&zoom=offersearchresult |
UT_FacetFilteredSearch | ${OfferSearchResultHref}?zoom=element,element:availability,element:definition,element:definition:assets:element,element:price,element:rate,element:code,element,element:availability,element:definition,element:definition:assets:element,element:pricerange,element:items,element:items:element,element:items:element:availability,element:items:element:definition,element:items:element:definition:assets:element,element:items:element:price,element:items:element:rate,element:items:element:code,facets,facets:element,facets:element:facetselector,facets:element:facetselector:choice:description,facets:element:facetselector:choice:selector,facets:element:facetselector:choice:selectaction,facets:element:facetselector:chosen:description,facets:element:facetselector:chosen:selector,facets:element:facetselector:chosen:selectaction |
UT_RetrieveRandomProduct | ${application}/items/${scope}/lookups/form?zoom=availability,addtocartform,addtowishlistform,price,rate,definition,definition:assets:element,definition:options:element,definition:options:element:value,definition:options:element:selector:choice,definition:options:element:selector:chosen,definition:options:element:selector:choice:description,definition:options:element:selector:chosen:description,definition:options:element:selector:choice:selector,definition:options:element:selector:chosen:selector,definition:options:element:selector:choice:selectaction,definition:options:element:selector:chosen:selectaction,recommendations,recommendations:crosssell,recommendations:recommendation,recommendations:replacement,recommendations:upsell,recommendations:warranty,recommendations:crosssell:element:code,recommendations:recommendation:element:code,recommendations:replacement:element:code,recommendations:upsell:element:code,recommendations:warranty:element:code,recommendations:crosssell:element:definition,recommendations:recommendation:element:definition,recommendations:replacement:element:definition,recommendations:upsell:element:definition,recommendations:warranty:element:definition,recommendations:crosssell:element:price,recommendations:recommendation:element:price,recommendations:replacement:element:price,recommendations:upsell:element:price,recommendations:warranty:element:price,recommendations:crosssell:element:availability,recommendations:recommendation:element:availability,recommendations:replacement:element:availability,recommendations:upsell:element:availability,recommendations:warranty:element:availability,code&followlocation |
Add to cart
The AddToCart session simulates an unauthenticated user or client browsing products and adding one to cart, then viewing the cart.
| User transaction | Zoom request (URL) |
|---|---|
UC_BrowseCategory | Note: Transactions in the UC_BrowseCategory use case above are executed. |
UT_AddToCart | ${AddToCartActionHref}?followlocation |
UT_ViewFullCart | ${application}?zoom=defaultcart,defaultcart:appliedpromotions:element,defaultcart:discount,defaultcart:lineitems:element,defaultcart:lineitems:element:appliedpromotions,defaultcart:lineitems:element:appliedpromotions:element,defaultcart:lineitems:element:availability,defaultcart:lineitems:element:item,defaultcart:lineitems:element:item:code,defaultcart:lineitems:element:item:definition,defaultcart:lineitems:element:item:definition:details,defaultcart:lineitems:element:item:definition:options:element,defaultcart:lineitems:element:item:definition:options:element:selector:choice,defaultcart:lineitems:element:item:definition:options:element:selector:choice:description,defaultcart:lineitems:element:item:definition:options:element:selector:chosen,defaultcart:lineitems:element:item:definition:options:element:selector:chosen:description,defaultcart:lineitems:element:item:definition:options:element:value,defaultcart:lineitems:element:price,defaultcart:lineitems:element:total,defaultcart:order:couponinfo:coupon,defaultcart:order:couponinfo:couponform,defaultcart:total |
Checkout and purchase
The Checkout session simulates a new user or client performing the following transactions:
- Adding to cart
- Creating a new profile
- Updating address
- Adding shipping and payment information
- Viewing the order
- Completing a purchase
- Logging out
| User transaction | Zoom request (URL) |
|---|---|
UC_AddToCart | Note: Transactions in UC_AddToCart use case above are executed. |
UT_GetRegistrationForm | ${application}/registrations/\${scope}/newaccount/form |
UT_SubmitRegistrationForm | ${registrationFormActionHref} |
UT_AuthenticationWithCreatedProfile | ${application} |
UT_ViewProfile | ${application}?zoom=defaultprofile,defaultprofile:purchases,defaultprofile:purchases:element,defaultprofile:addresses,defaultprofile:addresses:element,defaultprofile:addresses:billingaddresses:default,defaultprofile:paymentmethods,defaultprofile:paymentmethods:element |
UT_GetCountries | ${application}/geographies/${scope}/countries?zoom=element,element:regions,element:regions:element,countries:element,countries:element:regions,countries:element:regions:element |
UT_GetAddressForm | ${application}/addresses/${scope}/form |
UT_CreateDefaultAddress | ${createProfileLocationURL}/addresses |
UT_ViewProfile | ${application}?zoom=defaultprofile,defaultprofile:purchases,defaultprofile:purchases:element,defaultprofile:addresses,defaultprofile:addresses:element,defaultprofile:addresses:billingaddresses:default,defaultprofile:paymentmethods,defaultprofile:paymentmethods:element |
UT_GetPaymentTokenForm | ${application}?zoom=defaultcart:order:paymentmethodinfo:paymenttokenform,defaultprofile:paymentmethods:paymenttokenform |
UT_CreatePaymentToken | ${application}/paymenttokens/${scope} |
UT_ViewProfile | ${application}?zoom=defaultprofile,defaultprofile:purchases,defaultprofile:purchases:element,defaultprofile:addresses,defaultprofile:addresses:element,defaultprofile:addresses:billingaddresses:default,defaultprofile:paymentmethods,defaultprofile:paymentmethods:element |
UT_ViewFullCart | ${application}?zoom=defaultcart,defaultcart:appliedpromotions:element,defaultcart:discount,defaultcart:lineitems:element,defaultcart:lineitems:element:appliedpromotions,defaultcart:lineitems:element:appliedpromotions:element,defaultcart:lineitems:element:availability,defaultcart:lineitems:element:item,defaultcart:lineitems:element:item:code,defaultcart:lineitems:element:item:definition,defaultcart:lineitems:element:item:definition:details,defaultcart:lineitems:element:item:definition:options:element,defaultcart:lineitems:element:item:definition:options:element:selector:choice,defaultcart:lineitems:element:item:definition:options:element:selector:choice:description,defaultcart:lineitems:element:item:definition:options:element:selector:chosen,defaultcart:lineitems:element:item:definition:options:element:selector:chosen:description,defaultcart:lineitems:element:item:definition:options:element:value,defaultcart:lineitems:element:price,defaultcart:lineitems:element:total,defaultcart:order:couponinfo:coupon,defaultcart:order:couponinfo:couponform,defaultcart:total |
UT_ProceedToCheckout | ${application}?zoom=defaultcart,defaultcart:appliedpromotions:element,defaultcart:discount,defaultcart:order,defaultcart:order:billingaddressinfo:billingaddress,defaultcart:order:billingaddressinfo:selector:choice,defaultcart:order:billingaddressinfo:selector:choice:description,defaultcart:order:couponinfo:coupon,defaultcart:order:couponinfo:couponform,defaultcart:order:deliveries:element:destinationinfo:destination,defaultcart:order:deliveries:element:destinationinfo:selector:choice,defaultcart:order:deliveries:element:destinationinfo:selector:choice:description,defaultcart:order:deliveries:element:shippingoptioninfo:selector:choice,defaultcart:order:deliveries:element:shippingoptioninfo:selector:choice:description,defaultcart:order:deliveries:element:shippingoptioninfo:shippingoption,defaultcart:order:paymentmethodinfo:paymentmethod,defaultcart:order:paymentmethodinfo:selector:choice,defaultcart:order:paymentmethodinfo:selector:choice:description,defaultcart:order:tax,defaultcart:order:total,defaultcart:total |
UT_CompleteOrder | ${application}?zoom=defaultcart,defaultcart:appliedpromotions:element,defaultcart:discount,defaultcart:lineitems:element,defaultcart:lineitems:element:item,defaultcart:lineitems:element:item:code,defaultcart:lineitems:element:item:definition,defaultcart:lineitems:element:item:definition:options:element,defaultcart:lineitems:element:item:definition:options:element:value,defaultcart:lineitems:element:item:price,defaultcart:lineitems:element:total,defaultcart:order,defaultcart:order:billingaddressinfo:billingaddress,defaultcart:order:couponinfo:coupon,defaultcart:order:couponinfo:couponform,defaultcart:order:deliveries:element:destinationinfo:destination,defaultcart:order:deliveries:element:shippingoptioninfo:shippingoption,defaultcart:order:paymentmethodinfo:paymentmethod,defaultcart:order:purchaseform,defaultcart:order:tax,defaultcart:order:total,defaultcart:total |
UT_CompletePurchase | ${SubmitOrderHref}?followlocation&zoom=appliedpromotions:element,billingaddress,discount,lineitems:element,lineitems:element:options:element,lineitems:element:options:element:value,paymentmeans:element,shipments:element:destination,shipments:element:shippingoption |
Testing model
The testing model test uses a steady state model. For a specific period of time, each virtual user continuously simulates a series of sessions. The time period is broken up into the following phases:
- Warm-up phase (no measurements)
- Measurement phase
The test repeats with increasing virtual user and concurrency levels, and each data point has measurements recorded.
The virtual users work continuously. There is no allowance for think time between user transactions.
Session distribution
The following table shows the session distribution:
| Session | Representation |
|---|---|
| Browse | 40% |
| Search Offer | 45% |
| Add-to-Cart | 10% |
| Purchase | 5% |
Concurrency levels and simulation times
The range concurrency levels must be adjusted based on the configuration under test. The test must involve a single virtual user test at the beginning and end of the run. It should increase concurrency levels to the point where the system begins to perform unacceptably and does not meet the success criteria.
The warm-up phase and measurement phase must be adjusted based on the test configuration. The times for each phase must be long enough that each session type is measured at least 10 times.
The following table is a sample of the expected test configuration:
| Virtual Users | Warm-up phase (minutes) | Measurement phase (minutes) | Total cumulative time (minutes) |
|---|---|---|---|
| 1 | 1 | 5 | 6 |
| 5 | 1 | 5 | 12 |
| 15 | 1 | 5 | 18 |
| 30 | 1 | 5 | 24 |
| 45 | 1 | 5 | 30 |
| 60 | 1 | 5 | 36 |
| 80 | 1 | 5 | 42 |
| 100 | 1 | 5 | 48 |
Key metrics
Overall Apdex
For each execution, the Apdex calculates all measured user transactions. The T value key metric is the smallest Apdex value measured for any specific user transaction. All user transactions use a value of T = 500 milliseconds, yielding an F of 2000 milliseconds.
Minimum Apdex
For each execution, the Apdex calculates each user transactions independently, using a T value as defined in the Overall Apdex. The metric indicates how bad the user experience was for the worst user transaction.
Transactional throughput
The transaction throughput is the average number of user transactions per second that the system is able to perform over a measurement period. To measure this transaction, divide the number of measured user transactions by the measurement time in seconds.
Average response time
The average response time for all measured user transactions.
Maximum response time
The longest response time for a measured user transaction in seconds.
Sessions throughput per hour
The average rate of session throughput that the system was able to deliver, converted into an hourly metric. You can find the throughput per hour by using the following formula: number of sessions completed during the measurement time divided by the measurement time (in minutes), multiplied by 60 minutes.
User errors
The number of errors the virtual end user encounters and reports for each execution.
Success criteria
A test execution is deemed to be successful when the following criteria are met:
- Overall Apdex rating = Good: If the rating for a given data point is Good or better, then the system is deemed to have successfully supported that level of load, subject to other criteria. If the rating for a given data point is Fair or lower, then the system cannot support that load level
- Minimum Apdex >= Fair: If the rating for any user transaction falls above Fair, such as Good, the system is deemed to have successfully supported that level of load. If the rating for any user transaction falls below Fair, such as Poor or Unacceptable, the system cannot support that load level
- User errors < 1 %: To be successful, less than 1% of measured user transactions can result in errors
Test results
Review all available data to ensure that the results are valid. The following situations can invalidate test results:
- Load generators are underpowered and become the bottleneck
- A scripting error causes certain user transactions to not include all HTTP requests, reporting better results