Search Server Clustering

Elastic Path Search Server provides product discovery (searching and browsing) for Cortex and catalog management for Commerce Manager. This section provides advice for how to scale a Search Server deployment in a production environment. For staging environments, it is usually sufficient to have a single search leader and no followers.

Topology

Search Server is implemented using a leader-follower topology. Only a single leader server can run at a time, otherwise database conflicts occur updating the TINDEXBUILDSTATUS table.

Two common deployment topologies:

1. Small Deployment Topology

   The simplest topology suitable for small deployments is to deploy a Search follower on every Cortex node. Then access Search Server through the http://localhost:8082/searchserver URL.

2. Large Deployment Topology

   Large deployments typically have a separate cluster of Search followers behind a stateless load balancer. The cluster is accessed through the load balancer URL.

   As a starting point, we recommend deploying one Search follower for every three Cortex servers to support catalog searching and browsing. This ratio may vary depending on the searching and browsing patterns for the store and customizations that change search server usage. The following is a large deployment example:

   

Cluster Configuration

The leader Search Server and follower are deployed using the same WAR file. By default, the WAR is configured to run as a leader node with replication disabled. The following steps configure both leader and follower.

1. Externalize solrHome directory (leader and follower).

   Move the solrHome directory from the Search WAR to an external location. We recommend to include the solrHome in the deployment package as a separate artifact and then deploy the artifact to the desired location on the server. The benefits of this method are:

   • You can add replication configuration without invading the WAR
   • Search indexes are not deleted when a new WAR is deployed

   After solrHome is deployed to the desired location, set the following JVM (Java Virtual Machine) parameter in your appserver startup script:

   -Dsolr.solr.home=/ep/external-solrHome

2. Configure the leader replication.

   Each config file in solrHome/conf has an <xi:include> tag that points to a corresponding replication configuration file.

   <!-- include replication settings -->
   <xi:include href="replication/product.solr.replication.config.xml" xmlns:xi="http://www.w3.org/2001/XInclude">
   	<!-- Silent fallback if no external configuration found -->
   	<xi:fallback/>
   </xi:include>

   In this case, Solr looks for product.solr.replication.config.xml in the replication/product.solr.replication.config.xml relative path. The relative path is relative each of the core index folders in the Solr home folder, such as /ep/external-solrHome/cores/product (if -Dsolr.solr.home is set to /ep/external-solrHome).

   Here is an example of the product.solr.replication.config.xml file:

   <requestHandler name="/replication" class="solr.ReplicationHandler">
       <lst name="leader">
   
           <!--Replicate on 'startup' and 'commit'. 'optimize' is also a valid value for replicateAfter. -->
           <str name="replicateAfter">startup</str>
           <str name="replicateAfter">commit</str>
           <!-- <str name="replicateAfter">optimize</str> -->
   
           <!--Create a backup after 'optimize'. Other values can be 'commit', 'startup'. It is
           possible to have multiple entries of this config string. Note that this is just for backup,
           replication does not require this. -->
           <!-- <str name="backupAfter">optimize</str> -->
   
           <!--If configuration files need to be replicated give the names here, separated by comma -->
           <str name="confFiles">schema.xml,stopwords.txt</str>
   
           <!--The default value of reservation is 10 secs.See the documentation below. Normally, you
           should not need to specify this -->
           <str name="commitReserveDuration">00:00:10</str>
       </lst>
   </requestHandler>

   Create separate config files for each index you want replicated. You don’t have to replicate all indexes; just the ones that are actually needed. Here is the full list of config files:

   Index	Replication Config File	Index Used By
   Category	category.solr.replication.config.xml	Cortex, CM
   CM User	cmuser.solr.replication.config.xml	CM
   Product	product.solr.replication.config.xml	Cortex, CM
   Promotion	promotion.solr.replication.config.xml	CM
   SKU	sku.solr.replication.config.xml	CM

3. For each replication configuration file on the leader server, create a corresponding configuration file on the follower. Here is an example of the product.solr.replication.config.xml file:

   <requestHandler name="/replication" class="solr.ReplicationHandler" >
       <lst name="follower">
           <str name="enable">true</str>
   
           <!--fully qualified url for the replication handler of the leader. It is possible to pass on this
           as a request param for the fetchindex command-->
           <str name="leaderUrl">http://venus.elasticpath.net:8080/searchserver/product/replication</str>
   
           <!--Interval in which the replica should poll master .Format is HH:mm:ss . If this is absent replica does
           not poll automatically.
           But a fetchindex can be triggered from the admin or the http API -->
           <str name="pollInterval">00:00:20</str>
   
           <!-- THE FOLLOWING PARAMETERS ARE USUALLY NOT REQUIRED-->
           <!--to use compression while transferring the index files. The possible values are internal|external
           if the value is 'external' make sure that your leader Solr has the settings to honour the
           accept-encoding header.
           see here for details http://wiki.apache.org/solr/SolrHttpCompression
           If it is 'internal' everything will be taken care of automatically.
           USE THIS ONLY IF YOUR BANDWIDTH IS LOW . THIS CAN ACTUALLY SLOWDOWN REPLICATION IN A LAN-->
           <str name="compression">internal</str>
   
           <!--The following values are used when the follower connects to the master to download the index files.
           Default values implicitly set as 5000ms and 10000ms respectively. The user DOES NOT need to specify
           these unless the bandwidth is extremely low or if there is an extremely high latency-->
           <str name="httpConnTimeout">5000</str>
           <str name="httpReadTimeout">10000</str>
   
           <!-- If HTTP Basic authentication is enabled on the leader server, then the follower can be configured
           with the following
           <str name="httpBasicAuthUser">username</str>
           <str name="httpBasicAuthPassword">password</str>
           -->
   
       </lst>
   </requestHandler>

4. Set search server mode

   On each search server, configure the search mode through the -Dep.search.mode Java system property. Set it to PRIMARY (for the leader) or REPLICA (for the follower) as you require.

   For more information about the ep.search.mode setting, see the search mode property.

5. Configure the search server URLs

   Configure the search server URLs through the -Dep.search.primary.url and -Dep.search.replica.url Java system properties. The URLs that you set depends on your deployment topology.

   • Simple topology

     In a simple deployment topology, you only need to configure the replica value to point to http://localhost:<port>/<contextPath>/.

   • Leader/follower cluster topology

     If the search followers are clustered behind a load balancer, then you need two setting values or Java system properties. The replica value identifies the follower cluster and the primary value identifies the search leader.

     Setting Context Path	Java System Property	Value
     replica	-Dep.search.replica.url	URL of the load balancer for the follower cluster
     primary	-Dep.search.primary.url	URL of the search leader

Updating Search Server Cluster

1. Halt any user updates to the system and ensure no changes are in progress through Commerce Manager, the Import/Export tool, or the Data Sync tool.

2. In Commerce Manager, ensure the indexing activity is complete for all indexes. You can find the index status in the Configuration section under Search Indexes.

3. Back up all existing indexes by calling the following links. These links will call the Solr replication API:

   http://leader_host:port/searchserver/category/replication?command=backup
   http://leader_host:port/searchserver/cmuser/replication?command=backup
   http://leader_host:port/searchserver/product/replication?command=backup
   http://leader_host:port/searchserver/promotion/replication?command=backup
   http://leader_host:port/searchserver/shippingservicelevel/replication?command=backup
   http://leader_host:port/searchserver/sku/replication?command=backup

4. Disable replication on the follower servers by calling the following links. Ensure this is done for all followers.

   http://follower_host:port/searchserver/category/replication?command=disablepoll
   http://follower_host:port/searchserver/cmuser/replication?command=disablepoll
   http://follower_host:port/searchserver/product/replication?command=disablepoll
   http://follower_host:port/searchserver/promotion/replication?command=disablepoll
   http://follower_host:port/searchserver/shippingservicelevel/replication?command=disablepoll
   http://follower_host:port/searchserver/sku/replication?command=disablepoll

5. Shut down, update, and restart the leader search server:

   1. Shut down the application server.
   2. Redeploy the updated search server .war file.
   3. Restart the application server.

   If your search indexes are normally located within the searchserver/WEB-INF folder, you may need to restore them from your back up location or do a full rebuild.

6. Using Commerce Manager, rebuild any indexes that would be affected by the update.

   For step-by-step instructions on how to do this, see Requesting Search Index Rebuilds.

7. Verify the leader indexes.

   Since the leader server is a full search instance, you can check its state by doing one of the following:

   1. Issuing manual Solr queries.
   2. By redirecting Cortex server instance to point to the leader instead of a follower.

8. Do one of the following:

   • If you are applying code updates to the follower servers, do the following for all followers:

     1. Remove the replicas from the search cluster.

     2. Shutdown the server.

     3. Redeploy the updated Search Server war file.

     4. Restart the server.

        The restarted followers will immediately begin polling the leader for index updates on startup.

     5. Restore the followers to the search cluster.

   • If you are not updating the follower servers, resume index replication by calling the following links for all followers:

   http://follower_host:port/searchserver/category/replication?command=enablepoll
   http://follower_host:port/searchserver/cmuser/replication?command=enablepoll
   http://follower_host:port/searchserver/product/replication?command=enablepoll
   http://follower_host:port/searchserver/promotion/replication?command=enablepoll
   http://follower_host:port/searchserver/shippingservicelevel/replication?command=enablepoll
   http://follower_host:port/searchserver/sku/replication?command=enablepoll

Search Server Cluster Checklist

To verify if your search server cluster is running properly:

• Check if the indexes are being replicated to the follower server.
• Ensure requests are being redirected to the follower.
• Ensure the update requests are only redirected to the leader server.
• Trace requests in the Apache HTTP server’s log and Tomcat server’s access logs.
• See if requests are redirected to another server if a server is unavailable.