ArcGIS Data Store allows you to create each type of data store on more than one machine to make your data highly available and, in the case of tile cache data stores running in cluster mode and spatiotemporal big data stores, scalable.
Your options for adding machines vary by data store type and are explained as follows:
Relational data stores
The first machine on which you install ArcGIS Data Store and create a relational data store is the primary relational data store machine. You can install and configure one other relational data store with the same GIS Server site, making this second relational data store machine your standby machine. Hosted feature layer data is replicated from the primary to the standby.
If the primary becomes unavailable (for example, if the primary machine crashes, loses power, or loses network connectivity), the standby machine is promoted to primary. You will only be without your hosted feature layer data for the short period of time it takes for the standby machine to become the primary—this can take up to a few minutes. Once the standby becomes the primary relational data store, it sends its backup files to the same location as the former primary machine.
If you can correct the problem with the former primary machine—for example, if the machine was unplugged and you plugged it back in, or the machine temporarily lost network connectivity but network connectivity was later restored—the former primary becomes the standby machine for the data store.
Failover scenarios for relational data stores
The following is a list of situations for which the standby machine becomes the primary for your relational data store. Note that the following three situations involve hardware or software failures.
- The primary data store stops working. ArcGIS Data Store attempts to restart the data store on the primary machine. If it cannot restart, the data store fails over to the standby.
- The primary's web app stops running and attempts to restart the web app on the primary machine. In the rare case that this does not work, the data store fails over to the standby machine.
- The primary machine is unavailable. This can happen if the computer crashes, is unplugged, or loses network connectivity. ArcGIS Data Store makes five attempts to connect to the primary machine. If a connection is not possible after five attempts, the data store fails over to the standby machine.
The only human-initiated situations that cause a failover are if the primary data store machine is deliberately taken offline, or the ArcGIS Server site administrator runs the makePrimary REST command on the standby machine.
Tile cache data stores
The number of machines you can add to a tile cache data store depends on the deployment mode you use.
- In primary-standby mode, a tile cache data store can contain a single machine or, to provide fail-over functionality, it can contain two machines. The tile cache data store defaults to primary-standby mode.
- In cluster mode, a tile cache data store can contain multiple data store machines to provide for scalability.
Recovery after loss of a tile cache machine in cluster mode
If you have multiple machines in your tile cache data store for the purpose of high availability and one of the machines is lost or goes offline, you may need to redistribute scene cache data.
Attempt to bring the failed machine back online. For example, if a network or power cable is unplugged or the machine was shut off, you can rectify the issue by plugging in the network or power cable or turning on the computer. Once back online, the machine rejoins the tile cache data store and the data store continues to function.
If a machine crashes and cannot be resurrected, you need to take steps to ensure your scene layer caches are still highly available. See Recover a data store for instructions.
Spatiotemporal big data stores
You can install ArcGIS Data Store on several machines, create spatiotemporal big data stores on each, and register each one with the same hosting GIS Server site.
Note:
Always use an odd number of machines in your spatiotemporal big data store.
Data in the spatiotemporal big data store is duplicated on at least two spatiotemporal big data store machines registered to the same hosting server.
If any one machine fails, data still exists on at least one other machine. The data store replicates the data to another machine that is still functional.
You can install ArcGIS Data Store on more machines and create spatiotemporal big data stores that are registered with the same GIS Server sites. In this example, three machines are added: one to replace the one that failed, and two more to accommodate increased numbers of hosted spatiotemporal feature layers. Data is rebalanced across all seven machines. Data for newly published spatiotemporal feature layers is also distributed.
Note:
If you script the creation of multiple spatiotemporal big data store machines, one spatiotemporal big data store machine must be manually configured with the GIS Server before you can script the creation of additional spatiotemporal big data store machines. Include wait times in your script to be sure the additional spatiotemporal big data store machines are not added at the same time.
Object stores
You can create an object store that contains one machine or you can deploy an object store that contains one or more clusters of four machines each.
A single machine object store with a large amount of disk space will provide you the best drawing performance when viewing feature layers that are configured to use cached query responses. However, if you require high availability for cached feature query responses, you can deploy an object store in cluster mode. If one machine in the cluster fails, you can still create caches in and read caches from the object store. As long as two machines in the cluster are available, you can continue to read from the cache.
Note:
You must add all machines and clusters to the object store before you start using it; you cannot add machines later.
Each of the four machines in the cluster must have the same folder structure and same specifications. They must all have the same name with the addition of a number at the end of the name. If you cannot name the physical machines in this pattern, or if you are deploying in the cloud, you must edit the etc/hosts file on each machine to alias the machine name and IP address to a qualifying name.
For example, if your four machines are named conquest, war, famine, and plague, you could set aliases horse1, horse2, horse3, and horse4 in the hosts file on each respective machine.
For information on setting aliases, contact your IT department or consult instructions specific to your operating system.
If you need more than four machines to store cached queries, you can add a second set of four machines to the object store for a total of eight machines. The second set of four machines must also have the same folder structure and specifications, but you can start with a new set of names. In other words, if you added another four machines to the object store in the previous example, you do not have to name them horse4, horse5, horse6, and horse7 (though you can if you want to do so.) You could name or alias the second set of four machines to, for example, buggy1, buggy2, buggy3, and buggy4.
If one of the machines in an object store cluster fails, attempt to bring it back online. If you cannot recover the machine, install ArcGIS Data Store on a new machine that has the same specifications, folder structure, and exact same name as the machine that you are replacing, and run the configuredatastore utility to bring the new machine into the cluster.
Add a machine
Use the Data Store Configuration wizard or the configuredatastore utility to add a machine to an existing relational, tile cache, or spatiotemporal big data store. Use the configuredatastore utility to add a machine to an object store before you start using the object store; you cannot add a machine after the object store is already in use.
You can have one standby relational data store or tile cache data store that is running in primary-standby mode. You can have multiple machines in a tile cache data store (in cluster mode) or spatiotemporal big data store. Add each machine, one at a time. All machines that participate in the same data store must use the same operating system.
When configuring an object store to contain multiple machines, you must add machines in clusters of four machines, and all machines must have the same directory structure and specifications.
Use the Data Store Configuration wizard
Follow these steps to use the Data Store Configuration wizard to add a machine to a data store.
- Install ArcGIS Data Store on another server.
The standby and primary machines (for a relational data store or tile cache data store running in primary-standby mode) and each node in a tile cache data store running in cluster mode or spatiotemporal big data store must be installed on different physical servers. Be sure to use the same network account for the ArcGIS Data Store account on all machines in the data store.
- Open the ArcGIS Data Store Configuration wizard.
- Specify the URL of the GIS Server site.
The URL is in the format https://gisserver.domain.com:6443. Use the same GIS Server site as you did when configuring the primary relational data store or the other machines in your tile cache or spatiotemporal big data store.
- Type the user name and password of the ArcGIS Server primary site administrator and click Next.
- Choose the type of data store you want to create. Be sure to choose the same type as is on the data store machine to which you want to add this machine.
- Specify a location for the new data store's directory and click Next.
- Review the information in the Configuration Summary. If it is correct, click Finish. If changes are needed, click Back to make corrections.
The High Availability Role shows this is a standby machine for relational data stores.
When the setup completes successfully, you have an additional data store machine to which the primary relational data store or other machines in the spatiotemporal big data store can replicate data, or you have an additional machine that can store caches for newly published hosted scene layers.
If there were errors with the setup, click Next on the Configuration Summary dialog box to reopen the Data Store Configuration wizard so you can provide different information. See Troubleshoot ArcGIS Data Store for causes and resolutions to setup errors.
Use the configuredatastore utility
You can install ArcGIS Data Store on another server or servers and use the configuredatastore utility to add it to an existing data store. The utility is installed in <ArcGIS Data Store installation directory>\arcgis\datastore\tools.
The standby and primary machines (for a relational data store or tile cache data store running in primary-standby mode) and each node in a tile cache data store or object store running in cluster mode or spatiotemporal big data store must be installed on different physical servers. Be sure to use the same network account for the ArcGIS Data Store account on all machines in the data store.
Follow these steps to use the configuredatastore utility to add a machine to a data store.
- Open a command shell on the additional server where you installed ArcGIS Data Store.
- Run the configuredatastore utility, and provide the GIS Server URL and the ArcGIS Server primary site administrator user name and password. Also use the --stores operation to specify the type of data store to be created. Be sure to specify the same GIS Server URL and data store type that you used for the data store to which you want to add this machine.
For example, this command adds a standby relational data store to a GIS Server site with the URL https://agsserver.ntwk.com:6443/arcgis/admin. The data store directory on this machine is /home/dsuser/fsdata.
/configuredatastore https://agsserver.ntwk.com:6443/arcgis/admin siteadmin P$@pass /home/dsuser/fsdata --stores relational
An additional data store machine is added to your data store.