The customer wants to migrate their current Splunk Index cluster to new hardware to improve...

The correct process and procedure for migrating a Splunk index cluster to new hardware is as follows:

Install new indexers. This step involves installing the Splunk Enterprise software on the new machines and configuring them with the same network settings, OS settings, and hardware specifications as the original indexers.

Configure indexers into the cluster as peers; ensure they receive the cluster bundle and the same configuration as original peers. This step involves joining the new indexers to the existing cluster as peer nodes, using the same cluster master and replication factor. The new indexers should also receive the same configuration files as the original peers, either by copying them manually or by using a deployment server. The cluster bundle contains the indexes.conf file and other files that define the index settings and data retention policies for the cluster.

Decommission old peers one at a time. This step involves removing the old indexers from the cluster gracefully, using the splunk offline command or the REST API endpoint /services/cluster/master/control/control/decommission. This ensures that the cluster master redistributes the primary buckets from the old peers to the new peers, and that no data is lost during the migration process.

Remove old peers from the CM’s list. This step involves deleting the old indexers from the list of peer nodes maintained by the cluster master, using the splunk remove server command or the REST API endpoint /services/cluster/master/peers. This ensures that the cluster master does not try to communicate with the old peers or assign them any search or replication tasks.

Update forwarders to forward to the new peers. This step involves updating the outputs.conf file on the forwarders that send data to the cluster, so that they point to the new indexers instead of the old ones. This ensures that the data ingestion process is not disrupted by the migration.

References:

Splunk Core Certified Consultant Test Blueprint

Splunk Documentation: Migrate an indexer cluster

Splunk Documentation: Decommission a peer node

Splunk Documentation: Remove a peer node from an indexer cluster

Splunk Documentation: Configure forwarders with outputs.conf