Taking Elasticsearch snapshots using Curator

By | August 7, 2017

This tutorial on taking Elasticsearch snapshots using curator will be divided into sections. One obvious section is how to take snapshots. Other less obvious part will be on configuring a shared directory using Network file sharing on Linux. I will be using a RHEL 7 based cluster of three machines for this tutorial. Once you are done with the basics I outline here, you should start using curator to manage your aliases as my next post details.

As usual I will start with WHY followed by HOW.

WHY


You want to take backups. If you are running a ELK stack then sooner or later you will have old logs which you want to archive and free up space on your cluster. When you upgrade your cluster then you have to take snapshots before doing anything. And there is always a that hardware failure scenario.

HOW

You can take Elasticsearch snapshots in many ways. Simplest is via curl commands. But it is better to use the tool given by Elastic. It is called …..Drum rolls please.
Elasticsearch snapshots using curator

Steps to install curator on a RHEL/CentOS machine

Some housekeeping work. Since Elasticsearch is evolving rapidly you should check the latest instructions here.

Create a curator.repo file

and put this in content in it

Actual installation

Contrary to other tools it does not have a config file already prepared and ready for you to change.

Elasticsearch snapshots using curator
So you have to create a config file curator.yml yourself. A great starting template is located here. Just change the hosts and the port and you will be good to go. In case the curator is running on same machine as the elasticsearch is running you really can use this file as it is.
To make like easier store this file in ~/.curator location. Otherwise you have to pass the file location using –config option every time you run the tool to take Elasticsearch snapshots. And who wants to do that? Not me.

So create a directory

Create a file curator.yml in it.

Put this into the file.

For dumping the curator logs you need to have a folder. Hence the /home/elastic/logs folder which you see in the text above.

Create the folder and give the necessary permissions. Though I am logged in as elastic user (belonging to a group elk) I am doing this explicitly. The most common problem during setting up of Elasticsearch snapshots is permissions on folders. Hence making it very visible here.

Now check if everything is working fine.

Once things are working fine then its time to do something useful. Curator can do a variety of tasks on your indices. These are called Actions and the full list is here.
You pass curator the actions via an action file. You need to pass the location of the file at command line. One nice thing about the tool is the dry-run option which allows you to do a test run safely without actually changing anything in the cluster.
You can see a whole list of sample action files doing the stuff at this location.

I will pick up the snapshot action file and change it to suit my needs. Then I can take a snapshot of a given series of indices. Elasticsearch snapshots … here I come.

elasticsearch snapshot using curator

Tedious networking stuff
You have to create a shared directory for the nodes first. And this is the hard part where lot of networking issues can trip you. The short story is that you need a location which is visible “all” the nodes in the cluster. And these nodes should have read/write permissions on that shared location. Can’t stress this point enough. The idea is that when snapshot command is issued then all nodes start dumping the data from the part of index on them on to the shared location. If you already have this sorted out then skip this section by clicking here.

I will use Network File Service on RHEL 7 to create a shared directory. Then I will create a folder on each node. Folder name and path will be same. And I will mount this shared directory on each node at that particular folder.

Installing the needed softwares (I will keep this brief. I used this site. In case anything fails refer to this one or google).

On the machine where the NFS server will be running and the shared folder will be located.

Now you create the directory to share with the clients.

Modify /etc/exports

and put this in it.

Then export the shared directory.

On the cluster nodes some installs and configuration is needed too.
Let us create a folder on each of the nodes.

We will mount the shared directory at this location.

NFS related software installs

Check if the exported dir is visible on the client

This should show

You want the mounting of this shared directory to happen automatically on the clients when the reboot happens because reboots happen.
Open /etc/fstab

and add line

You want to check if the auto mount is happening. One option is to reboot the machine and then check it. But if you have any syntax mistakes in the /etc/fstab then there are chances that machine might not boot up. This can be an issue when the machine is a remote machine. So it is better to unmount the shared folder on the client machine and then just do a mount. The shared folder should get mounted automatically.

You should see that the shared directory is mounted.

Check if the mount is writeable after the automount

You should be able to see the file across all the nodes and inside the shared directory of on the ServerHostingNFS. Try out create and delete combinations to find if there are nay permission issues. Reboot the nodes and see if the automounting is happening.

Once you have everything on shared directory sorted out rest of stuff is actually easy.

Configuring Elasticsearch
You have to add an entry in the elasticsearch.yml
Open the file

and add

Then restart each elasticsearch on each of the node.
If you have any issues then elasticsearch will refuse to start up. Go through the logs to find the issue. Most of the times it is because of the permissions.

Now you have to create a repository in elasticsearch and map it to the location where the shared files service is mounted.
Use curl command in linux terminal

Now in elasticsearch I have registered a repository which has name logs_backup. All the nodes will dump the data to /Data/mount/backups which actually refers to Data/backups shared file system.

Now we have to have an action file for curator to work with. Let us call it action_snapshot.yml and put this content in it.

Line 11: We are passing the name of the repository we had registered earlier.
Line 14: This is the name of the snapshot. See how it will be appended with time information.
Line 24: The indices it will match. Here it will pick all the indices which start with log-production.

Now you can take Elasticsearch snapshots !! If you are a sane person you will like to do a dry run. Not me.

Output is something like this

Go and take a peek in the shared location, “Data/backups” in our case. They should have the backup files.
You can also issue a command on the terminal to see the snapshots.

Output is something like this

Now with that done only one thing is left. Do a restore using Elasticsearch snapshots you have taken. How you do it is something you have to decide. For me it is simple. Since I am working with test data I will count the number of documents in the indices whose snapshot was taken. Then I will delete the indices. Then restore. And if the count of document matches with the intial one I know that restore worked.

Count of intial docs

Output

Document count is 5368390

Then a delete

To restore you need an action file.
I will create an action file “action_snapshot_restore.yml”

Line 28: I want to work with the Elasticsearch snapshots with name beginning with ProductionLogs.
Line 10: I specify the repository to be used.
Line 12: I choose the snapshot. This is useful if you want to restore the indices only till a point in the past. To restore the indices till present leave it blank. Elasticsearch will use the latest snapshot.

System.Threading.Timer Will it work
Time to push the button.

Output

A quick curl command to check if Elasticsearch snapshots restore worked. See the count of documents restored.

Output

The count is spot on. You tamed the Elasticsearch snapshots. Now you are ready to take your curator skills to next level. Start managing your aliases with curator.

3 thoughts on “Taking Elasticsearch snapshots using Curator

  1. Pingback: Managing Elasticsearch Aliases using Curator

Leave a Reply