This tutorial on managing Elasticsearch aliases using Curator will help you to manage your Elasticsearch aliases better. There are not many detailed tutorials on this topic and hence this post. I hope that at the end of this tutorial you will appreciate the power curator brings to your hands. Continue reading →
This brief post is about cutomising putty terminal color schemes to something more pleasant. The default ones which come with PuTTY are a relic of past. So unless you love the retro there is no need to put up with the PuTTY defaults. You can change it to something more pleasant but that involves registry editing. What if it is disabled? Continue reading →
This tutorial on taking Elasticsearch snapshots using curator will be divided into sections. One obvious section is how to take snapshots. Other less obvious part will be on configuring a shared directory using Network file sharing on Linux. I will be using a RHEL 7 based cluster of three machines for this tutorial. Once you are done with the basics I outline here, you should start using curator to manage your aliases as my next post details.
As usual I will start with WHY followed by HOW.
You want to take backups. If you are running a ELK stack then sooner or later you will have old logs which you want to archive and free up space on your cluster. When you upgrade your cluster then you have to take snapshots before doing anything. And there is always a that hardware failure scenario.
You can take Elasticsearch snapshots in many ways. Simplest is via curl commands. But it is better to use the tool given by Elastic. It is called …..Drum rolls please.
Steps to install curator on a RHEL/CentOS machine
Some housekeeping work. Since Elasticsearch is evolving rapidly you should check the latest instructions here.
Contrary to other tools it does not have a config file already prepared and ready for you to change.
So you have to create a config file curator.yml yourself. A great starting template is located here. Just change the hosts and the port and you will be good to go. In case the curator is running on same machine as the elasticsearch is running you really can use this file as it is.
To make like easier store this file in ~/.curator location. Otherwise you have to pass the file location using –config option every time you run the tool to take Elasticsearch snapshots. And who wants to do that? Not me.
So create a directory
Create a file curator.yml in it.
Put this into the file.
# Remember, leave a key empty if there is no value. None will be a string,
## not a Python "NoneType"
hosts:<your machine ip address>
For dumping the curator logs you need to have a folder. Hence the /home/elastic/logs folder which you see in the text above.
Create the folder and give the necessary permissions. Though I am logged in as elastic user (belonging to a group elk) I am doing this explicitly. The most common problem during setting up of Elasticsearch snapshots is permissions on folders. Hence making it very visible here.
sudo chown-Relastic:elk logs
Now check if everything is working fine.
Once things are working fine then its time to do something useful. Curator can do a variety of tasks on your indices. These are called Actions and the full list is here.
You pass curator the actions via an action file. You need to pass the location of the file at command line. One nice thing about the tool is the dry-run option which allows you to do a test run safely without actually changing anything in the cluster.
You can see a whole list of sample action files doing the stuff at this location.
I will pick up the snapshot action file and change it to suit my needs. Then I can take a snapshot of a given series of indices. Elasticsearch snapshots … here I come.
Tedious networking stuff
You have to create a shared directory for the nodes first. And this is the hard part where lot of networking issues can trip you. The short story is that you need a location which is visible “all” the nodes in the cluster. And these nodes should have read/write permissions on that shared location. Can’t stress this point enough. The idea is that when snapshot command is issued then all nodes start dumping the data from the part of index on them on to the shared location. If you already have this sorted out then skip this section by clicking here.
I will use Network File Service on RHEL 7 to create a shared directory. Then I will create a folder on each node. Folder name and path will be same. And I will mount this shared directory on each node at that particular folder.
Installing the needed softwares (I will keep this brief. I used this site. In case anything fails refer to this one or google).
On the machine where the NFS server will be running and the shared folder will be located.
sudo yum install nfs-utils libnfsidmap
sudo systemctl enable rpcbind
sudo systemctl enable nfs-server
sudo systemctl start rpcbind
sudo systemctl start nfs-server
sudo systemctl start rpc-statd
sudo systemctl start nfs-idmapd
Now you create the directory to share with the clients.
and put this in it.
Then export the shared directory.
On the cluster nodes some installs and configuration is needed too.
Let us create a folder on each of the nodes.
You want to check if the auto mount is happening. One option is to reboot the machine and then check it. But if you have any syntax mistakes in the /etc/fstab then there are chances that machine might not boot up. This can be an issue when the machine is a remote machine. So it is better to unmount the shared folder on the client machine and then just do a mount. The shared folder should get mounted automatically.
You should see that the shared directory is mounted.
ServerHostingNFS:/home/elastic/backups on/home/elastic/mount/backups type nfs4.....blah blah
Check if the mount is writeable after the automount
You should be able to see the file across all the nodes and inside the shared directory of on the ServerHostingNFS. Try out create and delete combinations to find if there are nay permission issues. Reboot the nodes and see if the automounting is happening.
Once you have everything on shared directory sorted out rest of stuff is actually easy.
You have to add an entry in the elasticsearch.yml
Open the file
Then restart each elasticsearch on each of the node.
If you have any issues then elasticsearch will refuse to start up. Go through the logs to find the issue. Most of the times it is because of the permissions.
Now you have to create a repository in elasticsearch and map it to the location where the shared files service is mounted.
Use curl command in linux terminal
the repository filesystem access check.Usethe other options tocreate
# Leaving name blank will result in the default 'curator-%Y%m%d%H%M%S'
Line 11: We are passing the name of the repository we had registered earlier. Line 14: This is the name of the snapshot. See how it will be appended with time information. Line 24: The indices it will match. Here it will pick all the indices which start with log-production.
Now you can take Elasticsearch snapshots !! If you are a sane person you will like to do a dry run. Not me.
2017-08-0217:08:13,905INFO Trying Action ID:1,"snapshot":Snapshot log-production-prefixed indices older than1day(based on index creation_date)with the defaultsnapshot name pattern of'curator-%Y%m%d%H%M%S'.Wait forthe snapshot tocomplete.Donotskip the repository filesystem access check.Usethe other options tocreate the snapshot.
Now with that done only one thing is left. Do a restore using Elasticsearch snapshots you have taken. How you do it is something you have to decide. For me it is simple. Since I am working with test data I will count the number of documents in the indices whose snapshot was taken. Then I will delete the indices. Then restore. And if the count of document matches with the intial one I know that restore worked.
To restore you need an action file.
I will create an action file “action_snapshot_restore.yml”
Restore all indices inthe most recent curator-*snapshot with state
SUCCESS.Wait forthe restore tocomplete before continuing.Donotskip
the repository filesystem access check.Usethe other options todefine
the index/shard settings forthe restore.
# If name is blank, the most recent snapshot by age will be selected
# If indices is blank, all indices in the snapshot will be restored
Line 28: I want to work with the Elasticsearch snapshots with name beginning with ProductionLogs. Line 10: I specify the repository to be used. Line 12: I choose the snapshot. This is useful if you want to restore the indices only till a point in the past. To restore the indices till present leave it blank. Elasticsearch will use the latest snapshot.
2017-08-0312:54:33,499INFO Trying Action ID:1,"restore":Restore all indices inthe most recent curator-*snapshot with state SUCCESS.Wait forthe restore tocomplete before continuing.Donotskip the repository filesystem access check.Usethe other options todefine the index/shard settings forthe restore.