This tutorial on managing Elasticsearch aliases using Curator will help you to manage your Elasticsearch aliases better. There are not many detailed tutorials on this topic and hence this post. I hope that at the end of this tutorial you will appreciate the power curator brings to your hands.
WHY
We cannot keep storing logs in an ELK stack. Old logs are phased out in stages. New logs take their place. If you have got analytics running on top of these logs (which you should) then there will be this issue of changing indices name.
In world of logging the name of logs follows a pattern like MyAwesomeLogs-2017-12-30. Now depending on the configuration indices for storing logs might be created on monthly, weekly or daily basis. To visualise the data in logs mostly Kibana is used. And dashboards in Kibana are based on index names. So if you create a Dashboard on the most common error for month of March then in April you have to change the dashboard source of data to the index created in April.
Will it not be awesome if you just used current_month instead of March in making of your dashboard. And in backgound somehow current_month always keeps pointing to index for current month. In other words pointing to MyAwesomeLogs-2017-3-30 in month of march and pointing to MyAwesomeLogs-2017-4-30 in month of April.
HOW
Enter Elasticsearch aliases. Elasticsearch aliases allow you to assign an alias for a set of indices. So when you want to query something you can just issue the query against the alias instead of individually listing the target indices. Elasticsearch aliases allow us to solve the problem of rolling log indices. In the month of March if you have been creating an index everyday for storing logs then an alias of current_month targeted at MyAwesomeLogs-2017-04-* targets all the logs for the month of March!!! Write a query against the alias current_month and Elasticsearch in background hits all the indices for the month of March. Problem solved! Elasticsearch aliases here we come!
You see the process of managing Elasticsearch aliases is as manual as it can be. Every month you have to add the new indices to alias and remove the old ones from it to keep the alias meaningful. You can do it via the Kibana dev window or if you are a terminal ninja then you can use curl. There has to be a better way. And it is called……drum rolls please.
Curator allows you to automate management of Elasticsearch aliases. You might want to go through the previous post on installation and configuration of curator.
Let us create some indices first so that we can put curator through paces.
This is a quick and dirty shell script which does just that.
#!/bin/bash
CT="Content-Type:application/json"
DATE="$date"
if [ "$1" = "today" ]
then
PREV_DATE=$(date +%Y-%m-%d -d "$DATE + 0 day")
curl -XPUT 'http://localhost:9200/logs_'$PREV_DATE'?pretty' -H 'Content-Type: application/json' -d '{"settings" : {"number_of_shards" : 3,"number_of_replicas" : 2}}'
fi
if [ "$1" = "fwd" ]
then
for(( i=1; i<=$2; i++ ))
do
PREV_DATE=$(date +%Y-%m-%d -d "$DATE + $i day")
curl -XPUT 'http://localhost:9200/logs_'$PREV_DATE'?pretty' -H 'Content-Type: application/json' -d '{"settings" : {"number_of_shards" : 3,"number_of_replicas" : 2}}'
done
fi
if [ "$1" = "back" ]
then
for(( i=1; i<=$2; i++ ))
do
PREV_DATE=$(date +%Y-%m-%d -d "$DATE - $i day")
curl -XPUT 'http://localhost:9200/logs_'$PREV_DATE'?pretty' -H 'Content-Type: application/json' -d '{"settings" : {"number_of_shards" : 3,"number_of_replicas" : 2}}'
done
fi
As you can see it takes two parameters. First is either today, back or fwd. The second is the number of days.
./GenerateIndices.sh today will generate an index for the current date.
./GenerateIndices.sh back 5 will generate indices for last 5 days.
./GenerateIndices.sh fwd 5 will generate indices for coming 5 days.
Change the server ip and the port number to suit your setup.
Time to unleash Curator fire power.
To demonstrate the management of Elasticsearch aliases I need some indices first. Let use create indices for last 75 days.
./GenerateIndices.sh back 75
This a snipped version of indices now in our Elasticsearch cluster.
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open logs_2017-07-03 O2Ywiz-3TcyCNiSdUokC_w 3 2 0 0 1.1kb 390b green open logs_2017-06-27 rOSKf1Y1Tvylmqc-1pcOiw 3 2 0 0 1.1kb 390b green open logs_2017-07-12 FT63IMHmSWiC0aBvljGOxg 3 2 0 0 1.2kb 390b .... .... .... green open logs_2017-06-22 sug9WpNqR4KnXVeu3o7rdw 3 2 0 0 1.1kb 390b green open logs_2017-06-30 PRNzhqf0SaCcu9Rt3dTbgw 3 2 0 0 1.1kb 390b green open logs_2017-08-04 mSGSSf8WTlqXbseUDEfBEQ 3 2 0 0 1.1kb 390b
Now we will ask curator to create an alias called last_week which will target last week. As usual we will come up with an action file for the same. Let us call it action_alias.yml.
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True. If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
1:
action: alias
description: >-
Alias indices from last week, with a prefix of logs_ to 'last_week',
remove indices from the previous week.
options:
name: last_week
warn_if_no_indices: False
ignore_empty_list: True
disable_action: False
add:
filters:
- filtertype: pattern
kind: prefix
value: logs_
exclude:
- filtertype: period
source: name
range_from: -1
range_to: -1
timestring: '%Y-%m-%d'
unit: weeks
week_starts_on: sunday
remove:
filters:
- filtertype: pattern
kind: prefix
value: logs_
- filtertype: period
source: name
range_from: -2
range_to: -2
timestring: '%Y-%m-%d'
unit: weeks
week_starts_on: sunday
Line 14: The name of the alias is set to last_week
Line 16: This bit is to tell curator is that if no indices are found then please do not throw error and stop. Rather just carry on to the next action. By default curator will stop at first failed action.
Line 20: Setting the filter type to pattern so that I can target indices having a particular patter of naming. In my case all the indices will start with "logs_".
Line 21: Setting the filter to match all the indices which start with "logs_". You can use suffix to match the end if you want.
Line 22: This filter will now match all indices whose name starts with "logs_".
Eagle eyed readers would have noticed the problem by now. This filter will match all of my indices!!! All of them. From 2 years old ones to the one generated today. We need more filtering.
Line 18: Instructs the curator to add the indices which match the below filters.
Line 24: Setting the filter type to period. This will take what has been passed to it by previous filter and then perform time based filtering on it. It is called filter chaining.
Line 25: Setting the source to be of type name. This means that it will look for the time string in the name of the index.
Line 28: Since my indices will have name like logs_2017-08-17 I will configure the timestring to be hyphen seperated.
Line 29: Since my alias is all about last week I will set the unit to weeks. Curator now will do all date related calculations. This is beautiful. Check why.
Line 30: You can set it to monday. But sunday is more common.
In the lines below, e.g. line 26 and 27 I am asking curator to add the last week indices to the alias.
Line 26: range_from is a bit tricky to explain. Here it means start from the week before the current one. Negative means we are talking about past. Imagine negative side of number line.
Line 27: range_to tells where the period ends. Here it means that it ends at last week. So essentially I have chosen the last week. If this value was -2 then it means I have chosen last two weeks.
Line 31: Instructs the curator to remove from the alias the indices which match the below filters.
Line 38-39: I am asking curator to remove the indices which are two week old.
Time for action.
curator action_alias.yml
Result
2017-08-10 17:20:35,971 INFO Preparing Action ID: 1, "alias"
2017-08-10 17:20:35,979 INFO Trying Action ID: 1, "alias": Alias indices from last week, with a prefix of logs_ to 'last_week', remove indices from the previous week.
2017-08-10 17:20:36,043 INFO Updating aliases...
2017-08-10 17:20:36,044 INFO Alias actions: {'actions': [{'add': {'index': 'logs_2017-08-01', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-07-31', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-08-02', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-08-04', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-08-05', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-07-30', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-08-03', 'alias': 'last_week'}}]}
2017-08-10 17:20:36,081 INFO Action ID: 1, "alias" completed.
2017-08-10 17:20:36,081 INFO Job completed.
How to check that alias has been set?
curl -XGET 'http://localhost:9200/_cat/aliases?v&pretty'
Check the days on which the alias has been set. As per the date of writing it has picked up the last week correctly.
alias index filter routing.index routing.search last_week logs_2017-08-04 - - - last_week logs_2017-08-03 - - - last_week logs_2017-08-01 - - - last_week logs_2017-08-05 - - - last_week logs_2017-07-30 - - - last_week logs_2017-07-31 - - - last_week logs_2017-08-02 - - -
Now I will add another alias called this_week. I will add this section to the action_alias.yml.
action: alias
description: >-
Alias indices from current week, with a prefix of logs_ to 'this_week',
remove indices from the previous week.
options:
name: this_week
warn_if_no_indices: False
ignore_empty_list: True
disable_action: False
add:
filters:
- filtertype: pattern
kind: prefix
value: logs_
exclude:
- filtertype: period
source: name
range_from: 0
range_to: 0
timestring: '%Y-%m-%d'
unit: weeks
week_starts_on: sunday
remove:
filters:
- filtertype: pattern
kind: prefix
value: logs_
- filtertype: period
source: name
range_from: -1
range_to: -1
timestring: '%Y-%m-%d'
unit: weeks
week_starts_on: sunday
Time for action.
First create a index for today.
./GenerateIndices.sh today
curator action_alias.yml
Result
2017-08-10 17:30:29,401 INFO Preparing Action ID: 1, "alias"
2017-08-10 17:30:29,408 INFO Trying Action ID: 1, "alias": Alias indices from last week, with a prefix of logs_ to 'last_week', remove indices from the previous week.
2017-08-10 17:30:29,490 INFO Updating aliases...
2017-08-10 17:30:29,490 INFO Alias actions: {'actions': [{'add': {'index': 'logs_2017-08-01', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-07-31', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-08-02', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-08-04', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-08-05', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-07-30', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-08-03', 'alias': 'last_week'}}]}
2017-08-10 17:30:29,492 INFO Action ID: 1, "alias" completed.
2017-08-10 17:30:29,492 INFO Preparing Action ID: 2, "alias"
2017-08-10 17:30:29,496 INFO Trying Action ID: 2, "alias": Alias indices from current week, with a prefix of logs_ to 'this_week', remove indices from the previous week.
2017-08-10 17:30:29,567 INFO Updating aliases...
2017-08-10 17:30:29,568 INFO Alias actions: {'actions': [{'add': {'index': 'logs_2017-08-10', 'alias': 'this_week'}}, {'add': {'index': 'logs_2017-08-07', 'alias': 'this_week'}}, {'add': {'index': 'logs_2017-08-09', 'alias': 'this_week'}}, {'add': {'index': 'logs_2017-08-08', 'alias': 'this_week'}}, {'add': {'index': 'logs_2017-08-06', 'alias': 'this_week'}}]}
2017-08-10 17:30:29,582 INFO Action ID: 2, "alias" completed.
2017-08-10 17:30:29,582 INFO Job completed.
Check that alias has been set?
curl -XGET 'http://localhost:9200/_cat/aliases?v&pretty'
Check the days on which the alias this_week has been set. It is Thursday today and curator has put this_week alias correctly on Sun, Mon, Tue, Wed and Thrs.
alias index filter routing.index routing.search this_week logs_2017-08-06 - - - last_week logs_2017-08-03 - - - this_week logs_2017-08-07 - - - last_week logs_2017-08-04 - - - this_week logs_2017-08-10 - - - this_week logs_2017-08-08 - - - last_week logs_2017-07-30 - - - last_week logs_2017-08-05 - - - last_week logs_2017-08-02 - - - last_week logs_2017-07-31 - - - this_week logs_2017-08-09 - - - last_week logs_2017-08-01 - - -
Now I will add more to the action_alias.yml to create aliases this_day, this_month and last_month.
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True. If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
1:
action: alias
description: >-
Alias indices from last week, with a prefix of logs_ to 'last_week',
remove indices from the previous week.
options:
name: last_week
warn_if_no_indices: False
ignore_empty_list: True
disable_action: False
add:
filters:
- filtertype: pattern
kind: prefix
value: logs_
exclude:
- filtertype: period
source: name
range_from: -1
range_to: -1
timestring: '%Y-%m-%d'
unit: weeks
week_starts_on: sunday
remove:
filters:
- filtertype: pattern
kind: prefix
value: logs_
- filtertype: period
source: name
range_from: -2
range_to: -2
timestring: '%Y-%m-%d'
unit: weeks
week_starts_on: sunday
2:
action: alias
description: >-
Alias indices from current week, with a prefix of logs_ to 'this_week',
remove indices from the previous week.
options:
name: this_week
warn_if_no_indices: False
ignore_empty_list: True
disable_action: False
add:
filters:
- filtertype: pattern
kind: prefix
value: logs_
exclude:
- filtertype: period
source: name
range_from: 0
range_to: 0
timestring: '%Y-%m-%d'
unit: weeks
week_starts_on: sunday
remove:
filters:
- filtertype: pattern
kind: prefix
3:
action: alias
description: >-
Alias indices from last month, with a prefix of logs_ to 'last_month',
remove indices from the previous month.
options:
name: last_month
warn_if_no_indices: False
ignore_empty_list: True
disable_action: False
add:
filters:
- filtertype: pattern
kind: prefix
value: logs_
exclude:
- filtertype: period
source: name
range_from: -1
range_to: -1
timestring: '%Y-%m-%d'
unit: months
remove:
filters:
- filtertype: pattern
kind: prefix
value: logs_
- filtertype: period
source: name
range_from: -2
range_to: -2
timestring: '%Y-%m-%d'
unit: months
4:
action: alias
description: >-
Alias indices from current month, with a prefix of logs_ to 'this_month',
remove indices from the previous month.
options:
name: this_month
warn_if_no_indices: False
ignore_empty_list: True
disable_action: False
add:
filters:
- filtertype: pattern
kind: prefix
value: logs_
exclude:
- filtertype: period
source: name
range_from: 0
range_to: 0
timestring: '%Y-%m-%d'
unit: months
remove:
filters:
- filtertype: pattern
kind: prefix
value: logs_
- filtertype: period
source: name
range_from: -1
range_to: -1
timestring: '%Y-%m-%d'
unit: months
5:
action: alias
description: >-
Alias indices of current day, with a prefix of logs_ to 'this_day',
remove indices from the previous month.
options:
name: this_day
warn_if_no_indices: False
ignore_empty_list: True
disable_action: False
add:
filters:
- filtertype: pattern
kind: prefix
value: logs_
exclude:
- filtertype: period
source: name
range_from: 0
range_to: 0
timestring: '%Y-%m-%d'
unit: days
remove:
filters:
- filtertype: pattern
kind: prefix
value: logs_
- filtertype: period
source: name
range_from: -1
range_to: -1
timestring: '%Y-%m-%d'
unit: days
Run it
curator action_alias.yml
Result
2017-08-10 17:40:40,856 INFO Preparing Action ID: 1, "alias"
2017-08-10 17:40:40,863 INFO Trying Action ID: 1, "alias": Alias indices from last week, with a prefix of logs_ to 'last_week', remove indices from the previous week.
2017-08-10 17:40:40,966 INFO Updating aliases...
2017-08-10 17:40:40,966 INFO Alias actions: {'actions': [{'add': {'index': 'logs_2017-08-01', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-07-31', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-08-02', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-08-04', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-08-05', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-07-30', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-08-03', 'alias': 'last_week'}}]}
2017-08-10 17:40:40,969 INFO Action ID: 1, "alias" completed.
2017-08-10 17:40:40,969 INFO Preparing Action ID: 2, "alias"
2017-08-10 17:40:40,973 INFO Trying Action ID: 2, "alias": Alias indices from current week, with a prefix of logs_ to 'this_week', remove indices from the previous week.
2017-08-10 17:40:41,046 INFO Updating aliases...
2017-08-10 17:40:41,046 INFO Alias actions: {'actions': [{'add': {'index': 'logs_2017-08-10', 'alias': 'this_week'}}, {'add': {'index': 'logs_2017-08-07', 'alias': 'this_week'}}, {'add': {'index': 'logs_2017-08-09', 'alias': 'this_week'}}, {'add': {'index': 'logs_2017-08-08', 'alias': 'this_week'}}, {'add': {'index': 'logs_2017-08-06', 'alias': 'this_week'}}]}
2017-08-10 17:40:41,047 INFO Action ID: 2, "alias" completed.
2017-08-10 17:40:41,047 INFO Preparing Action ID: 3, "alias"
2017-08-10 17:40:41,051 INFO Trying Action ID: 3, "alias": Alias indices from last month, with a prefix of logs_ to 'last_month', remove indices from the previous month.
2017-08-10 17:40:41,126 INFO Updating aliases...
2017-08-10 17:40:41,126 INFO Alias actions: {'actions': [{'add': {'index': 'logs_2017-07-22', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-28', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-24', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-12', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-10', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-31', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-26', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-18', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-11', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-23', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-17', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-19', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-20', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-05', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-27', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-16', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-25', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-21', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-15', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-06', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-03', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-30', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-04', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-08', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-02', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-09', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-14', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-07', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-13', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-01', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-29', 'alias': 'last_month'}}]}
2017-08-10 17:40:41,255 INFO Action ID: 3, "alias" completed.
2017-08-10 17:40:41,255 INFO Preparing Action ID: 4, "alias"
2017-08-10 17:40:41,260 INFO Trying Action ID: 4, "alias": Alias indices from current month, with a prefix of logs_ to 'this_month', remove indices from the previous month.
2017-08-10 17:40:41,328 INFO Updating aliases...
2017-08-10 17:40:41,328 INFO Alias actions: {'actions': [{'add': {'index': 'logs_2017-08-01', 'alias': 'this_month'}}, {'add': {'index': 'logs_2017-08-10', 'alias': 'this_month'}}, {'add': {'index': 'logs_2017-08-07', 'alias': 'this_month'}}, {'add': {'index': 'logs_2017-08-09', 'alias': 'this_month'}}, {'add': {'index': 'logs_2017-08-02', 'alias': 'this_month'}}, {'add': {'index': 'logs_2017-08-04', 'alias': 'this_month'}}, {'add': {'index': 'logs_2017-08-05', 'alias': 'this_month'}}, {'add': {'index': 'logs_2017-08-08', 'alias': 'this_month'}}, {'add': {'index': 'logs_2017-08-06', 'alias': 'this_month'}}, {'add': {'index': 'logs_2017-08-03', 'alias': 'this_month'}}]}
2017-08-10 17:40:41,373 INFO Action ID: 4, "alias" completed.
2017-08-10 17:40:41,373 INFO Preparing Action ID: 5, "alias"
2017-08-10 17:40:41,377 INFO Trying Action ID: 5, "alias": Alias indices of current day, with a prefix of logs_ to 'this_day', remove indices from the previous month.
2017-08-10 17:40:41,438 INFO Updating aliases...
2017-08-10 17:40:41,438 INFO Alias actions: {'actions': [{'add': {'index': 'logs_2017-08-10', 'alias': 'this_day'}}]}
2017-08-10 17:40:41,447 INFO Action ID: 5, "alias" completed.
2017-08-10 17:40:41,447 INFO Job completed.
Now to check if the Elasticsearch aliases are setup as expected.
curl -XGET 'http://localhost:9200/_cat/aliases?v&pretty'
I will leave the counting to you.
alias index filter routing.index routing.search last_week logs_2017-08-03 - - - this_month logs_2017-08-03 - - - last_month logs_2017-07-21 - - - last_week logs_2017-08-05 - - - this_month logs_2017-08-05 - - - last_month logs_2017-07-28 - - - last_month logs_2017-07-04 - - - last_month logs_2017-07-13 - - - last_month logs_2017-07-18 - - - last_month logs_2017-07-24 - - - last_month logs_2017-07-06 - - - last_month logs_2017-07-05 - - - last_month logs_2017-07-01 - - - last_month logs_2017-07-11 - - - last_month logs_2017-07-27 - - - last_month logs_2017-07-02 - - - last_week logs_2017-08-04 - - - this_month logs_2017-08-04 - - - last_month logs_2017-07-15 - - - last_month logs_2017-07-17 - - - last_month logs_2017-07-31 - - - last_week logs_2017-07-31 - - - last_month logs_2017-07-16 - - - last_month logs_2017-07-14 - - - last_month logs_2017-07-22 - - - last_month logs_2017-07-12 - - - last_month logs_2017-07-19 - - - last_month logs_2017-07-23 - - - last_month logs_2017-07-09 - - - last_week logs_2017-08-02 - - - this_month logs_2017-08-02 - - - this_month logs_2017-08-07 - - - this_week logs_2017-08-07 - - - this_month logs_2017-08-08 - - - this_week logs_2017-08-08 - - - last_month logs_2017-07-29 - - - last_month logs_2017-07-07 - - - last_month logs_2017-07-25 - - - last_month logs_2017-07-10 - - - last_month logs_2017-07-30 - - - last_week logs_2017-07-30 - - - last_month logs_2017-07-26 - - - last_month logs_2017-07-03 - - - last_month logs_2017-07-08 - - - last_week logs_2017-08-01 - - - this_month logs_2017-08-01 - - - this_month logs_2017-08-06 - - - this_week logs_2017-08-06 - - - this_day logs_2017-08-10 - - - this_month logs_2017-08-10 - - - this_week logs_2017-08-10 - - - this_month logs_2017-08-09 - - - this_week logs_2017-08-09 - - - last_month logs_2017-07-20 - - -
Now that is nice. Curator is managing Elasticsearch aliases as expected. The only thing left is to schedule curator to run as a cron job. This is easy.
crontab -e
and add this line
* * * * * curator /home/elastic/curator_scripts/action_alias.yml
I am configuring it to run every minute for demo purpose.
Things to remember when managing your Elasticsearch aliases using curator:
1. Each action you see above has two parts. Add indices and remove indices. They are done atomically. So there is no time in between when the alias is not pointing to anything.
2. The action will fail if add or remove alias section fails. For example if you have index just for today. You will expect that setting the this_day alias should be a success. It will not be because the remove setion of action looks for the index of previous day. Since it does not find it the whole action of setting the this_day index fails. If this is a concern then separate add and remove in two separate actions.
And that is it. Curator will now take care of managing your Elasticsearch aliases on the indices.
Pingback: Taking Elasticsearch snapshots using Curator