Managing Elasticsearch aliases using Curator

By | August 15, 2017

This tutorial on managing Elasticsearch aliases using Curator will help you to manage your Elasticsearch aliases better. There are not many detailed tutorials on this topic and hence this post. I hope that at the end of this tutorial you will appreciate the power curator brings to your hands.

WHY

We cannot keep storing logs in an ELK stack. Old logs are phased out in stages. New logs take their place. If you have got analytics running on top of these logs (which you should) then there will be this issue of changing indices name.

In world of logging the name of logs follows a pattern like MyAwesomeLogs-2017-12-30. Now depending on the configuration indices for storing logs might be created on monthly, weekly or daily basis. To visualise the data in logs mostly Kibana is used. And dashboards in Kibana are based on index names. So if you create a Dashboard on the most common error for month of March then in April you have to change the dashboard source of data to the index created in April.

Managing elasticsearch aliases
Will it not be awesome if you just used current_month instead of March in making of your dashboard. And in backgound somehow current_month always keeps pointing to index for current month. In other words pointing to MyAwesomeLogs-2017-3-30 in month of march and pointing to MyAwesomeLogs-2017-4-30 in month of April.

HOW

Enter Elasticsearch aliases. Elasticsearch aliases allow you to assign an alias for a set of indices. So when you want to query something you can just issue the query against the alias instead of individually listing the target indices. Elasticsearch aliases allow us to solve the problem of rolling log indices. In the month of March if you have been creating an index everyday for storing logs then an alias of current_month targeted at MyAwesomeLogs-2017-04-* targets all the logs for the month of March!!! Write a query against the alias current_month and Elasticsearch in background hits all the indices for the month of March. Problem solved! Elasticsearch aliases here we come!
Managing elasticsearch aliases
You see the process of managing Elasticsearch aliases is as manual as it can be. Every month you have to add the new indices to alias and remove the old ones from it to keep the alias meaningful. You can do it via the Kibana dev window or if you are a terminal ninja then you can use curl. There has to be a better way. And it is called……drum rolls please.
managing elasticsearch aliases

Curator allows you to automate management of Elasticsearch aliases. You might want to go through the previous post on installation and configuration of curator.
Let us create some indices first so that we can put curator through paces.
This is a quick and dirty shell script which does just that.

#!/bin/bash
CT="Content-Type:application/json"
DATE="$date"

if [ "$1" = "today" ]
then
        PREV_DATE=$(date +%Y-%m-%d -d "$DATE + 0 day")
        curl -XPUT 'http://localhost:9200/logs_'$PREV_DATE'?pretty' -H 'Content-Type: application/json' -d '{"settings" : {"number_of_shards" : 3,"number_of_replicas" : 2}}'
fi

if [ "$1" = "fwd" ]
then
        for(( i=1; i<=$2; i++ ))
        do
                PREV_DATE=$(date +%Y-%m-%d -d "$DATE + $i day")
                curl -XPUT 'http://localhost:9200/logs_'$PREV_DATE'?pretty' -H 'Content-Type: application/json' -d '{"settings" : {"number_of_shards" : 3,"number_of_replicas" : 2}}'
        done
fi


if [ "$1" = "back" ]
then
        for(( i=1; i<=$2; i++ ))
        do
                PREV_DATE=$(date +%Y-%m-%d -d "$DATE - $i day")
                curl -XPUT 'http://localhost:9200/logs_'$PREV_DATE'?pretty' -H 'Content-Type: application/json' -d '{"settings" : {"number_of_shards" : 3,"number_of_replicas" : 2}}'
        done
fi

As you can see it takes two parameters. First is either today, back or fwd. The second is the number of days.
./GenerateIndices.sh today will generate an index for the current date.
./GenerateIndices.sh back 5 will generate indices for last 5 days.
./GenerateIndices.sh fwd 5 will generate indices for coming 5 days.

Change the server ip and the port number to suit your setup.

Time to unleash Curator fire power.
Managing elasticsearch aliases
To demonstrate the management of Elasticsearch aliases I need some indices first. Let use create indices for last 75 days.

./GenerateIndices.sh back 75

This a snipped version of indices now in our Elasticsearch cluster.

health status index                  uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   logs_2017-07-03        O2Ywiz-3TcyCNiSdUokC_w   3   2          0            0      1.1kb           390b
green  open   logs_2017-06-27        rOSKf1Y1Tvylmqc-1pcOiw   3   2          0            0      1.1kb           390b
green  open   logs_2017-07-12        FT63IMHmSWiC0aBvljGOxg   3   2          0            0      1.2kb           390b
....
....
....
green  open   logs_2017-06-22        sug9WpNqR4KnXVeu3o7rdw   3   2          0            0      1.1kb           390b
green  open   logs_2017-06-30        PRNzhqf0SaCcu9Rt3dTbgw   3   2          0            0      1.1kb           390b
green  open   logs_2017-08-04        mSGSSf8WTlqXbseUDEfBEQ   3   2          0            0      1.1kb           390b

Now we will ask curator to create an alias called last_week which will target last week. As usual we will come up with an action file for the same. Let us call it action_alias.yml.

# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: alias
    description: >-
      Alias indices from last week, with a prefix of logs_ to 'last_week',
      remove indices from the previous week.
    options:
      name: last_week
      warn_if_no_indices: False
      ignore_empty_list: True
      disable_action: False
    add:
      filters:
      - filtertype: pattern
        kind: prefix
        value: logs_
        exclude:
      - filtertype: period
        source: name
        range_from: -1
        range_to: -1
        timestring: '%Y-%m-%d'
        unit: weeks
        week_starts_on: sunday
    remove:
      filters:
      - filtertype: pattern
        kind: prefix
        value: logs_
      - filtertype: period
        source: name
        range_from: -2
        range_to: -2
        timestring: '%Y-%m-%d'
        unit: weeks
        week_starts_on: sunday

Line 14: The name of the alias is set to last_week
Line 16: This bit is to tell curator is that if no indices are found then please do not throw error and stop. Rather just carry on to the next action. By default curator will stop at first failed action.
Line 20: Setting the filter type to pattern so that I can target indices having a particular patter of naming. In my case all the indices will start with "logs_".
Line 21: Setting the filter to match all the indices which start with "logs_". You can use suffix to match the end if you want.
Line 22: This filter will now match all indices whose name starts with "logs_".

Eagle eyed readers would have noticed the problem by now. This filter will match all of my indices!!! All of them. From 2 years old ones to the one generated today. We need more filtering.

Line 18: Instructs the curator to add the indices which match the below filters.

Line 24: Setting the filter type to period. This will take what has been passed to it by previous filter and then perform time based filtering on it. It is called filter chaining.
Line 25: Setting the source to be of type name. This means that it will look for the time string in the name of the index.
Line 28: Since my indices will have name like logs_2017-08-17 I will configure the timestring to be hyphen seperated.
Line 29: Since my alias is all about last week I will set the unit to weeks. Curator now will do all date related calculations. This is beautiful. Check why.
Line 30: You can set it to monday. But sunday is more common.

In the lines below, e.g. line 26 and 27 I am asking curator to add the last week indices to the alias.

Line 26: range_from is a bit tricky to explain. Here it means start from the week before the current one. Negative means we are talking about past. Imagine negative side of number line.
Line 27: range_to tells where the period ends. Here it means that it ends at last week. So essentially I have chosen the last week. If this value was -2 then it means I have chosen last two weeks.

Line 31: Instructs the curator to remove from the alias the indices which match the below filters.
Line 38-39: I am asking curator to remove the indices which are two week old.

Time for action.

curator action_alias.yml

Result

2017-08-10 17:20:35,971 INFO      Preparing Action ID: 1, "alias"
2017-08-10 17:20:35,979 INFO      Trying Action ID: 1, "alias": Alias indices from last week, with a prefix of logs_ to 'last_week', remove indices from the previous week.
2017-08-10 17:20:36,043 INFO      Updating aliases...
2017-08-10 17:20:36,044 INFO      Alias actions: {'actions': [{'add': {'index': 'logs_2017-08-01', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-07-31', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-08-02', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-08-04', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-08-05', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-07-30', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-08-03', 'alias': 'last_week'}}]}
2017-08-10 17:20:36,081 INFO      Action ID: 1, "alias" completed.
2017-08-10 17:20:36,081 INFO      Job completed.

How to check that alias has been set?

curl -XGET 'http://localhost:9200/_cat/aliases?v&pretty'

Check the days on which the alias has been set. As per the date of writing it has picked up the last week correctly.

alias     index           filter routing.index routing.search
last_week logs_2017-08-04 -      -             -
last_week logs_2017-08-03 -      -             -
last_week logs_2017-08-01 -      -             -
last_week logs_2017-08-05 -      -             -
last_week logs_2017-07-30 -      -             -
last_week logs_2017-07-31 -      -             -
last_week logs_2017-08-02 -      -             -

Now I will add another alias called this_week. I will add this section to the action_alias.yml.

action: alias
description: >-
  Alias indices from current week, with a prefix of logs_ to 'this_week',
  remove indices from the previous week.
options:
  name: this_week
  warn_if_no_indices: False
  ignore_empty_list: True
  disable_action: False
add:
  filters:
  - filtertype: pattern
    kind: prefix
    value: logs_
    exclude:
  - filtertype: period
    source: name
    range_from: 0
    range_to: 0
    timestring: '%Y-%m-%d'
    unit: weeks
    week_starts_on: sunday
remove:
  filters:
  - filtertype: pattern
    kind: prefix
    value: logs_
  - filtertype: period
    source: name
    range_from: -1
    range_to: -1
    timestring: '%Y-%m-%d'
    unit: weeks
    week_starts_on: sunday

Time for action.

First create a index for today.

./GenerateIndices.sh today
curator action_alias.yml

Result

2017-08-10 17:30:29,401 INFO      Preparing Action ID: 1, "alias"
2017-08-10 17:30:29,408 INFO      Trying Action ID: 1, "alias": Alias indices from last week, with a prefix of logs_ to 'last_week', remove indices from the previous week.
2017-08-10 17:30:29,490 INFO      Updating aliases...
2017-08-10 17:30:29,490 INFO      Alias actions: {'actions': [{'add': {'index': 'logs_2017-08-01', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-07-31', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-08-02', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-08-04', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-08-05', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-07-30', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-08-03', 'alias': 'last_week'}}]}
2017-08-10 17:30:29,492 INFO      Action ID: 1, "alias" completed.
2017-08-10 17:30:29,492 INFO      Preparing Action ID: 2, "alias"
2017-08-10 17:30:29,496 INFO      Trying Action ID: 2, "alias": Alias indices from current week, with a prefix of logs_ to 'this_week', remove indices from the previous week.
2017-08-10 17:30:29,567 INFO      Updating aliases...
2017-08-10 17:30:29,568 INFO      Alias actions: {'actions': [{'add': {'index': 'logs_2017-08-10', 'alias': 'this_week'}}, {'add': {'index': 'logs_2017-08-07', 'alias': 'this_week'}}, {'add': {'index': 'logs_2017-08-09', 'alias': 'this_week'}}, {'add': {'index': 'logs_2017-08-08', 'alias': 'this_week'}}, {'add': {'index': 'logs_2017-08-06', 'alias': 'this_week'}}]}
2017-08-10 17:30:29,582 INFO      Action ID: 2, "alias" completed.
2017-08-10 17:30:29,582 INFO      Job completed.

Check that alias has been set?

curl -XGET 'http://localhost:9200/_cat/aliases?v&pretty'

Check the days on which the alias this_week has been set. It is Thursday today and curator has put this_week alias correctly on Sun, Mon, Tue, Wed and Thrs.

alias     index           filter routing.index routing.search
this_week logs_2017-08-06 -      -             -
last_week logs_2017-08-03 -      -             -
this_week logs_2017-08-07 -      -             -
last_week logs_2017-08-04 -      -             -
this_week logs_2017-08-10 -      -             -
this_week logs_2017-08-08 -      -             -
last_week logs_2017-07-30 -      -             -
last_week logs_2017-08-05 -      -             -
last_week logs_2017-08-02 -      -             -
last_week logs_2017-07-31 -      -             -
this_week logs_2017-08-09 -      -             -
last_week logs_2017-08-01 -      -             -

Now I will add more to the action_alias.yml to create aliases this_day, this_month and last_month.

# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: alias
    description: >-
      Alias indices from last week, with a prefix of logs_ to 'last_week',
      remove indices from the previous week.
    options:
      name: last_week
      warn_if_no_indices: False
      ignore_empty_list: True
      disable_action: False
    add:
      filters:
      - filtertype: pattern
        kind: prefix
        value: logs_
        exclude:
      - filtertype: period
        source: name
        range_from: -1
        range_to: -1
        timestring: '%Y-%m-%d'
        unit: weeks
        week_starts_on: sunday
    remove:
      filters:
      - filtertype: pattern
        kind: prefix
        value: logs_
      - filtertype: period
        source: name
        range_from: -2
        range_to: -2
        timestring: '%Y-%m-%d'
        unit: weeks
        week_starts_on: sunday
  2:
    action: alias
    description: >-
      Alias indices from current week, with a prefix of logs_ to 'this_week',
      remove indices from the previous week.
    options:
      name: this_week
      warn_if_no_indices: False
      ignore_empty_list: True
      disable_action: False
    add:
      filters:
      - filtertype: pattern
        kind: prefix
        value: logs_
        exclude:
      - filtertype: period
        source: name
        range_from: 0
        range_to: 0
        timestring: '%Y-%m-%d'
        unit: weeks
        week_starts_on: sunday
    remove:
      filters:
      - filtertype: pattern
        kind: prefix
  3:
    action: alias
    description: >-
      Alias indices from last month, with a prefix of logs_ to 'last_month',
      remove indices from the previous month.
    options:
      name: last_month
      warn_if_no_indices: False
      ignore_empty_list: True
      disable_action: False
    add:
      filters:
      - filtertype: pattern
        kind: prefix
        value: logs_
        exclude:
      - filtertype: period
        source: name
        range_from: -1
        range_to: -1
        timestring: '%Y-%m-%d'
        unit: months
    remove:
      filters:
      - filtertype: pattern
        kind: prefix
        value: logs_
      - filtertype: period
        source: name
        range_from: -2
        range_to: -2
        timestring: '%Y-%m-%d'
        unit: months
  4:
    action: alias
    description: >-
      Alias indices from current month, with a prefix of logs_ to 'this_month',
      remove indices from the previous month.
    options:
      name: this_month
      warn_if_no_indices: False
      ignore_empty_list: True
      disable_action: False
    add:
      filters:
      - filtertype: pattern
        kind: prefix
        value: logs_
        exclude:
      - filtertype: period
        source: name
        range_from: 0
        range_to: 0
        timestring: '%Y-%m-%d'
        unit: months
    remove:
      filters:
      - filtertype: pattern
        kind: prefix
        value: logs_
      - filtertype: period
        source: name
        range_from: -1
        range_to: -1
        timestring: '%Y-%m-%d'
        unit: months
  5:
    action: alias
    description: >-
      Alias indices of current day, with a prefix of logs_ to 'this_day',
      remove indices from the previous month.
    options:
      name: this_day
      warn_if_no_indices: False
      ignore_empty_list: True
      disable_action: False
    add:
      filters:
      - filtertype: pattern
        kind: prefix
        value: logs_
        exclude:
      - filtertype: period
        source: name
        range_from: 0
        range_to: 0
        timestring: '%Y-%m-%d'
        unit: days
    remove:
      filters:
      - filtertype: pattern
        kind: prefix
        value: logs_
      - filtertype: period
        source: name
        range_from: -1
        range_to: -1
        timestring: '%Y-%m-%d'
        unit: days

Run it

curator action_alias.yml

Result

2017-08-10 17:40:40,856 INFO      Preparing Action ID: 1, "alias"
2017-08-10 17:40:40,863 INFO      Trying Action ID: 1, "alias": Alias indices from last week, with a prefix of logs_ to 'last_week', remove indices from the previous week.
2017-08-10 17:40:40,966 INFO      Updating aliases...
2017-08-10 17:40:40,966 INFO      Alias actions: {'actions': [{'add': {'index': 'logs_2017-08-01', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-07-31', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-08-02', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-08-04', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-08-05', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-07-30', 'alias': 'last_week'}}, {'add': {'index': 'logs_2017-08-03', 'alias': 'last_week'}}]}
2017-08-10 17:40:40,969 INFO      Action ID: 1, "alias" completed.
2017-08-10 17:40:40,969 INFO      Preparing Action ID: 2, "alias"
2017-08-10 17:40:40,973 INFO      Trying Action ID: 2, "alias": Alias indices from current week, with a prefix of logs_ to 'this_week', remove indices from the previous week.
2017-08-10 17:40:41,046 INFO      Updating aliases...
2017-08-10 17:40:41,046 INFO      Alias actions: {'actions': [{'add': {'index': 'logs_2017-08-10', 'alias': 'this_week'}}, {'add': {'index': 'logs_2017-08-07', 'alias': 'this_week'}}, {'add': {'index': 'logs_2017-08-09', 'alias': 'this_week'}}, {'add': {'index': 'logs_2017-08-08', 'alias': 'this_week'}}, {'add': {'index': 'logs_2017-08-06', 'alias': 'this_week'}}]}
2017-08-10 17:40:41,047 INFO      Action ID: 2, "alias" completed.
2017-08-10 17:40:41,047 INFO      Preparing Action ID: 3, "alias"
2017-08-10 17:40:41,051 INFO      Trying Action ID: 3, "alias": Alias indices from last month, with a prefix of logs_ to 'last_month', remove indices from the previous month.
2017-08-10 17:40:41,126 INFO      Updating aliases...
2017-08-10 17:40:41,126 INFO      Alias actions: {'actions': [{'add': {'index': 'logs_2017-07-22', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-28', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-24', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-12', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-10', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-31', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-26', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-18', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-11', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-23', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-17', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-19', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-20', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-05', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-27', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-16', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-25', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-21', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-15', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-06', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-03', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-30', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-04', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-08', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-02', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-09', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-14', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-07', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-13', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-01', 'alias': 'last_month'}}, {'add': {'index': 'logs_2017-07-29', 'alias': 'last_month'}}]}
2017-08-10 17:40:41,255 INFO      Action ID: 3, "alias" completed.
2017-08-10 17:40:41,255 INFO      Preparing Action ID: 4, "alias"
2017-08-10 17:40:41,260 INFO      Trying Action ID: 4, "alias": Alias indices from current month, with a prefix of logs_ to 'this_month', remove indices from the previous month.
2017-08-10 17:40:41,328 INFO      Updating aliases...
2017-08-10 17:40:41,328 INFO      Alias actions: {'actions': [{'add': {'index': 'logs_2017-08-01', 'alias': 'this_month'}}, {'add': {'index': 'logs_2017-08-10', 'alias': 'this_month'}}, {'add': {'index': 'logs_2017-08-07', 'alias': 'this_month'}}, {'add': {'index': 'logs_2017-08-09', 'alias': 'this_month'}}, {'add': {'index': 'logs_2017-08-02', 'alias': 'this_month'}}, {'add': {'index': 'logs_2017-08-04', 'alias': 'this_month'}}, {'add': {'index': 'logs_2017-08-05', 'alias': 'this_month'}}, {'add': {'index': 'logs_2017-08-08', 'alias': 'this_month'}}, {'add': {'index': 'logs_2017-08-06', 'alias': 'this_month'}}, {'add': {'index': 'logs_2017-08-03', 'alias': 'this_month'}}]}
2017-08-10 17:40:41,373 INFO      Action ID: 4, "alias" completed.
2017-08-10 17:40:41,373 INFO      Preparing Action ID: 5, "alias"
2017-08-10 17:40:41,377 INFO      Trying Action ID: 5, "alias": Alias indices of current day, with a prefix of logs_ to 'this_day', remove indices from the previous month.
2017-08-10 17:40:41,438 INFO      Updating aliases...
2017-08-10 17:40:41,438 INFO      Alias actions: {'actions': [{'add': {'index': 'logs_2017-08-10', 'alias': 'this_day'}}]}
2017-08-10 17:40:41,447 INFO      Action ID: 5, "alias" completed.
2017-08-10 17:40:41,447 INFO      Job completed.

Now to check if the Elasticsearch aliases are setup as expected.

curl -XGET 'http://localhost:9200/_cat/aliases?v&pretty'

I will leave the counting to you.

alias      index           filter routing.index routing.search
last_week  logs_2017-08-03 -      -             -
this_month logs_2017-08-03 -      -             -
last_month logs_2017-07-21 -      -             -
last_week  logs_2017-08-05 -      -             -
this_month logs_2017-08-05 -      -             -
last_month logs_2017-07-28 -      -             -
last_month logs_2017-07-04 -      -             -
last_month logs_2017-07-13 -      -             -
last_month logs_2017-07-18 -      -             -
last_month logs_2017-07-24 -      -             -
last_month logs_2017-07-06 -      -             -
last_month logs_2017-07-05 -      -             -
last_month logs_2017-07-01 -      -             -
last_month logs_2017-07-11 -      -             -
last_month logs_2017-07-27 -      -             -
last_month logs_2017-07-02 -      -             -
last_week  logs_2017-08-04 -      -             -
this_month logs_2017-08-04 -      -             -
last_month logs_2017-07-15 -      -             -
last_month logs_2017-07-17 -      -             -
last_month logs_2017-07-31 -      -             -
last_week  logs_2017-07-31 -      -             -
last_month logs_2017-07-16 -      -             -
last_month logs_2017-07-14 -      -             -
last_month logs_2017-07-22 -      -             -
last_month logs_2017-07-12 -      -             -
last_month logs_2017-07-19 -      -             -
last_month logs_2017-07-23 -      -             -
last_month logs_2017-07-09 -      -             -
last_week  logs_2017-08-02 -      -             -
this_month logs_2017-08-02 -      -             -
this_month logs_2017-08-07 -      -             -
this_week  logs_2017-08-07 -      -             -
this_month logs_2017-08-08 -      -             -
this_week  logs_2017-08-08 -      -             -
last_month logs_2017-07-29 -      -             -
last_month logs_2017-07-07 -      -             -
last_month logs_2017-07-25 -      -             -
last_month logs_2017-07-10 -      -             -
last_month logs_2017-07-30 -      -             -
last_week  logs_2017-07-30 -      -             -
last_month logs_2017-07-26 -      -             -
last_month logs_2017-07-03 -      -             -
last_month logs_2017-07-08 -      -             -
last_week  logs_2017-08-01 -      -             -
this_month logs_2017-08-01 -      -             -
this_month logs_2017-08-06 -      -             -
this_week  logs_2017-08-06 -      -             -
this_day   logs_2017-08-10 -      -             -
this_month logs_2017-08-10 -      -             -
this_week  logs_2017-08-10 -      -             -
this_month logs_2017-08-09 -      -             -
this_week  logs_2017-08-09 -      -             -
last_month logs_2017-07-20 -      -             -

Now that is nice. Curator is managing Elasticsearch aliases as expected. The only thing left is to schedule curator to run as a cron job. This is easy.

crontab -e

and add this line

* * * * * curator /home/elastic/curator_scripts/action_alias.yml

I am configuring it to run every minute for demo purpose.

Things to remember when managing your Elasticsearch aliases using curator:
1. Each action you see above has two parts. Add indices and remove indices. They are done atomically. So there is no time in between when the alias is not pointing to anything.
2. The action will fail if add or remove alias section fails. For example if you have index just for today. You will expect that setting the this_day alias should be a success. It will not be because the remove setion of action looks for the index of previous day. Since it does not find it the whole action of setting the this_day index fails. If this is a concern then separate add and remove in two separate actions.

And that is it. Curator will now take care of managing your Elasticsearch aliases on the indices.

One thought on “Managing Elasticsearch aliases using Curator

  1. Pingback: Taking Elasticsearch snapshots using Curator

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.