This short tutorial on Elasticsearch dynamic templates using match_mapping_type will teach you how to control the mappings of the dynamically added fields in Elasticsearch. If you are reading this then it means that you understand the importance of mappings and how to manage them using templates. Elasticsearch dynamic templates are a natural progression of templates. At the end of this tutorial you should be a power user of this critical but often ignored feature.
NOTE: Instead of writing a big post I will be dividing this tutorial into three parts. Elasticsearch dynamic templates based on the type detected by Elasticsearch(match_mapping_type), Elasticsearch dynamic templates based on the name of the fields(match, unmatch, match_pattern) and Elasticsearch dynamic templates based on full path of the field(path_match and path_unmatch)
WHY
So what is the deal with Elasticsearch dynamic templates? Regular templates help you define the mappings for the fields YOU KNOW will be the part of data which gets pushed in. You know the field names and are (rightly) going to decide if they are mapped as long, double or keyword etc. However we also know that by default Elasticsearch will add the NEW fields to your existing data (It is known as Dynamic mapping).
Who then decides what mappings are to be applied to these newly added fields? Answer is Elasticsearch. And this can bite you in some really weird mapping errors. Enter the Elasticsearch dynamic templates feature. Using it you can decide what mapping is to be applied on newly discovered added fields. And you can do it on basis of their name, their detected type, by path or regex (No life without regex).
HOW
Best way to show is via an example. So fire up the Elasticsearch instance and start the kibana. I am using Elasticsearch version 7.2.0.
Based on detected type of the fields
Let us start with the one where we decide on mappings based purely on the type detected by Elasticsearch. For example Elasticsearch errs on the side of caution and by default maps integer to long so as not to lose any data. Now if you are sure that the numbers you are dealing with are small then you can force Elasticsearch to map the numbers as integer. Immediate benefit is that it can lead to space savings. And in some rare cases it prevent errors when doing metrics over data.
Let us just put in an index. Then we check the mapping created.
PUT sampleindex/_doc/1
{
"Value":123
}
GET sampleindex/_mapping
The is the Elasticsearch generated default mapping for the index.
{
"sampleindex" : {
"mappings" : {
"properties" : {
"Value" : {
"type" : "long"
}
}
}
}
}
That got mapped as long. I want it to be integer. So time to put in an Elasticsearch dynamic template.
PUT _template/sample_dynamic_template
{
"index_patterns": ["sample*"],
"mappings": {
"dynamic_templates": [
{
"handle_integers": {
"match_mapping_type": "long",
"mapping": {
"type": "integer"
}
}
}
]
}
}
A bit of explanation on the template above.
Line 03 : The index_patterns as usual decides which index this is going to be applied to.
Line 07 : The “handle_integers” is a user defined name. You can use any other name.
Line 08: “match_mapping_type” decides which mapping (detected by Elasticsearch) will be overridden. Here is is long.
Line 10 : I tell Elasticsearch to override/ignore the mapping detected and map the field as integer.
Now to check.
We will clean up everything and start over.
DELETE sampleindex
Put in the index again and then check the mapping.
PUT sampleindex/_doc/1
{
"Value":123
}
GET sampleindex/_mapping
Check the mapping for Value.
{
"sampleindex" : {
"mappings" : {
"dynamic_templates" : [
{
"handle_integers" : {
"match_mapping_type" : "long",
"mapping" : {
"type" : "integer"
}
}
}
],
"properties" : {
"Value" : {
"type" : "integer"
}
}
}
}
}
Success !! It is mapped as integer.
This is the most common usage of the Elasticsearch dynamic template. And it is also the most far reaching one. It does not matter what the field is. Only thing needed is that Elasticsearch detected type matches what we have set in the dynamic template. The field then gets mapped as laid out in dynamic template.
Many a times you will want a finer control over things. What if you want to put the mapping only on those fields whose name matched to a certain pattern? Like map all the fields starting with YearOfBirth to simple integer? Possible? Yes. And that is what we cover in the next post.