I Am Your Father - Parent-Child Modeling in Elasticsearch

March 18, 2019

«No! That's not true! That's impossible!»

In 1980, The Empire Strikes Back revealed that Darth Vader was Luke's father, a twist that has become one of the most iconic or shocking movie moments to that time in movie history. It is one IMHO of the best movies in the Star Wars franchise of all time.

The parent-child relationship within the Elasticsearch universe can also be shocking if you have noticed the impact of the removal of mapping types in Elasticsearch. We are looking at the relationship from the supported versions (Elasticsearch v5.x) to the current present and announced changes of the future. This design is an advanced method of document modelling in Elasticsearch.

Stats for Nerds

Data Modeling for Elasticsearch

Elasticsearch is not a relational database. If you come from relational databases or SQL background, you need to change your thought process for modelling data concerning Elasticsearch. In SQL, you typically normalise your data. Elasticsearch is about search. Search requires different considerations. In Elasticsearch, you typically denormalise your data!

A flat world has its advantages :

Elasticsearch is about search efficiency, not storage efficiency.

Relationships Matter

Four common techniques exist for managing relational data in Elasticsearch:

  1. Denormalizing: Flatten your data
  2. Application-side joins: Run multiple queries on normalised data
  3. Nested objects: Store arrays of objects
  4. Parent-child relationships: Store multiple documents through joins

As long as you understand the tradeoffs of each technique, there's no difficulty to use it. We are going to look in detail about the parent-child relationship or in-depth grandparents and grandchildren. Sometimes relationships matter. It depends on your case.

The Problem

For the general audience to understand the challenges we might face, I am going to use examples to illustrate the parent-child relationship. The examples are a work of fiction. Names, characters, places and incidents either are products of the author’s imagination or are used fictitiously. Any resemblance to actual events or locales or persons is entirely coincidental.

Let us look into the following situation. Assuming we have a social music service. We store information about artists, their songs and the likes from users to measure their popularity. This model would result in the following relationship hierarchy.

Relationship Hierachy

Let's look into one complex example:

Music Example

Another information of interest is the count of plays of each song by the respective user. I am leaving that detail out because it does not provide more value for the explanation.

Other Examples

There are many examples, which also suits the parent-child demonstration.

Talent Management

If you want to keep track of your talents (employees), try for instance Talent Management with Elasticsearch.

Talent Management

Job Ads

Matchmaking between Workers and Employers.

Jobad

Team Composition

A football (soccer) team.

FC Barcelona

Benefits of Parent-Child Relationships

A parent-child relationship in Elasticsearch results in two documents that remain on the same index, or better index shard. You can't split parents and children into two separate indices.

The only case where the parent-child relationship makes sense is if your data contains a one-to-many relationship where one entity significantly outnumbers the other entity. The song (parent) won't change, but the likes (children) for that song may grow steadily.

If you would use a nested object for the above use case, the update is expensive. Updating a nested object requires a complete reindexing of the root object and a complete reindexing of all its nested objects!

Technical Facts

As mentioned above the parent and the children must live on the same shard. This physical constraint makes the query-time join faster if a search for the parent or child occurs.

The relationship join lives inside the memory. Each join, has_child or has_parent query adds a significant tax to your query performance.

Using Parent-Child Relationships

To use parent-child relationships we give you some examples with the Elasticsearch REST API. Use the Kibana Console (previously known as Marvel Sense) to execute the examples.

  1. Define the mapping
  2. Index some parent documents
  3. Index some child documents
  4. Query the documents

We have two indices:

  1. The data for parent-child relationships before version 5.6 is in index music.
  2. The data for recent Elasticsearch versions is in index music-5_6.

Mapping (before version 5.6)

If you are using an Elasticsearch version less than 5.6 read this section, else skip to the next section.

Parent-child documents have different mapping types. See below the index mapping.

PUT music
{
  "settings": { "number_of_shards": 1, "number_of_replicas": 0 },
  "mappings": {
    "artist": {
      "properties": { "name": { "type": "text" }}
    },
    "song": {
      "properties": { "title": { "type": "text" }},
      "_parent": { "type": "artist"}
    },
    "likes": {
      "properties": { "user": { "type": "keyword" }},
      "_parent": { "type": "song"}
    }
  }
}

To declare the parent we reference the (mapping) type of the parent. In the above example, the parent for song is artist.

Mapping (after version 5.6 to version 6.x)

Version 5.6 introduces a new field join type to declare parent-child relationships. We have to prevent multiple mapping types in order to use the join type. This mapping type setting allows us to easily upgrade the Elasticsearch index to Version 6 without any impacts or side-effects.

In the index settings we set "mapping.single_type": true. You don't need this setting if you are using an Elasticsearch version greater than 5.6. Elasticsearch 7 completely removes the mapping type information.

PUT /music-5_6
{
  "settings": {
    "number_of_shards": 1, "number_of_replicas": 0,
    "mapping.single_type": true
  },
  "mappings": {
    "doc": {
      "properties": {
        "artist": { "type": "text" },
        "song": { "type": "text" },
        "user": { "type": "keyword" },
        "artist_relations": {
          "type": "join",
          "relations": {
            "artist": "song",
            "song": "user"
          }
        }
      }
    }
  }
}

Using multiple levels of relations to replicate a relational model is not recommended. Each level of relation adds overhead at query time in terms of memory and computation. Each index may have only one join field, where all relations are declared. The artist_relations field on the above example is that join field.

Index data (ev < v5.6)

If you are using an Elasticsearch version less than 5.6 read this section, else skip to the next section.

Index artists (parents)

POST /music/artist/_bulk
{ "index": { "_id": "john-legend" }}
{ "name": "John Legend" }
{ "index": { "_id": "ariana-grande" }}
{ "name": "Ariana Grande" }

Index songs (children) and assign the respective parent

POST /music/song/_bulk
{ "index": { "_id": 1, "parent": "john-legend" }}
{ "title": "All of Me" }
{ "index": { "_id": 2, "parent": "john-legend" }}
{ "title": "Beauty and the Beast" }
{ "index": { "_id": 3, "parent": "ariana-grande" }}
{ "title": "Beauty and the Beast" }

Pay attention that the song Beauty and the Beast (a duet of John Legend and Ariana Grande) has two documents. This situation illustrates another issue. In an Elasticsearch parent-child relationship, a child can not have multiple parents. To comply with the example, we created two song documents and assigned each to both artists.

Index user likes to song (grandchildren).

POST /music/likes/_bulk
{ "index": { "_id": "l-1", "parent": 1 }}
{ "user": "Gabriel" }
{ "index": { "_id": "l-2", "parent": 1 }}
{ "user": "Berte" }
{ "index": { "_id": "l-3", "parent": 1 }}
{ "user": "Emma" }
{ "index": { "_id": "l-4", "parent": 2 }}
{ "user": "Berte" }
{ "index": { "_id": "l-5", "parent": 3 }}
{ "user": "Emma" }

Index data (ev > v5.6)

To index parents with the join field, we have to specify its relation type.

POST /music-5_6/doc/_bulk
{"index":{"_id":1}}
{"name":"John Legend","artist_relations":{"name":"artist"}}
{"index":{"_id":2}}
{"name":"Ariana Grande","artist_relations":{"name":"artist"}}

We can only index child documents with the respective parent. The parent is additionally used in the routing information, since as you can remember parent and child must reside on the same shard. The parent’s id serves as the routing value for the child document.

PUT music-5_6/doc/3?routing=1
{"song":"All of Me","artist_relations":{"name":"song","parent":1}}

PUT music-5_6/doc/4?routing=1
{"song":"Beauty and the Beast","artist_relations":{"name":"song","parent":1}}

PUT music-5_6/doc/5?routing=2
{"song":"Beauty and the Beast","artist_relations":{"name":"song","parent":2}}

To index the likes. A bulk request for the song All of Me.

POST music-5_6/doc/_bulk?routing=3
{"index":{"_id":"l-1"}}
{"user":"Gabriel","artist_relations":{"name":"user","parent":3}}
{"index":{"_id":"l-2"}}
{"user":"Berte","artist_relations":{"name":"user","parent":3}}
{"index":{"_id":"l-3"}}
{"user":"Emma","artist_relations":{"name":"user","parent":3}}

Simple Indexing for grandchildren

POST music-5_6/doc/l-4?routing=4
{"user":"Berte","artist_relations":{"name":"user","parent":4}}
POST music-5_6/doc/l-5?routing=5
{"user":"Emma","artist_relations":{"name":"user","parent":5}}

Finding Children by their Parents

Now if we want to know, how many songs for John Legend exists, we use the has_parent query. Therefore we search in the mapping type song.

Finding Children by their Parents Photo by Daniel Cheung on Unsplash

Version < v5.6

If you are using an Elasticsearch version less than 5.6 read this section, else skip to the next section.

GET /music/song/_search
{"query":{"has_parent":{"type":"artist","query":{"match":{"name":"John Legend"}}}}}

The above search request let Elasticsearch return these songs:

{
  "hits": {
    "total": 2,
    "max_score": 1,
    "hits": [
      {
        "_index": "music",
        "_type": "song",
        "_id": "1",
        "_score": 1,
        "_routing": "john-legend",
        "_parent": "john-legend",
        "_source": {
          "title": "All of Me"
        }
      },
      {
        "_index": "music",
        "_type": "song",
        "_id": "2",
        "_score": 1,
        "_routing": "john-legend",
        "_parent": "john-legend",
        "_source": {
          "title": "Beauty and the Beast"
        }
      }
    ]
  }
}

Search for all user-likes of a song.

GET /music/likes/_search
{"query":{"has_parent":{"type":"song","query":{"match":{"title":"all of me"}}}}}

Version >= v.5.6

Search for all songs (child) of an artist (parent).

GET music-5_6/_search
{
  "query": {
    "has_parent": {
      "parent_type": "artist",
      "query": { "match": { "name": "John Legend" } }
    }
  }
}

Elasticsearch yields this resultset:

{
  "hits": {
    "total": 2,
    "max_score": 1,
    "hits": [
      {
        "_index": "music-5_6",
        "_type": "doc",
        "_id": "3",
        "_score": 1,
        "_routing": "1",
        "_source": {
          "song": "All of Me",
          "artist_relations": { "name": "song", "parent": 1 }
        }
      },
      {
        "_index": "music-5_6",
        "_type": "doc",
        "_id": "4",
        "_score": 1,
        "_routing": "1",
        "_source": {
          "song": "Beauty and the Beast",
          "artist_relations": { "name": "song", "parent": 1 }
        }
      }
    ]
  }
}

Search for grandchildren, in our example, search for all user-likes (grandchild of the artist) of a song (child of the artist).

GET music-5_6/_search
{
  "query": {
    "has_parent": {
      "parent_type": "song",
      "query": {
        "match": { "song": "all of Me" }
      }
    }
  }
}

Elasticsearch returns this response:

{
  "hits": {
    "total": 3, "hits": [
      {
        "_source": { "user": "Gabriel", "artist_relations": { "name": "user", "parent": 3 }}
      },
      {
        "_source": { "user": "Berte", "artist_relations": { "name": "user", "parent": 3 }}
      },
      {
        "_source": { "user": "Emma", "artist_relations": { "name": "user", "parent": 3 }}
      }
    ]
  }
}

Finding Parents by their Children

The has_child query and filter can be used to find parent documents based on the contents of their children.

Searching for all artists (parents) that have one to ten (min_children and max_children) songs.

GET music/_search
{
  "query": {
    "has_child": {
      "type": "song",
      "min_children": 1, "max_children": 10, 
      "query": { "match_all": {} }
    }
  }
}

This will return:

{"hits": {"total":2,"hits":[
  {"_index":"music","_type":"artist","_id":"john-legend","_score":1,"_source":{"name":"John Legend"}},
  {"_index":"music","_type":"artist","_id":"ariana-grande","_score":1,"_source":{"name":"Ariana Grande"}}]}
}

Return Parents and Children

Having the artists we know that have songs, but we do not know which songs caused the hit. Use the inner_hits query to get the relevant children from the has_child query.

GET music/_search
{
  "query": {
    "has_child": {
      "type": "song",
      "min_children": 1, "max_children": 10, 
      "query": { "match_all": {} },
      "inner_hits": {}
    }
  }
}

This response shows the artists with their songs.

{"hits":{"total":2,"hits":[
  {"_source":{"name":"John Legend"},"inner_hits":{
     "song":{"hits":{"total":2,"hits":[
       {"_source":{"title":"All of Me"}},
       {"_source":{"title":"Beauty and the Beast"}}]}}}},
  {"_source":{"name":"Ariana Grande"},"inner_hits":{
     "song":{"hits":{"total":1,"max_score":1,"hits":[
       {"_source":{"title":"Beauty and the Beast"}}]}}}}]}}

Accessing a Child Document

The Elasticsearch REST API is going to change. Evaluate the respective versions. We have multiple mapping types enforce the need to address the type in the URL. To access a child document, we must know its parent and provide the routing information.

Search with multiple mapping types (Elasticsearch < v5.6):

GET music/song/1?routing=1
{
  "_index": "music",
  "_type": "song",
  "_id": "1",
  "_version": 3,
  "_routing": "john-legend",
  "_parent": "john-legend",
  "found": true,
  "_source": {
    "title": "All of Me"
  }
}

See below the example for current versions.

GET music-5_6/doc/3?routing=1
{
  "_index": "music-5_6",
  "_type": "doc",
  "_id": "3",
  "_version": 3,
  "_routing": "1",
  "found": true,
  "_source": {
    "song": "All of Me",
    "artist_relations": {
      "name": "song",
      "parent": 1
    }
  }
}

Updating Child Documents

One of the significant benefits of a parent-child relationship is the ability to modify a child object independent of the parent. To update a document, we must also know the parent and provide the routing information.

For instance change the song name for the following song, because it is a remake of a previous version:

For version, less than 5.6 provide the mapping type.

POST music/song/2/_update?routing=ariana-grande
{
  "doc": { "song": "Beauty and the Beast (2017)" }
}
POST music/song/3/_update?routing=ariana-grande
{
  "doc": { "song": "Beauty and the Beast (2017)" }
}

For version greater than 5.6.

POST music-5_6/doc/5/_update?routing=2
{
  "doc": { "song": "Beauty and the Beast (2017)"}
}

The song document is in the middle, but it does not affect the artist and the user-likes.

Aggregations

Dealing with parent-child relationships also adds complexity. You can also aggregate over child documents. Following full monty example could be a typical question in the Elastic Certified Engineer exam. We have two song documents for the song Beauty and the Beast. One is assigned to the artist John Legend and the other to Ariana Grande. Now we would like to know the count of user likes and see the user names that liked that song.

Strom-Trooper in Desert Photo by Daniel Cheung on Unsplash

GET music/_search
{
  "query":{"bool":{
  "must":[{"match":{"title":"Beauty and the Beast"}}],
  "should":[{"has_child":{"type":"likes","query":{
    "match_all":{}},"inner_hits":{}}}]}
  },
  "aggs":{"user_likes":{
    "children":{"type":"likes"}}}
}

The response from Elasticsearch:

{
  "took": 2,
  "hits": {
    "total": 2,
    "hits": [
      {
        "_source": { "title": "Beauty and the Beast" },
        "inner_hits": {
          "likes": {
            "hits": { "total": 1,"hits": [ { "_source": { "user": "Berte" } } ] }
          }
        }
      },
      {
        "_source": { "title": "Beauty and the Beast" },
        "inner_hits": {
          "likes": {
            "hits": { "total": 1, "hits": [ { "_source": { "user": "Emma" } } ] }
          }
        }
      }
    ]
  },
  "aggregations": {
    "user_likes": { "doc_count": 2 }
  }
}

Summary

Using parent-child relationships in Elasticsearch is an advanced method. Use it only when needed. A parent-child relationship searches slower and has more needs for system resources. If you can, always upgrade at least to Version 5.6 - otherwise, your implementation will and upgrade to Version 6 breaks. Avoid multiple mapping types if you can. The time you invest now is handsomely rewarded in the future; otherwise, anticipate a major migration or upgrade challenge. One possible outcome could be to stay on the specific Elasticsearch version, which is IMHO unwise.

Parent-child relationships have their advantages. As long as you are aware of the tradeoffs like the physical storage constraint of parent and child document and added complexity, you can efficiently use it. Try to keep the relationships levels simple. Having more than 3 levels makes it hard to maintain.

Literature

Find below some official articles and reference documentation.

About the author: Vinh Nguyên

Loves to code, hike and mostly drink black coffee. Favors Apache Kafka, Elasticsearch, Java Development and 80's music.

Comments
Join us