You Complete Me - Custom Region Maps for Kibana
In my past Crime Mapping article, I introduced Coordinate Maps in Kibana. In this article, I continue with Regions Maps in Kibana and how to extend Kibana with custom Region Maps. We are looking into several possibilities for hosting custom region maps.
The Need for Region Maps
Coordinate Maps are a good solution if you have an appropriate amount of coordinates to show. If they are too few or too many, it is counterproductive. Furthermore having too much density makes the visualisation unfavourable. Region Maps in Kibana are a better solution to represent a whole area or region. See the identical information represented as Coordinate and Region Map as a sneak preview for a future article about bike rental in New York City by Nadine Piveteau.
Elastic Map Service
TLDR: 20 years ago, as a young aspiring Software Engineer, I worked on an MIS (Management Information System) Project that has to provide the feature of displaying geographical regions of each respective subsidiary of the holding. Roughly it was divided between Germany, Division West (Western Europe and America) and Division East (East Europe and Asia). So we had to compose multiple regions to an organisational unit.
Twenty years ago, Open Source also existed but hadn't got that offering as it has today. Today Open Source is well established and accepted. In the past, we had to implement the features and spent much time in creating these region maps. To accomplish that I had to draw the geographical maps by hand with a pencil and digitally transform it into compatible images. Today Kibana of the Elastic Stack offers that functionality for free! Elastic has a fantastic offering of region maps with the Elastic Maps Service. It comes for free and could have saved me much work in the past. I think we don't appreciate Open Source enough and take everything for granted. If the Elastic Map Service does not provide the map you need, you can extend Kibana and host your region map.
Region Map Usage
See below a region map example for Switzerland served by the Elastic Map Service. It illustrates the publications made by each canton for today (2019-02-19). The data is from the Schweizerisches Handelsamtblatt, the Swiss Official Gazette of Commerce.
Another government use case is to present the latest voting results. See below an example for the recent vote 626 Zersiedelungsinitiative. The region map illustrates the highest (Schaffhausen) and lowest participation (Glarus).
Integrate Custom Region Maps
For the simplicity of my example, I take an existing region map of the planning bureau New York City, which serves our purpose. It contains several layers of regional information. The first information is the neighbourhood like Tribeca, Soho, Upper West Side, Harlem etc. Each district is part of a more significant region called borough. New York City consists of the boroughs Manhattan, Staten Island, Bronx, Brooklyn and Queens.
GeoJSON
GeoJSON is an open standard format designed for representing simple geographical features, along with their non-spatial attributes, based on the JSON format.
Following code excerpt is essential:
{
"type": "Feature",
"properties": {
"ntacode": "MN12",
"shape_area": "34379942.3628",
"county_fips": "061",
"ntaname": "Upper West Side",
"shape_leng": "29160.2062868",
"boro_name": "Manhattan",
"boro_code": "1"
}
}
We are planning to use the ntaname, boro_name and boro_cd fields in Kibana to match the data to the map region. They appear in the following Kibana configuration.
GeoJSON also contains the geometry of your region(s). With Open Source tools like QGIS or geojson.io you can easily define your region maps. See below an example with geojson.io for the location Eglisau (Switzerland).
Hosting Your Maps
There are several possibilities to host your custom maps.
- For local development the
http-serverpackage is the fastest way for development and localised testing. - To serve the maps within the organisation Nginx or the Apache Web Server are good choices.
- Outside your organisation you can use AWS S3 to host geo-maps.
CORS
CORS (Cross-origin resource sharing) is an important issue we have to overcome. The browser security model behaviour is simple. Browsers do not allow Javascript to load or post any content cross-origin. Serving the maps, we have to enable known origins, and this would be typically the domain-name of the request. For the simplicity of our example, we allow everything.
One possible pitfall to avoid, Kibana doesn't like mixed content, i.e. accessing Kibana over HTTPS with X-Pack security and accessing the maps via HTTP makes it impossible to use custom region maps. Using plain HTTP on all network activity is ok for development and evaluation but not recommended for production usage. Therefore always serve the maps with HTTPS to avoid problems in production environments.
Local
To serve locally static files, we take a small, lightweight HTTP server. Install it with yarn or npm.
yarn add http-server
# npm install http-server -g
Change to the directory of your map files and run this command:
http-server --cors='*' -p 8000
Access your maps under http://localhost:8000.
In-House
To have a production setup you can either use a web server like Nginx or Apache HTTP Server. See below an example configuration for Nginx to host the maps under port 8000 over https. We add some custom headers for Kibana.
server {
listen 8000 ssl http2 default_server;
listen [::]:8000 ssl http2 default_server;
location / {
if ($request_method = 'OPTIONS') {
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'kbn-version,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
add_header 'Access-Control-Max-Age' 1728000;
add_header 'Content-Type' 'text/plain charset=UTF-8';
add_header 'Content-Length' 0;
add_header 'Access-Control-Allow-Headers' 'Authorization';
add_header 'Access-Control-Allow-Credentials' 'true';
return 204;
}
if ($request_method = 'POST') {
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'kbn-version,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
}
if ($request_method = 'GET') {
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'kbn-version,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
}
}
include snippets/self-signed.conf;
include snippets/ssl-params.conf;
}
Cloud
You could also easily host your region maps on AWS S3 if you have no requirements or restrictions for hosting it in the cloud. S3 allows CORS and you have to set it up for your bucket explicitly. Copy the map files to your bucket. Go to the Permissions tab of your S3 bucket and add following CORS configuration to enable map serving from AWS S3.
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>HEAD</AllowedMethod>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>POST</AllowedMethod>
<MaxAgeSeconds>1728000</MaxAgeSeconds>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>
Kibana Configuration
Add to the Kibana configuration the region map information. Switch to HTTPS if you have X-Pack security enabled!
regionmap:
layers:
- name: "NYC Neighborhoods"
#url: "http://localhost:8000/nyc/NYC_Neighborhood.geojson"
url: "https://geo.mimacom.com/nyc/NYC_Neighborhood.geojson"
fields:
- name: "ntaname"
description: "Neighborhood Name"
- name: "boro_name"
description: "Borough Name"
- name: "boro_code"
description: "Borough Code"
server.cors: true
server.cors.origin: ['*']
#server.cors.origin: ['*.mimacom.com']
We take the address geo.mimacom.com as an example and allow traffic from any origin. You may restrict it to your company network. The CORS configuration is not documented in the official Kibana reference, but in the excellent blog post by Mark Walkom.
Now you can select the custom map in the region map visualisation of Kibana.
Summary
Kibana offers many region maps through the Elastic Map Service. If you need a custom map, you can easily create one and serve it on your own or cloud infrastructure. Essential is to allow CORS on both endpoints (Kibana and Maps host). The process of creating custom regions is simple. In this article, I did not elaborate in detail on it, but we have plenty of GIS experts, who can do that in no time and no effort. Doing is actually cheaper than explaining.
The Elastic Stack is going further with the GIS support. As a preview to the Elastic Stack 7, Kibana 7 provides a new map view which allows having multiple GIS layers. See below an example with airports.
Airport origins or departing flights are red, and airport destinations or arriving flights are green. Some points are both: origin and destination and have the colour dark green.