Spring microservices monitoring: /metrics endpoint and ELK, Part II: Improvements

 Introduction

 As we saw in the first part of this series, it is essential necessary to monitor the health of our microservices, and  to improve the tool we developed which helps us get metrics and forward them to an Elasticsearch instance. In this chapter we will pursue the question of how can we improve the processing of the result of the metrics endpoint?

Continue Reading

Novedades con Elastic Stack 5.x (Spanish Version)

En octubre del año pasado ha sido liberada de manera oficial la versión 5.0.0. Desde entonces, han ido saliendo periódicamente actualizaciones y nuevas versiones del stack (la última mientras escribo este artículo es la 5.2.1, del 14 de febrero de 2017).

Sin embargo, desde mimacom hemos visto que a muchos de nuestros clientes les cuesta dar el paso y decidirse a cambiar a estas últimas versiones. Dado que han venido cargadas de novedades en múltiples aspectos, creemos que es un paso lógico y que todos deberíamos de plantearnos llegado el momento.

Continue Reading

Summary Grassroot Smart Cities

Introduction

Cities are complex systems, characterized by massive numbers of interconnected citizens. In the advent of new technologies, such as IoT or mobile applications, various domains within these systems can be enhanced to support the decisions of its citizens, businesses, or administration.

Ideally, a city follows a vision how to improve its smartness to generate value for its citizens. However, this top-down approach is often not applied consequently in many cities. Reasons are a reluctance to invest in new approaches due to a critical financial state or a conservative view averse to an approach that makes only sense regionally. A typical city already manages a big portfolio of IT solutions and it’s difficult to know in advance, what technologies and innovations will be used by end users.

Continue Reading

Components and usage of Spring Data Elasticsearch (Part 2)

In the previous post we discussed how to read data in chunks from a relational database as a Spring Batch process using Spring Boot runner. This post will explain how to import that data into Elasticsearch.

Spring Boot provides out of the box support for Elasticsearch that saves a lot of configuration time so we are going to take full advantage of that. However, in real world application, you will need to use the Elasticsearch Java Client that requires additional instantiating and configuring. When no configuration is provided, the Spring Boot integration for Elasticsearch will automatically create an embedded Elasticsearch instance which is very useful for testing….

Continue Reading

Import Relational Database Data to Elasticsearch with Spring Batch (Part 1)

Recently I had a customer project that required transferring large amounts of data from a relational database to the NoSQL database that is Elasticsearch in order to take advantage of its famous fast searching capabilities. Elasticsearch is part of the ELK stack that is released and maintained by Elastic.co. The abbreviation ELK stand for Elasticsearch, Logstash, and Kibana.

The easiest way to transfer data from a traditional relational database into Elasticsearch is by using the “L” in the “ELK” stack: Logstash. Unfortunately, Logstash has some limitations, and one of those limitations is directly related to reading records from a relational database because, although database entry and Elasticsearch entry may seem very similar, it’s not possible to match single database entry to a single Elasticsearch document. This difference originates from the fact that Elasticsearch doesn’t use the notion of “relations” between its “records”, but instead it uses flat documents structure to store its data and flat documents have no relations between each other. In addition, the customer database have several millions entries which made the situation even more complicated. So, instead of using Logstash, a decision was made to write our own importer that was going to use batch processing and bulk writing into Elasticsearch.

Continue Reading