Spring microservices monitoring: /metrics endpoint and ELK, Part II: Improvements

 Introduction

 As we saw in the first part of this series, it is essential necessary to monitor the health of our microservices, and  to improve the tool we developed which helps us get metrics and forward them to an Elasticsearch instance. In this chapter we will pursue the question of how can we improve the processing of the result of the metrics endpoint?

Continue Reading

QCON London 2017

I found this experience really nice – to be able to get some insights on how people have solved big issues and how they got pwned.  What is more, it was good to see that I am personally going in the right direction (or at least, let’s say, getting on the trends)

These are technologies and trends that got my attention as the most used:

  •  Technologies/languages/tools:
    • Go
    • ELK
    • Luigi/Airflow – Tools for pipelining jobs
    • Redis
    • Hadoop
    • Kafka
    • Java 9
  • Trends:
    • Data Science
      •  Machine learning for prediction and fraud detection
    • Microservices
      • NodeJS

I chose to follow two main paths, with some extra things from here and there: Machine Learning/Data science, Microservices, and let’s call it “miscellaneous”. Below, an overview of the most interesting things:

Continue Reading

Novedades con Elastic Stack 5.x (Spanish Version)

En octubre del año pasado ha sido liberada de manera oficial la versión 5.0.0. Desde entonces, han ido saliendo periódicamente actualizaciones y nuevas versiones del stack (la última mientras escribo este artículo es la 5.2.1, del 14 de febrero de 2017).

Sin embargo, desde mimacom hemos visto que a muchos de nuestros clientes les cuesta dar el paso y decidirse a cambiar a estas últimas versiones. Dado que han venido cargadas de novedades en múltiples aspectos, creemos que es un paso lógico y que todos deberíamos de plantearnos llegado el momento.

Continue Reading

Components and usage of Spring Data Elasticsearch (Part 2)

In the previous post we discussed how to read data in chunks from a relational database as a Spring Batch process using Spring Boot runner. This post will explain how to import that data into Elasticsearch.

Spring Boot provides out of the box support for Elasticsearch that saves a lot of configuration time so we are going to take full advantage of that. However, in real world application, you will need to use the Elasticsearch Java Client that requires additional instantiating and configuring. When no configuration is provided, the Spring Boot integration for Elasticsearch will automatically create an embedded Elasticsearch instance which is very useful for testing….

Continue Reading

Spring microservices monitoring: /metrics endpoint and ELK

Introduction

On the actual vibe of microservices using, it becomes more and more necessary to monitorize the health of our distributed systems. Here we will expose a few ways to do it, concentrating us in Spring boot applications. Hands on!

Current tools available

On the ELK community already exist two easy solutions to poll for the /metrics endpoint – the Logstash Http_poller input plugin and the community beat springbeat. Both work on a similar way (configure a URL and voilá! data is forwarded to Elasticsearch) but in different contexts, the former is a plugin part of Logstash, while the latter is an independent beat application. And both seem to be nice options, if we want to know the state of microservices with a defined URL or a single instance or if we use a service discovery with a gateway pattern, i.e. zuul+Eureka. And both share the same limitation: There is no way to configure service discovery to poll for all the instances of a service (As of now…). Whether or not this limitation is a problem, depends on what we want and what we have. Do we have a single instance of each microservice? or if we have multiple instances, do we want to know the general aggregated state of them? (I would put an alarm here, because some data is tightly coupled to the instance or it’s JVM and aggregating it might be a problem). An option to solve this limitation is to write a little client, a Spring boot application that will use the service discovery provided by Spring Cloud Eureka to get all the instances, poll them, process data, and forward it to the ELK stack.

Continue Reading