Digital.
Innovation.
Engineers

Components and usage of Spring Data Elasticsearch (Part 2)

In the previous post we discussed how to read data in chunks from a relational database as a Spring Batch process using Spring Boot runner. This post will explain how to import that data into Elasticsearch.

Spring Boot provides out of the box support for Elasticsearch that saves a lot of configuration time so we are going to take full advantage of that. However, in real world application, you will need to use the Elasticsearch Java Client that requires additional instantiating and configuring. When no configuration is provided, the Spring Boot integration for Elasticsearch will automatically create an embedded Elasticsearch instance which is very useful for testing….

Continue Reading

Spring microservices monitoring: /metrics endpoint and ELK

Introduction

On the actual vibe of microservices using, it becomes more and more necessary to monitorize the health of our distributed systems. Here we will expose a few ways to do it, concentrating us in Spring boot applications. Hands on!

Current tools available

On the ELK community already exist two easy solutions to poll for the /metrics endpoint – the Logstash Http_poller input plugin and the community beat springbeat. Both work on a similar way (configure a URL and voilá! data is forwarded to Elasticsearch) but in different contexts, the former is a plugin part of Logstash, while the latter is an independent beat application. And both seem to be nice options, if we want to know the state of microservices with a defined URL or a single instance or if we use a service discovery with a gateway pattern, i.e. zuul+Eureka. And both share the same limitation: There is no way to configure service discovery to poll for all the instances of a service (As of now…). Whether or not this limitation is a problem, depends on what we want and what we have. Do we have a single instance of each microservice? or if we have multiple instances, do we want to know the general aggregated state of them? (I would put an alarm here, because some data is tightly coupled to the instance or it’s JVM and aggregating it might be a problem). An option to solve this limitation is to write a little client, a Spring boot application that will use the service discovery provided by Spring Cloud Eureka to get all the instances, poll them, process data, and forward it to the ELK stack.

Continue Reading

Performance considerations when using Apache Kafka with SSL/TLS

Running a cluster of a distributed software such as Apache Kafka in a production environment will lead to operational concerns. One of these concerns is certainly security. Companies have varying guidelines and policies when it comes to security. This depends on various factors for example the type of application data that the system is processing or in which network infrastructure the system is running and what security concerns this zone implies on the servers that are running within.

A probable scenario is that such a cluster is run in an environment which dictates restrictiveness in regards to access control and protocol security. Access control means that identified clients communicate with the cluster only. When looking at protocol security the company or project might have a guideline that clients must use a secure protocol like SSL/TLS. Furthermore, communication among the cluster servers itself might also have to be secured.

Continue Reading

Import Relational Database Data to Elasticsearch with Spring Batch (Part 1)

Recently I had a customer project that required transferring large amounts of data from a relational database to the NoSQL database that is Elasticsearch in order to take advantage of its famous fast searching capabilities. Elasticsearch is part of the ELK stack that is released and maintained by Elastic.co. The abbreviation ELK stand for Elasticsearch, Logstash, and Kibana.

The easiest way to transfer data from a traditional relational database into Elasticsearch is by using the “L” in the “ELK” stack: Logstash. Unfortunately, Logstash has some limitations, and one of those limitations is directly related to reading records from a relational database because, although database entry and Elasticsearch entry may seem very similar, it’s not possible to match single database entry to a single Elasticsearch document. This difference originates from the fact that Elasticsearch doesn’t use the notion of “relations” between its “records”, but instead it uses flat documents structure to store its data and flat documents have no relations between each other. In addition, the customer database have several millions entries which made the situation even more complicated. So, instead of using Logstash, a decision was made to write our own importer that was going to use batch processing and bulk writing into Elasticsearch.

Continue Reading

Testing an Angular 2 Service with a Mocked HTTP Interface

Angular 2 is the successor of the famous and broadly used AngularJS Javascript Framework, for building dynamic web applications. Due to the huge success and popularity of its predecessor, some client projects started using Angular 2 before its official release.This resulted in Angular 2 becoming one of the most famous platforms for web and hybrid application development. Although this framework is relatively new there are already a lot of tutorials and examples explaining how to develop an Angular 2 application. However, there is one aspect of Angular 2 that is not widely covered, but is still very important for every project: Writing tests. The command used to generate the default test cases is “ng generate” and it’s contained in the Angular 2 CLI. However, it only generates the boilerplate code that is needed for each test case. The actual tests have to be created separately and the API is still experimental, so examples are often outdated. In example, most of the available tutorials use the “beforeEachProviders” method, which was removed in an earlier Angular 2 pre-release and was replaced by “TestBed.configureTestingModule({providers:…})” with a completely different signature. In the following example we will test a simple “ItemService” which executes one HTTP GET request against a REST endpoint.

Continue Reading