Search is something that any application should have. Shay Banon was guided by this fact when he developed a product called Compass in 2004. It was built on top of Apache Lucene, a great search engine library entirely written in Java.
In 2010, Shay completely rewrote Compass with two main objectives:
- Distributed from the ground up in its design;
- Easily used by any other programming language.
And, he called it ElasticSearch.
ElasticSearch is distributed and scales horizontally:
ElasticSearch provides REST APIs for communicating with a cluster over HTTP and allows client applications to be written in any language.
I used a high-level .NET client called Nest. There is solid documentation for Nest on the net, but the examples are, as usual, very simple, and for requirements that are more complex, you need to experiment a little bit. It is recommended, if you need to write queries, to do it first in Kibana, and only then to translate such a code into Nest.
Elastic Stack Overview
The Elastic Stack is a collection of products with ElasticSearch at the heart. It is a group of open source products from Elastic designed to help users take data from any type of source and in any format and search, analyze, and visualize that data in real time.
These products are: Beats, Logstash, ElasticSearch and Kibana.
Beats – single-purpose data shippers. They send data from hundreds or thousands of machines and systems to Logstash or ElasticSearch. The Beats family consists of several members, who are in charge of different types of data: Filebeat, Metricbeat, Packetbeat, Winlogbeat, Auditbeat, Functionbeat and Heartbeat.
Logstash – really great tool that dynamically ingests, transforms, and ships data regardless of format or complexity. In the previous blog, I mentioned that it is very important to do data synchronization between primary storage and ElasticSearch. The advent of Logstash has helped to overcome this problem, and that we no longer have to write custom code for synchronization. It allows us to filter and transform data in a way that suits us for search. The name of this tool associates us that it ingests data almost from logs, but that is not exactly the case. In a large number of cases, Logstash is used to transfer and synchronize data from the primary database, e.g. MS SQL Server, and ElasticSearch.
Kibana – great analytics and visualization platform designed to work with ElasticSearch. You use Kibana to search, view, and interact with data stored in ElasticSearch indices. You can easily perform advanced data analysis and visualize your data in a variety of charts, tables, and maps.
It is very likely that you will use at least three of these four products in your projects. In all future examples, I will use Kibana for query writing, data analysis, monitoring and visualization.
Next
In the next blog, we will pay more attention to ElasticSearch CRUD operations: index, create, read, update and delete. We will explain the structure of documents stored in the ElasticSearch and how it differs in relation to data storage in traditional relational database systems.
Author: Budimir Bogićević, Senior Software Engineer at enjoy.ing
