The Elk stack is a serious competitor in the log analysis market. ELK Stands for Elastic, LogStash and Kibana, three individual products that work together to collect, index and display the information in various ways useful for the end user.
Now let us look at the 3 major components of the stack.
Elastic –
Elastic is based in Lucene search engine, and includes automatic clustering, sharding, REST api support for managment. That being said there are few core concepts that you have to be familiar with before using Elastic.
Features:
Elastic is near realtime. – Meaning there is a slight latency from the time a document or file is indexed to until it becomes searchable.
Out of the box cluster support – Nodes are configured to participate in a cluster by using the same cluster name across nodes. The default cluster name is ‘elasticsearch’ and this makes it all the more important to define a unique name across different environments for maintenance
Node – A single server that is part of your cluster is called as a node. It can store data and takes part in the clusters indexing and search capabilities. It is imperative nodes are named property as they would be used for administering the cluster. There is no limit to the number of nodes in a cluster. Even a single node elastic instance would form a single node cluster with the given name or ‘elasticsearch’ default name.
Index – A collection of documents that have similar characteristics. For e.g. an index of member data, an index of client data etc. It is indentified by a name, all in lower case, and this name is used for all operations on the index like indexing, searching, update and deletion of documents. There are no limits to the number of indexes one can define in a cluster.
Type – A logical category/partition of the index. The semantics is upto the creator of the type. A type is defined for documents that have a set of common fields.
Document – A basic unit of information that can be indexed. The document is normally expressed in JSON format. Any number of documents can be stored within an index or type.
Shards and Replicas – Elasticsearch provides the ability to subdivide your index into multiple pieces called shards. The number of shards per index can be defined at creation time, and each shard is a fully functional and independent index that can be hosted on any node in the cluster.
Logstash –
Opensource data collection engine. Logstash was originally written to collect logs from multiple sources. Its current capabilities extend much more than data collection. With Logstash you collect data from multiple sources, format them, and send the formatted messages to multiple destinations. Any event or log that is captured can be enriched using GROK filters and transformed to the correct format that allows elastic to index the messages. In a nutshell it is the ingestion workhorse for ElasticSearch with a pluggable architecture and community driven and developer friendly plugin ecosystem.
Features:
Multiple formats – Handle all types of logging data from syslogs, apache logs, applications logs, windows event log etc
Log Forwarding – Secure log forwarding capabilites with Filebeat
Metrics – Collect metrics from Netflow, JMX, etc over TCP and UDP protocols
Events – Transform HTTP requests to events
Query apps – Polling feature for creating events for REST end points like a spring boot application
Data stores – Ingest data from any relational or noSql data store
Message Queues – Consume data from messaging queues like Apache Kafka, RabbitMQ, etc
New data – Capture data from sensors and IoT devices
Enriched Data – Easily enrich data being parsed and processed through logstash
Kibana –
An opensource analytics and visualization platform specifically designed to integrate and work with Elastic Search. Kibana allows a user to search, interact and view data that is stored in elastic search.
It allows the user to perform advanced data analytics and visualize data in a variety of charts, tables, maps etc. It makes it real easy to understand and render large amounts of data using a simple browser based interface with no config or development skills required to create detailed charts or visualizations.
Advertisements