In this post we are going to look at an ELK stack architecture for a small scale implementation. Keep in mind that this architecture is suitable for a small sized on-prem installation and the index capacity is determined by the hardware and disk space availability.
The following diagram shows the various components that were used for a specific use case to index about 30 GB of data a day with a 20 day retention requirement.
ELK stack beat’s agents are installed on servers to collect logs, metrics and network packet data. The collected data is then piped through logstash instances where the data is transformed,tags added, and then sent to elastic nodes to be indexed
The diagram shows one instance of logstash for clarity purposes. In a production setup they would be multiple instances of logstash servers reading from beats agents and shipping to elastic instances.
When setting up elastic nodes, there should be at-least 3 master nodes to avoid split brain scenario. This was previously discussed in Elastic Zen Discovery post. There are no limits on the number of data nodes that can be added to the cluster provided there are enough hardware resources. It is also recommended to setup elastic nodes that would help with indexing data using ingest nodes along with master and data nodes.
Finally Kibana would be hosted in a separate server that is shown as Kibana in the diagram above. This can be single instance as Kibana works really well with even a single instance with multiple users at the same time.
The following is a general recommendation of hardware specs for each of the server nodes. The OS is Linux
Logstash and Kibana nodes:
Processor : 4 Core
RAM : 16 GB
HDD : 40GB
Elastic Data nodes:
Processor : 8 Core
RAM : 24GB
HDD : 250 GB SSD
Elastic Master nodes:
Processor : 4 Core
RAM : 16 GB
HDD : 120 GB SSD
NAS Storage:
HDD : 750 GB
More to follow in future articles.