In this post we are going to look at an ELK stack architecture for a small scale implementation. Keep in mind that this architecture is suitable for a small sized on-prem installation and the index capacity is determined by the hardware and disk space availability.
How often should a new log index be created? Once a day, Once a week, Once a month? A simple search in Google would return various responses each arguing the pros and cons of creating indexes daily or weekly. Lets look at how to do that with logstash.
Logstash comes with a bunch of plugins for input, filter and output processing pipelines. When monitoring logs using a platform like ELK the most requested feature is email notifications in case of sever error conditions or issues. There are many options to enable sending emails using custom python scripts, Elastic watcher, elastalert, etc.