How often should a new log index be created? Once a day, Once a week, Once a month? A simple search in Google would return various responses each arguing the pros and cons of creating indexes daily or weekly. Lets look at how to do that with logstash.
My take on that is “once a month” index is the best option. The following is my reasoning for this.
Pros:
- Easier back up with a monthly index
- Simple to create snapshots and restore
- One index to backup externally on NAS or other storage outside of ELK stack
- Works well where the retention policy for active logs is 30 days or 60 days etc
- Allows complete logs for the whole month to be restored in one go.
Cons:
- Potential for large index sizes
- Have to restore the whole index to search for a specific day of the month
- Backup and restore of these indexes can take some time in slower systems or single node instances
Creating a monthly rolling index file
In order to create a new index each month automatically ensure you have the following setting in your logstash config file for e.g. devlogstash.conf.
elasticsearch{
index => “dev-logstash-%{+YYYY.MM}”
}
Creating a weekly rolling index file
The weekly name format would be YYYY.ww as in 2017.01 for the first week of the year in number. Config setting would be as shown below.
elasticsearch{
index => “dev-logstash-%{+YYYY.ww}”
}
Creating a daily rolling index file
Just add MM.DD instead of WW to the setting above to create a daily rolling index as shown below.
elasticsearch{
index => “dev-logstash-%{+YYYY.MM.DD}”
}
Creating a Year, month and week rolling index file
And that could be defined as YYYY.MM.ww to create a weekly rolling index as shown below.
elasticsearch{
index => “dev-logstash-%{+YYYY.MM.ww}”
}