Article: Structured Logging with Rsyslog and Elasticsearch Author: Radu Gheorghe
As more and more organizations are starting to use our Performance Monitoring and Search Analytics services, we have more and more logs from various components that make up these applications. So what do we do? Do we just keep logging everything to files, rotating them, and grepping them when we need to troubleshoot something? There must be something better we can do! And indeed, there is – so much so, that we’ll soon be launching Logsene – a Log Analytics service to complement SPM. When your applications generate a lot of logs, you’d probably want to make some sense of them by searching and/or statistics. Here’s when structured logging comes in handy, and I would like to share some thoughts and configuration examples of how you could use a popular syslog daemon like rsyslog to handle both structured and unstructured logs. Then I’m going to look at how you can take those logs, format them in JSON, and index them with Elasticsearch – for some fast and easy searching and statistics. If you are going to Berlin Buzzwords this year and you are into logging, Logstash, ElasticSearch, or Kibana, I’ll be talking about them in my JSON logging with ElasticSearch presentation.