Why DENIc has chosen Elasticsearch
Reliable features, easy handling and a reasonable price ultimately tipped the scales.
For a few months now, DENIC has its own Cloud. Like for any infrastructure, logging was an issue in the development process from the very beginning. Two options had to be evaluated: Elasticsearch, which was new to us, and Splunk Enterprise, with which we already were familiar.
The Requirements
The task to be fulfilled was to centrally collect logs from all components of the platform (hardware computers, network components, virtual machines (VMs), containers, etc.) and to make these searchable. Moreover, the solution we chose should smoothly fit in with our new container-based environment.
Initial Decision
We subjected the two eligible products to a proof of concept (POC). The POC results supported Elasticsearch and its web interface Kibana. At that time, Elasticsearch already was well established in the community, and thanks to the cloud native approach, could be integrated into the new Kubernetes cluster immediately in form of containers. Additionally, being an open source software, the tool presented a well-priced option for use on the DENIC Cloud platform.
Second Check
When the Cloud platform had reached a certain degree of maturity, we addressed the issue of logging again. The former infrastructure platform of DENIC had used virtual machines exclusively. Logging was performed with Splunk, a long-term market leader in this field. We asked ourselves which of the two logging tools we now knew was the best solution for the DENIC Cloud. Should we stay with "new" Elasticsearch or switch back to the former well-tried Splunk?
Splunk and Elasticsearch in Comparison
Basically, Elasticsearch and Splunk offer many features that are equivalent to each other. A plus of Splunk we identified was its good and intuitive search function which is able to handle even unstructured data like log entries of different systems efficiently. Another benefit was the easy migration of existing dashboards and reports to the new Cloud platform. Elasticsearch on the other hand was already well integrated into the new platform and operation and maintenance had proven easy and comfortable. The search function of Elasticsearch/Kibana worked well, too, but it benefited from log data that have been compiled in a structured form. But this was part of the migration, anyway. And of course, those who would use the new logging system would have to be trained in Elasticsearch.
We then compared the two options also with regard to deployment, configurability, and maintenance in general. It quickly turned out that Splunk would require much higher resources than Elasticsearch. In addition to that, various principles and mechanisms had apparently been retained in Splunk over the past years which from today's perspective conflict with use under cloud paradigms.
Since the new platform will continue to include the operation of virtual machines, it was desirable that the logging agent of each of the products could be integrated via an official repository. While Elasticsearch offers the most recent versions of the necessary applications, corresponding software packages of Splunk that can be installed automatically could not be found. The integration of Splunk into the system and session manager "systemd" was also nearly not existent, which was another factor that would have complicated the installation and operation.
The Decision
Even though we appreciated the intuitive search function of Splunk and the easy migration of data inventories from an older to a newer version was a plus, we finally dropped Splunk. All these benefits could not make up for the operative disadvantages. Therefore, we opted for Elasticsearch with its web interface Kibana in the end, the tool with which we had started off.
Our decision was supported by the fact that next to higher operational expenses, the Splunk license model also offered the poorer solution. We would have needed to vastly upgrade our licenses, which would have resulted in significantly higher costs. Our current Elasticsearch license, in contrast, is better priced and still has capacities for further growth of the platform. So the decision was backed also from a financial point of view: We will stay with Elasticsearch/Kibana.
The Implementation
The only thing that then remained to be done was to transfer the logging system originally started as a POC to productive operation. With only a few adjustments, we were able to configure Elasticsearch in such a way that all logs could be reliably collected, processed and searched even under peak load situations.
Conclusion:With Elasticsearch/Kibana, DENIC now has a future-proof, lean and easily maintainable logging system.
