The ELK Stack, and the Licensing Question Underneath It

The componentsWhat the three letters stand for, and what broke them

The ELK stack has three components with three jobs, and they are easier to keep straight if you follow the data rather than the letters.

Elasticsearch

Elasticsearch is a distributed search and analytics engine built on Apache Lucene. It stores data as JSON documents in indexes, splits each index into shards across the nodes of a cluster, and answers queries over a REST API.

New data is searchable in near real time, which is what makes it suit log data and monitoring.

Logstash

Logstash is the data processing pipeline. Each pipeline has three stages: inputs that receive data from sources such as files, syslog and Beats, filters that parse and transform it, and outputs that send it on, normally to Elasticsearch.

The grok filter, which turns an unstructured log line into named fields, is the plugin most log analysis depends on.

Kibana

Kibana is the visualization layer. It runs in a browser, searches the log data in Elasticsearch, and turns queries into charts, maps and dashboards that update in real time. Discover is the view for reading raw events during troubleshooting. Dashboards are what a monitoring screen or a weekly security review is built from.

Beats, the fourth component

Elastic's own description of the stack names four things, not three: it is "comprised of Elasticsearch, Kibana, Beats, and Logstash (also known as the ELK Stack)". Beats is the fourth, and it arrived after the acronym was already in circulation.

Beats are single purpose shippers. Elastic describes the platform as "single-purpose data shippers" that "send data from hundreds or thousands of machines and systems to Logstash or Elasticsearch". Note the or. Six are named:

That "or" is the part people miss. A Beat can write straight into Elasticsearch, which means Logstash is optional, which means a working deployment of the ELK stack often contains no L at all.

Logstash earns its place when the data needs real transformation on the way in: parsing an unstructured log format, enriching records from another source, routing to more than one destination. For shipping Windows event logs to a store, it is a component you can leave out.

The pipelineThe pipeline, in the order data moves

Data moves through the ELK stack in four stages, and each one is a place where a deployment goes wrong.

Collection. A Beat or an agent on the machine reads the log file, the event channel or the metric and sends it on. This is where volume is decided, and the mistake is shipping everything because disk was cheap when the estimate was made.

Ingest. Logstash, if present, parses each record into fields, for example turning a web server access log line into a client address, a status code and a response time. Unparsed logs still store and still search as text, and they will not aggregate, so a dashboard built on them counts nothing useful.

Storage. Elasticsearch holds the log data as indexed documents, ready for search and analysis. Indexes have a retention policy or they do not have an end, and this is the single most common way an ELK stack falls over: nobody set one.

Display. Kibana searches the data in Elasticsearch and builds visualizations and dashboards from it. It holds no data of its own, so a Kibana that shows nothing is almost never a Kibana problem.

Installing the ELK stack, in outline

The components install separately, on Linux, on Windows or in containers, and the sensible order runs against the flow of data. Install Elasticsearch first and confirm it answers on port 9200. Install Kibana next and point it at Elasticsearch. Kibana serves its interface on port 5601.

Then install Beats on the machines to collect from. Add Logstash only if the data needs it, listening for Beats on port 5044 by default.

Keep every component on the same version, because the stack is released and tested as a set. Configuration lives in one file per component: elasticsearch.yml, kibana.yml, and the pipeline configuration files Logstash reads.

A single node is enough to learn on. Production clusters run at least three nodes, so that losing one loses neither data nor the cluster.

Use casesWhat the ELK stack is used for

The ELK stack is a general tool for data analytics: collecting, searching and visualizing time stamped data from any system that produces it. In practice four use cases account for nearly every deployment.

Log management and analysis. Centralized logging is the original use case. Logs from servers, applications and network devices land in one place, so a search across every system takes seconds instead of a login to each machine.

Infrastructure and application monitoring. Metricbeat and application logs feed dashboards of CPU, memory, response time and error rates. The same data serves troubleshooting in real time and performance analysis after the fact.

Security analytics. Authentication events, firewall logs and audit data in one searchable store is the core of a SIEM. Security teams use the stack as one, or as a long retention log store beside a commercial one, because it gives an analyst the search and the timeline during an investigation.

Search. Elasticsearch began as a search engine. It still powers site search, product catalogs and document search in applications that have nothing to do with logs.

For a small IT team the first three are usually one project. Get syslog from network devices, event logs from Windows and logs from applications into one cluster, then build the monitoring and security views on the same data.

The licenseThe license history, which is the part that decides things

This is the section that is missing from most explanations of the ELK stack, and it is the one that changes what you may do.

Elastic was explicit that the 2024 change took nothing away: "We are simply adding another option, and not removing anything." Its reason for choosing AGPL was the label rather than the terms: "Being able to use the term Open Source, by using AGPL, an OSI approved license, removes any questions, or fud, people might have."

So the source is available today under three licenses, and you choose which one you are relying on. AGPLv3 is OSI approved and carries the obligations AGPL always carries. The Elastic License is more permissive in most respects and has three limitations, in Elastic's own words:

• "You may not provide the products to others as a managed service"
• "You may not circumvent the license key functionality or remove/obscure features protected by license keys"
• "You may not remove or obscure any licensing, copyright, or other notices"

The first one is the one that matters to anybody running IT for somebody else.

If you run it for othersThe question a provider has to answer first

Elastic draws the line at whether your customer touches the product, and it is worth reading twice because the two cases sit close together.

Permitted: "If your customers do not access Elasticsearch and Kibana, this is permitted under ELv2." Standing up a stack inside a client's own environment, for that client's own staff, is a contractor doing infrastructure work.

Not necessarily permitted: "If your customers do have access to substantial portions of the functionality of either Elasticsearch and Kibana as part of your service, this may not be permitted." Running one central stack and giving several clients a Kibana login to it is the case the restriction was written for.

Three ways out of that corner, and all three are ordinary:

Run it under AGPLv3 instead and accept the obligations that license carries. It is an option Elastic added deliberately.

Run OpenSearch. It is Apache 2.0, so the managed service restriction does not exist. It is a fork of 7.10.2 and has moved a long way since, so it is a different product now rather than an older copy of the same one.

Ask. Elastic publishes an address for exactly this question, and a written answer is worth more than an interpretation of a FAQ.

What to send itWhat earns its storage

An ELK stack that collects all the data it can reach gets switched off when the disk fills. Log management is mostly the decision about what to keep. The sources below are the ones that repay the space, roughly in the order they pay it back.

Authentication. Successful and failed logins, from the directory and from anything exposed to the internet. This is the first thing asked for after an incident and the first thing missing.

Firewall and VPN. Denied connections tell you what is being tried; accepted ones tell you what got in. Both matter and only one is usually kept.

Windows event logs, filtered. Winlogbeat will ship every channel if you let it. Security, System and the application channels that belong to what you run are enough, and the rest is volume.

Patch and update outcomes. Not the schedule, the result. This is the evidence an audit asks for and the record that shows a fleet drifted.

Backup job results. A failed backup that nobody read is the classic finding after a restore fails, and it is three lines a day to collect.

The pattern in that list is that every entry answers a question somebody will actually ask. Debug output from an application nobody supports answers none, and it is usually the largest index on the cluster.

PitfallsWhere people go wrong

Assuming ELK means the current product. The vendor calls it the Elastic Stack. Searching the documentation for ELK finds the phrase acknowledged and little else.

Deploying Logstash because it is the L. Beats ship to Elasticsearch directly. Add Logstash when something needs parsing or routing, not because the acronym has three letters.

Reading a tutorial without checking its version. Anything written before 2021 describes an Apache 2.0 product. The instructions may still work and the licensing sentence in them is out of date.

Treating OpenSearch as old Elasticsearch. It forked at 7.10.2 and has been developed separately since. Feature parity is not the assumption to start from in either direction.

Storing everything forever. Elasticsearch will accept log data until the disk is full. An index lifecycle policy is not an optimization, it is the thing that stops the cluster ending.

Blaming Kibana for an empty dashboard. Kibana stores nothing. An empty dashboard is a collection, ingest or retention problem, one layer down.

ComparisonThe Elastic Stack and OpenSearch, side by side

The row that decides it is the fourth. If you are running the stack for somebody else and want to stop reading license FAQs, the Apache 2.0 fork ends the question. If you want the vendor's own roadmap and support, take the Elastic Stack and pick the license deliberately rather than by default.

FAQFrequently asked questions

Keep readingRelated concepts

Read next · Operations What Is a Cron Job? The other half of an operations toolkit, and the thing that most often writes the logs. Open this next11 min

• Operations · 13 min Patch Management, and Why the Hard Part Is Not the Patching The discipline whose evidence lives in a log store, and which the audit asks about.
• Operations · 9 min MTTR, and Why Two Teams Quoting the Same Number Disagree What the data in that store is usually being collected to measure in the first place.