How to get a unique field counter using Kibana + Elastic Search

Question

Can I request a separate / unique field counter using Kibana? I use elastic search as my backend for Kibana.

If so, what is the query syntax? Here is a link to the Kibana interface. I would like to make my request: http://demo.kibana.org/#/dashboard

I parse nginx access logs using logstash and save the data in search mode. Then I use Kibana to run queries and visualize my data in charts. In particular, I want to find out the number of unique IP addresses for a certain period of time using Kibana.

Answer 1

For Kibana 4, go to this answer

This is easy to do with the terms panel:

If you want to select the number of individual IP addresses that are in your logs, you must specify in the clientip field, you must put a sufficiently large number in length (otherwise it will connect to another IP address in the same group) and specify in the stylesheet. After adding the panel, you will get a table with an IP address and the number of IP addresses:

Answer 2

Now Kibana 4 allows you to use units. Besides creating a panel similar to the one that was explained in this answer for Kibana 3, now we can see the number of unique IP addresses in different periods, that is (IMO) what the FI wanted first.

To create such a panel, you must go to Visualize -> Select your index -> Select a vertical histogram, and then in the visualization panel:

• On the Y axis, we want to get a unique number of IP addresses (select the field where you saved the IP address), and on the X axis we want the date histogram to use our time field.

• After clicking the "Apply" button, we should have a graph that shows a unique number of IP addresses distributed over time. We can change the time interval on the x-axis to see unique IP hourly / daily ...

Just keep in mind that unique counts are approximate . For more information, check this answer as well.

Answer 3

Remember that with a unique counter, you use the "power" metric, which does not always guarantee an accurate unique count .:-)

power indicator is an approximate algorithm. It is based on the HyperLogLog ++ Algorithm (HLL). HLL works by hashing your input and using bits from the hash to make probabilistic power estimates.

Depending on the amount of data, I can get a difference of 700+ records that are not in the 300,000 dataset through Unique Count in Elastic, which are actually really unique.

More details here: https://www.elastic.co/guide/en/elasticsearch/guide/current/cardinality.html

Answer 4

Create a query “topN” on “clientip” and then a histogram counting on “clientip” and set the query “topN” as the source. Then you will see the number of different ips in time.

Answer 5

Unique values ​​of field values ​​are achieved using faces. See the ES documentation for the full story, but the bottom line is that you will create a query and then ask ES to prepare faces for the results to count the values ​​found in the fields. It is up to you to customize the fields used and even describe how you want the values ​​returned. The simplest of the faceted types is simply to group by dates, which will look like the IP address above. You can get quite complicated with them, even requiring a request within your aspect!

 { "query": { "match_all": {} }, "facets": { "terms": { "field": "ip_address" } } }