mjt.me.

uk

https://www.mjt.me.uk/posts/kibana-101/

Kibana Log Searching 101

So someone has just given you access to Kibana and you're having trouble answering the kind of questions you
could have answered easily with a sql- or grep-based system.
Hopefully this cheat sheet will help get you started!

How do I do a basic search?

The 'query' box works a bit Google: unstructured text search, with some special commands, and if you get the
command syntax wrong it just does an unstructured text search.
Unlike Google, by default it searches for entries containing any of your search terms, and it considers hyphen a
delimiter.
Example queries:
addressregistry
solution-addressregistry <-- Finds entries containing 'solution' or 'addressregistry'

What if I want to search for a string literal?

Double quotes.
'asdf-addressregistry' <-- Wrong, single quotes get ignored - this searches for 'asdf' or 'addressregistry'
"geo-address-registry" <-- That's how you quote things properly

So it doesn't tell you if you've got the syntax wrong?

I know, right? Pretty much the opposite of SQL. Takes some getting used to, and makes it harder to figure out
through experimentation - hence this documentation!

How do I search a single field?

Field name, then a colon.
Example: tags:"address-registry"

What about searching for missing (or present) fields?

exists:exception
missing:message

What about searching for several things?

exists:exception AND tags:routing
exists:exception AND routing <-- Second becomes free text search
exists:exception AND ( tags:routing OR appID:geo ) <-- This does what it looks like

AND and OR are case-sensitive. Example:

tags:routing AND kjrnglkjerghljkf <-- No results, no entry tagged routing also contains free text
kjrnglkjerghljkf
tags:routing and kjrnglkjerghljkf <-- Wrong, this finds anything tagged routing, or with free text 'and'
or with free text kjrnglkjerghljkf

So is it it all case-sensitive, then?

Text matching is case-insensitive, but field names and AND and OR are case insensitive.
tags:routing <-- Normal query
tags:ROUTING <-- Matches the same thing (text match case insensitive)
Tags:routing <-- No results (field name case sensitive)

Why doesn't appID:geo-addressregistry-v1 do what I expect?

The hyphens are delimiters. It's searching for anything with appID containing geo, or free text containing
addressregistry or v1.

Well then, why doesn't appId:"geo-address-registry-v1" do what I expect?

Check the capitalisation of that tag. Are you looking for appId when you should be looking for appID?

Well then, why doesn't appID:"geo-addressregistry-v1" do what I expect?

Because you're missing a hyphen. Try appID:"geo-address-registry-v1" instead.

So what are these filter things?

They're like query strings, except the results get cached. I think. And you can toggle them and add them
automatically from that magnifying glass symbol! Pretty weird design if you ask me.

What's with the 'save' and 'apply' buttons?

The 'apply' button saves the filter and updates your results.
The 'save' button saves the filter but doesn't update the results.
I'm not sure why you'd want to do that, but it's there if you do.

How do you find the distinct values for a field?

This can be done, but it's a bit odd. First you want to add an panel
to a dashboard row:
Set the type to 'terms', the 'field' to whatever field you want the
distinct values of, and the length to some big number.

You probably want the style to be 'table' - easy to experiment with it though.

I deleted the only filter and now it's telling me "No time filter Timestamped indices
are configured without a failover. Waiting for time filter."
I think the first index on the database is on time or something. Anyway, you have to add it back in, you can do that
with this dropdown:

How do I find the log lines immediately

before and after a given line, as one
might with grep -c20 ?
Unfortunately this isn't supported.

How do I share a query with my

colleagues by e-mail?
There's a 'share button' which will generate a
permalink:
Note that if your query includes 'the last hour' or similar, in an hour the results will have all
changed.

What about a permalink to a log line?

This feature might be in a future version - in the meantime you can do this:
Add a filter on _id so your query returns one line.
Mess with the timestamp so your link works when your line exits the 'last 30 minutes' window or whatever
you have selected.
Then e-mail a permalink.
Not the most elegant method, obviously.

What if I want to copy a query between Prod and Dev or similar?

Permalinks are just references to a database entry, not a complete encoding of the dashboard. Export the
schema as JSON and import it somewhere else like so:

Why is googling for kibana query strings so hard?

Kibana is backed by ElasticSearch so sometimes google helpfully adds elasticsearch query documentation to
your search for kibana query documentation. But ElasticSearch has a bunch of features that don't work in the
kibana query box.
For example, when you look at this documentation the one-liners at the bookmarked point in the page will work but if you scroll up to the JSON stuff, that won't work in the kibana query box.
You may have better results searching for 'lucene query syntax' which is the syntax used by the kibana 'query'

box

How do I get a graph with multiple lines?

So it turns out the green dot is a button!

If you set the type to 'topN' and the field to whatever you want to chart, it'll chart the frequency
of the most frequent N values:

You can also add multiple queries by using the '+' to the right of the query box

Unfortunately this graph seems to count hyphenated values multiple times (e.g. 'address-registry' counts for both
'address' and 'registry') them's the breaks.