Mastering Elasticsearch - Second Edition

Table of Contents

Mastering Elasticsearch Second Edition

Credits

About the Author

Acknowledgments

About the Author

Acknowledgments

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why subscribe?

Free access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Introduction to Elasticsearch

Introducing Apache Lucene

Getting familiar with Lucene

Overall architecture

Getting deeper into Lucene index

Norms

Term vectors

Posting formats

Doc values

Analyzing your data

Indexing and querying

Lucene query language

Understanding the basics

Querying fields

Term modifiers

Handling special characters

Introducing Elasticsearch

Basic concepts

Index

Document

Type

Mapping

Node

Cluster

Shard

Replica

Key concepts behind Elasticsearch architecture

Workings of Elasticsearch

The startup process

Failure detection

Communicating with Elasticsearch

Indexing data

Querying data

The story

Summary

2. Power User Query DSL

Default Apache Lucene scoring explained

When a document is matched

TF/IDF scoring formula

Lucene conceptual scoring formula

Lucene practical scoring formula

Elasticsearch point of view

An example

Query rewrite explained

Prefix query as an example

Getting back to Apache Lucene

Query rewrite properties

Query templates

Introducing query templates

Templates as strings

The Mustache template engine

Conditional expressions

Loops

Default values

Storing templates in files

Handling filters and why it matters

Filters and query relevance

How filters work

Bool or and/or/not filters

Performance considerations

Post filtering and filtered query

Choosing the right filtering method

Choosing the right query for the job

Query categorization

Basic queries

Compound queries

Not analyzed queries

Full text search queries

Pattern queries

Similarity supporting queries

Score altering queries

Position aware queries

Structure aware queries

The use cases

Example data

Basic queries use cases

Searching for values in range

Simplified query for multiple terms

Compound queries use cases

Boosting some of the matched documents

Ignoring lower scoring partial queries

Not analyzed queries use cases

Limiting results to given tags

Efficient query time stopwords handling

Full text search queries use cases

Using Lucene query syntax in queries

Handling user queries without errors

Pattern queries use cases

Autocomplete using prefixes

Pattern matching

Similarity supporting queries use cases

Finding terms similar to a given one

Finding documents with similar field values

Score altering queries use cases

Favoring newer books

Decreasing importance of books with certain value

Pattern queries use cases

Matching phrases

Spans, spans everywhere

Structure aware queries use cases

Returning parent documents having a certain nested document

Affecting parent document score with the score of nested documents

Summary

3. Not Only Full Text Search

Query rescoring

What is query rescoring?

An example query

Structure of the rescore query

Rescore parameters

Choosing the scoring mode

To sum up

Controlling multimatching

Multimatch types

Best fields matching

Cross fields matching

Most fields matching

Phrase matching

Phrase with prefixes matching

Significant terms aggregation

An example

Choosing significant terms

Multiple values analysis

Significant terms aggregation and full text search fields

Additional configuration options

Controlling the number of returned buckets

Background set filtering

Minimum document count

Execution hint

More options

There are limits

Memory consumption

Shouldn't be used as top-level aggregation

Counts are approximated

Floating point fields are not allowed

Documents grouping

Top hits aggregation

An example

Additional parameters

Relations between documents

The object type

The nested documents

Parent–child relationship

Parent–child relationship in the cluster

A few words about alternatives

Scripting changes between Elasticsearch versions

Scripting changes

Security issues

Groovy – the new default scripting language

Removal of MVEL language

Short Groovy introduction

Using Groovy as your scripting language

Variable definition in scripts

Conditionals

Loops

An example

There is more

Scripting in full text context

Field-related information

Shard level information

Term level information

More advanced term information

Lucene expressions explained

The basics

An example

There is more

Summary

4. Improving the User Search Experience

Correcting user spelling mistakes

Testing data

Getting into technical details

Suggesters

Using the _suggest REST endpoint

Understanding the REST endpoint suggester response

Including suggestion requests in query

The term suggester

Configuration

Common term suggester options

Additional term suggester options

The phrase suggester

Usage example

Configuration

Basic configuration

Configuring smoothing models

Configuring candidate generators

Configuring direct generators

The completion suggester

The logic behind the completion suggester

Using the completion suggester

Indexing data

Querying data

Custom weights

Additional parameters

Improving the query relevance

Data

The quest for relevance improvement

The standard query

The multi match query

Phrases comes into play

Let's throw the garbage away

Now, we boost

Performing a misspelling-proof search

Drill downs with faceting

Summary

5. The Index Distribution Architecture

Choosing the right amount of shards and replicas

Sharding and overallocation

A positive example of overallocation

Multiple shards versus multiple indices

Replicas

Routing explained

Shards and data

Let's test routing

Indexing with routing

Routing in practice

Querying

Aliases

Multiple routing values

Altering the default shard allocation behavior

Allocation awareness

Forcing allocation awareness

Filtering

What include, exclude, and require mean

Runtime allocation updating

Index level updates

Cluster level updates

Defining total shards allowed per node

Defining total shards allowed per physical server

Inclusion

Requirement

Exclusion

Disk-based allocation

Query execution preference

Introducing the preference parameter

Summary

6. Low-level Index Control

Altering Apache Lucene scoring

Available similarity models

Setting a per-field similarity

Similarity model configuration

Choosing the default similarity model

Configuring the chosen similarity model

Configuring the TF/IDF similarity

Configuring the Okapi BM25 similarity

Configuring the DFR similarity

Configuring the IB similarity

Configuring the LM Dirichlet similarity

Configuring the LM Jelinek Mercer similarity

Choosing the right directory implementation – the store module

The store type

The simple filesystem store

The new I/O filesystem store

The MMap filesystem store

The hybrid filesystem store

The memory store

Additional properties

The default store type

The default store type for Elasticsearch 1.3.0 and higher

The default store type for Elasticsearch versions older than 1.3.0

NRT, flush, refresh, and transaction log

Updating the index and committing changes

Changing the default refresh time

The transaction log

The transaction log configuration

Near real-time GET

Segment merging under control

Choosing the right merge policy

The tiered merge policy

The log byte size merge policy

The log doc merge policy

Merge policies' configuration

The tiered merge policy

The log byte size merge policy

The log doc merge policy

Scheduling

The concurrent merge scheduler

The serial merge scheduler

Setting the desired merge scheduler

When it is too much for I/O – throttling explained

Controlling I/O throttling

Configuration

The throttling type

Maximum throughput per second

Node throttling defaults

Performance considerations

The configuration example

Understanding Elasticsearch caching

The filter cache

Filter cache types

Node-level filter cache configuration

Index-level filter cache configuration

The field data cache

Field data or doc values

Node-level field data cache configuration

Index-level field data cache configuration

The field data cache filtering

Adding field data filtering information

Filtering by term frequency

Filtering by regex

Filtering by regex and term frequency

The filtering example

Field data formats

String-based fields

Numeric fields

Geographical-based fields

Field data loading

The shard query cache

Setting up the shard query cache

Using circuit breakers

The field data circuit breaker

The request circuit breaker

The total circuit breaker

Clearing the caches

Index, indices, and all caches clearing

Clearing specific caches

Summary

7. Elasticsearch Administration

Discovery and recovery modules

Discovery configuration

Zen discovery

Multicast Zen discovery configuration

The unicast Zen discovery configuration

Master node

Configuring master and data nodes

Configuring data-only nodes

Configuring master-only nodes

Configuring the query processing-only nodes

The master election configuration

Zen discovery fault detection and configuration

The Amazon EC2 discovery

The EC2 plugin installation

The EC2 plugin's generic configuration

Optional EC2 discovery configuration options

The EC2 nodes scanning configuration

Other discovery implementations

The gateway and recovery configuration

The gateway recovery process

Configuration properties

Expectations on nodes

The local gateway

Low-level recovery configuration

Cluster-level recovery configuration

Index-level recovery settings

The indices recovery API

The human-friendly status API – using the Cat API

The basics

Using the Cat API

Common arguments

The examples

Getting information about the master node

Getting information about the nodes

Backing up

Saving backups in the cloud

The S3 repository

The HDFS repository

The Azure repository

Federated search

The test clusters

Creating the tribe node

Using the unicast discovery for tribes

Reading data with the tribe node

Master-level read operations

Writing data with the tribe node

Master-level write operations

Handling indices conflicts

Blocking write operations

Summary

8. Improving Performance

Using doc values to optimize your queries

The problem with field data cache

The example of doc values usage

Knowing about garbage collector

Java memory

The life cycle of Java objects and garbage collections

Dealing with garbage collection problems

Turning on logging of garbage collection work

Using JStat

Creating memory dumps

More information on the garbage collector work

Adjusting the garbage collector work in Elasticsearch

Using a standard start up script

Service wrapper

Avoid swapping on Unix-like systems

Benchmarking queries

Preparing your cluster configuration for benchmarking

Running benchmarks

Controlling currently run benchmarks

Very hot threads

Usage clarification for the Hot Threads API

The Hot Threads API response

Scaling Elasticsearch

Vertical scaling

Horizontal scaling

Automatically creating replicas

Redundancy and high availability

Cost and performance flexibility

Continuous upgrades

Multiple Elasticsearch instances on a single physical machine

Preventing the shard and its replicas from being on the same node

Designated nodes' roles for larger clusters

Query aggregator nodes

Data nodes

Master eligible nodes

Using Elasticsearch for high load scenarios

General Elasticsearch-tuning advices

Choosing the right store

The index refresh rate

Thread pools tuning

Adjusting the merge process

Data distribution

Advices for high query rate scenarios

Filter caches and shard query caches

Think about the queries

Using routing

Parallelize your queries

Field data cache and breaking the circuit

Keeping size and shard_size under control

High indexing throughput scenarios and Elasticsearch

Bulk indexing

Doc values versus indexing speed

Keep your document fields under control

The index architecture and replication

Tuning write-ahead log

Think about storage

RAM buffer for indexing

Summary

9. Developing Elasticsearch Plugins

Creating the Apache Maven project structure

Understanding the basics

The structure of the Maven Java project

The idea of POM

Running the build process

Introducing the assembly Maven plugin

Creating custom REST action

The assumptions

Implementation details

Using the REST action class

The constructor

Handling requests

Writing response

The plugin class

Informing Elasticsearch about our REST action

Time for testing

Building the REST action plugin

Installing the REST action plugin

Checking whether the REST action plugin works

Creating the custom analysis plugin

Implementation details

Implementing TokenFilter

Implementing the TokenFilter factory

Implementing the class custom analyzer

Implementing the analyzer provider

Implementing the analysis binder

Implementing the analyzer indices component

Implementing the analyzer module

Implementing the analyzer plugin

Informing Elasticsearch about our custom analyzer

Testing our custom analysis plugin

Building our custom analysis plugin

Installing the custom analysis plugin

Checking whether our analysis plugin works

Summary

Index

Mastering Elasticsearch Second Edition

Mastering Elasticsearch Second Edition

Copyright © 2015 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: October 2013

Second edition: February 2015

Production reference: 1230215

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78355-379-2

www.packtpub.com

Credits

Authors

Rafał Kuć

Marek Rogoziński

Reviewers

Hüseyin Akdoğan

Julien Duponchelle

Marcelo Ochoa

Commissioning Editor

Akram Hussain

Acquisition Editor

Rebecca Youé

Content Development Editors

Madhuja Chaudhari

Anand Singh

Technical Editors

Saurabh Malhotra

Narsimha Pai

Copy Editors

Stuti Srivastava

Sameen Siddiqui

Project Coordinator

Akash Poojary

Proofreaders

Paul Hindle

Joanna McMahon

Indexer

Hemangini Bari

Graphics

Sheetal Aute

Valentina D'silva

Production Coordinator

Alwin Roy

Cover Work

Alwin Roy

About the Author

Rafał Kuć is a born team leader and software developer. Currently, he is working as a consultant and a software engineer at Sematext Group, Inc., where he concentrates on open source technologies, such as Apache Lucene, Solr, Elasticsearch, and the Hadoop stack. He has more than 13 years of experience in various software branches—from banking software to e-commerce products. He is mainly focused on Java but is open to every tool and programming language that will make the achievement of his goal easier and faster. Rafał is also one of the founders of the solr.pl website, where he tries to share his knowledge and help people with their problems related to Solr and Lucene. He is also a speaker at various conferences around the world, such as Lucene Eurocon, Berlin Buzzwords, ApacheCon, Lucene Revolution, and DevOps Days.

He began his journey with Lucene in 2002, but it wasn't love at first sight. When he came back to Lucene in late 2003, he revised his thoughts about the framework and saw the potential in search technologies. Then came Solr, and that was it. He started working with Elasticsearch in the middle of 2010. Currently, Lucene, Solr, Elasticsearch, and information retrieval are his main points of interest.

Rafał is the author of Solr 3.1 Cookbook, its update—Solr 4.0 Cookbook—and its third release—Solr Cookbook, Third Edition. He is also the author of Elasticsearch Server and its second edition, along with the first edition of Mastering Elasticsearch, all published by Packt Publishing.

Acknowledgments

With Marek, we were thinking about writing an update to Mastering Elasticsearch, Packt Publishing. It was not a book for everyone, but the first version didn't put enough emphasis on that—we were treating Mastering Elasticsearch as an update to Elasticsearch Server. The same goes with Mastering Elasticsearch Second Edition. The book you are holding in your hands was written as an extension to Elasticsearch Server Second Edition, Packt Publishing, and should be treated as a continuation to that book. Because of such an approach, we could concentrate on topics such as choosing the right queries, scaling Elasticsearch, extensive scoring descriptions with examples, internals of filtering, new aggregations, comparison to documents' relations handling, and so on. Hopefully, after reading this book, you'll be able to easily get all the details about Elasticsearch and the underlying Apache Lucene architecture; this will let you get the desired knowledge easier and faster.

I would like to thank my family for the support and patience during all those days and evenings when I was sitting in front of a screen instead of being with them.

I would also like to thank all the people I'm working with at Sematext, especially Otis, who took his time and convinced me that Sematext is the right company for me.

Finally, I would like to thank all the people involved in creating, developing, and maintaining Elasticsearch and Lucene projects for their work and passion. Without them, this book wouldn't have been written and open source search wouldn't have been the same as it is today.

Once again, thank you.

About the Author

Marek Rogoziński is a software architect and consultant with over 10 years of experience. He specializes in solutions based on open source search engines, such as Solr and Elasticsearch, and software stack for Big Data analytics, including Hadoop, Hbase, and Twitter Storm.

He is also a cofounder of the solr.pl website, which publishes information and tutorials about Solr and Lucene libraries. He is the coauthor of Mastering ElasticSearch, ElasticSearch Server, and Elasticsearch Server Second Edition, both published by Packt Publishing.

Currently, he holds the position of chief technology officer and lead architect at ZenCard, a company processing and analyzing large amounts of payment transactions in real time, allowing automatic and anonymous identification of retail customers on all retailer channels (m-commerce / e-commerce / brick and mortar) and giving retailers a customer retention and loyalty tool.

Acknowledgments

This is our fourth book about Elasticsearch and, again, I am fascinated by how quickly Elasticsearch is evolving. We always have to find the balance between describing features marked as experimental or work in progress, and we have to take the risk that the final code might behave differently or even ignore some of the interesting features. The second edition of this book has quite a large number of rewrites and covers some new features; however, this comes at the cost of the removal of some information that was less useful for readers. With this book, we've tried to introduce some additional topics connected to Elasticsearch. However, the whole ecosystem and the ELK stack (Elasticsearch, Logstash, and Kibana) or Hadoop integration deserves a dedicated book.

Now, it is time to say thank you.

Thanks to all the people who created Elasticsearch, Lucene, and all the libraries and modules published around these projects or used by these projects.

I would also like to thank the team that worked on this book. First of all, thanks to the ones who worked on the extermination of all my errors, typos, and ambiguities. Many thanks to all the people who sent us remarks or wrote constructive reviews. I was surprised and encouraged by the fact that someone found our work useful. Thank you.

Last but not least, thanks to all the friends who stood by me and understood my constant lack of time.

About the Reviewers

Hüseyin Akdoğan's software adventure began with the GwBasic programming language. He started learning the Visual Basic language after QuickBasic, and developed many applications with it until 2000 when he stepped into the world of Web with PHP. After that, his path crossed with Java! In addition to counseling and training activities since 2005, he developed enterprise applications with Java EE technologies. His areas of expertise are JavaServer Faces, Spring frameworks, and Big Data technologies such as NoSQL and Elasticsearch. In addition, he is trying to specialize in other Big Data technologies.

Julien Duponchelle is a French engineer. He is a graduate of Epitech. During his professional career, he contributed to several open source projects and focused on tools that make the work of IT teams easier.

After he led the educational field at ETNA, a French IT school, Julien accompanied several start-ups as a lead backend engineer and participated in many significant and successful fundraising events (Plizy and Youboox).

I want to thank Maëlig, my girlfriend, for her benevolence and great patience during so many evenings when I was working on this book or on open source projects in general.

Marcelo Ochoa works at the system laboratory of Facultad de Ciencias Exactas of the Universidad Nacional del Centro de la Provincia de Buenos Aires and is the CTO at Scotas.com, a company that specializes in near real-time search solutions using Apache Solr and Oracle. He divides his time between university jobs and external projects related to Oracle and big data technologies. He has worked on several Oracle-related projects, such as the translation of Oracle manuals and multimedia CBTs. His background is in database, network, web, and Java technologies. In the XML world, he is known as the developer of the DB Generator for the Apache Cocoon project. He has worked on the open source projects DBPrism and DBPrism CMS, the Lucene-Oracle integration using the Oracle JVM Directory implementation, and the Restlet.org project, where he worked on the Oracle XDB Restlet Adapter, which is an alternative to writing native REST web services inside a database resident JVM.

Since 2006, he has been part of an Oracle ACE program. Oracle ACEs are known for their strong credentials as Oracle community enthusiasts and advocates, with candidates nominated by ACEs in the Oracle technology and applications communities.

He has coauthored Oracle Database Programming using Java and Web Services by Digital Press and Professional XML Databases by Wrox Press, and has been the technical reviewer for several PacktPub books, such as "Apache Solr 4 Cookbook, ElasticSearch Server", and others.