QNUE

Welcome Pack
January 2015

Agenda

The basics
Main concepts
QNUE
Development environment
Configuring a new event
Resources

The basics
In theory

Source

Task

Task

Task

Target

The basics
The practice

The basics

Happenings > Events

Cardinality
1>1
1>N
N>M
Happening Model
Event Model
Rules
Rendering
Delivering

The basics
Storm to the rescue

Is a highly distributed realtime computation system.

Has a very simple programming model.
Provides general primitives to do near time computation
Can be used with any JVM programming language.
Its scalable and fault-tolerant.

The basics
Storm to the rescue

Created by Nathan Marz @ BackType >> Twitter

Analyze twits, links, users on Twitter
Opensourced at Sep 2011 Eclipse Public License 1.0
16k java and 7k Clojure Loc
Current stable release 0.9.X major core improvement
Active user group
Most watched java repo at GitHub (>4k watcher)
Used by over 30 big companies: Twitter, Groupon, .

Main concepts

Main concepts
Storm

Topology

Main concepts

Main concepts

Main concepts
Bolts could implement:

Functions
Filters
Aggregation
Enrichment
Database connections

Design principles:

SRP
Implements the Command Pattern
Everything runs in parallel except
your code
Parallelization tuning is out of your
code

Main concepts
Topology: a network of spouts and bolts

Has a Name
Has a Main class
Is delivered as a JAR
Includes all dependencies

Main concepts
Topology: a network of spouts and bolts

Deploy a new Topology

$ storm jar mypology.jar poc.
storm.MyMainClass
demoTopology

Kill
$ storm kill demoTopology

Main concepts
System architecture

Nimbus
ZK
Supervisor
Workers
UI

QNUE
Goals

Support event processing through a distributed infraestructure

Features

Configurable event processing definitions (topology & bolts)

Reusable bolts & spouts (SRP)
Multiple cardinality support
Data enrichment
Priority & flow management
...

QNUE
Logical architecture
Ra
bbit
MQ

Rabbit
Spout

Identifi
cation

Parser

Event
Builder

Valida
tion

Ren
derer

Dispat
cher

Deli
very

Ra
bbit
MQ

QNUE
Logical architecture
Ra
bbit
MQ

Rabbit
Spout

Identifi
cation

Parser

Event
Builder

Valida
tion

Ren
derer

And what about 1 to N ?

And N to M ?
Message processing guarantee ?
What about historify in bulks?
Enrichment?
Error management?

Dispat
cher

Deli
very

Ra
bbit
MQ

Logical architecture

QNUE

Development environment

JDK 1.7
SVN
Maven
M2_REPO
RabbitMQ (local)
Your preferred text editor
(sublime/Notepad++)

Configuring a new event

Resources
Development Guide
Support
francisco.aranda@adesis.com
javier.antoniucci@adesis.com