Esper event-stream processing
[quote]
Esper has been issued in its first production release of its
open-source Esper event-stream processing and event-correlation engine.

Applications can define time-based, interval-based, length-based and
sorted windows and perform inner-joins and outer joins of an unlimited
number of windows and event streams. Esper also performs complex event
pattern matching allowing logical and temporal event correlation.

To evaluate the java beans, Esper compares against developer-written
Event Query Language (EQL) statements which are matched against events
as they come in and used to trigger POJO listener/handler classes.
EQL allows the expressing of complex matching conditions that include
temporal windows, and joining different event streams, filtering,
sorting, grouping, etc:

Rules engines are cumbersome and slow for this use, what was really
needed was real time event correlation engine.


Esper take much of the complexity out of developing applications that
detect patterns among events, filter events, aggregate time or length
windows of events, join event streams, trigger based on absence of
events etc. Esper will ease development of applications that filter
event streams and detect patterns, aggregate data over time- and
length-windows, join streams and trigger actions based on the presence
or absence of particular events.

Esper can run standalone, the 1.0 release assumes events to be POJOs.
The Esper team is currently working on providing native XML format
support.

It performs grouping, aggregation, sorting, filtering, merging and
joining of event streams using a tailored SQL-like query language.
Applications can define time-based, interval-based, length-based and
sorted windows and perform inner-joins and outer joins of an unlimited
number of windows and event streams. Esper also performs complex event
pattern matching allowing logical and temporal event correlation.

A primary difference with systems relying on classical SQL databases
is that events are not stored and later on polled in a massive hence
underperformant repository. Instead as the flow of events come
matching registered event conditions, Esper triggers customized
registered actions (POJO) - hence drastically reducing the latency.

ESPER deals with the task of processing multiple streams of event
data with the goal of identifying the meaningful events within those
streams, and deriving meaningful information from them.

Some typical examples of applications are:

* Business process management and automation (process monitoring,
BAM, reporting exceptions, operational intelligence)
* Sensor network applications (RFID reading, scheduling and
control of frabrication lines, air traffic)

Features

* Event Stream Processing
o Time-based, interval-based, length-based and sorted windows
o Grouping, aggregation, sorting, filtering and merging of
event streams
o Tailored SQL-like query language using insert into,
select, from, where, group-by, having and order-by clauses
o Inner-joins and outer joins (left, right, full) of an
unlimited number of windows
o Output rate limiting and stabilizing
* Event Pattern Matching
o Logical and temporal event correlation
o Crontab-like timer 'at' operator
o Lifecyle of pattern can be controlled by timer and via
operators
o Pattern-matched events provided to listeners
* Supports both listener (push) and consumer (pull) model
* Supports event-type inheritance and polymorphism as provided by
the Java language
* Events can be plain Java objects, XML document object model
(DOM) and java.util.Map including nested objects
* Event properties can be simple, indexed, mapped or nested -
allows querying of deep Java object graphs and XML structures
* Supports externally-supplied time as well as Java system time
* Supports multiple independent Esper engines per JavaVM. Each
engine instance itself is NOT multithread-safe, requiring workload
structuring per engine if necessary (also see FAQ on multithread-safety).
* Pluggable architecture for event pattern and event stream
analysis user-defined functions

Key considerations for these types of applications are the complexity
of the logic required, throughput and latency.

* Complex computations - applications that detect patterns among
events (event correlation), filter events, aggregate time or length
windows of events, join event streams, trigger based on absence of
events etc.
* High throughput - applications that process large volumes of
messages (between 1,000 to 100k messages per second)
* Low latency - applications that react in real-time to
conditions that occur (from a few milliseconds to a few seconds)

Known Limitations

* Esper requires a Java Virtual Machine version 5.0 runtime, or
above.
* Esper will not work with JavaVM versions 1.4.2 or below.
* A Esper engine instance itself is NOT multithread-safe, i.e. 2
or more threads sending events to the same engine instance is not
supported. However multiple engine instances with a maximum of one
thread per engine instance is supported.

This example joins 2 event streams. The first event stream consists
of fraud warning events for which we keep the last 30 minutes (1800
seconds). The second stream is withdrawal events for which we consider
the last 30 seconds. The streams are joined on account number.

select fraud.accountNumber as accntNum, fraud.warning as warn,
withdraw.amount as amount,
MAX(fraud.timestamp, withdraw.timestamp) as timestamp,
'withdrawlFraud' as desc
from FraudWarningEvent.win:time(1800) as fraud,
WithdrawalEvent.win:time(30) as withdraw
where fraud.accountNumber = withdraw.accountNumber

Event Pattern Matching

Event patterns match when an event or multiple events occur that
match the pattern's definition. Patterns can also be temporal
(time-based). Pattern matching is implemented via state machines.

Pattern expressions can consist of filter expressions combined with
pattern operators. Expressions can contain further nested pattern
expressions by including the nested expression(s) in round brackets.

There are 5 types of operators:

1. Operators that control pattern finder creation and termination:
every
2. Logical operators: and, or, not
3. Temporal operators that operate on event order: -> (followed-by)
4. Guards are where-conditions that filter out events and cause
termination of the pattern finder, such as timer:within
5. Observers observe time events as well as other events, such as
timer:interval, timer:at

A sample pattern that alerts on each IBM stock tick with a price
greater then 80 and within the next 60 seconds:

every StockTickEvent(symbol="IBM", price>80) where timer:within(60)

An event pattern where a property of a following event must match a
property from the first event:

every a=EventX -> every b=EventY(objectID=a.objectID)

Event Representation in Java

Events are represented as JavaBean instances. Esper can interrogate
any plain-old Java object representing an event as long as the object
exposes the required getter methods following the JavaBean conventions.

Esper can also query object graphs consisting of arrays, mapped or
nested event properties, or any combination. The following are
examples of valid event properties for EQL and pattern statements:

OrderEvent.item[1].description
RFIDEvent.attribute('shelfcode')

© 2005-2006 Floyd Marinescu


References:
Seth Grimes
Intelligent Enterprise Fri Sep 1, 12:00 AM ET
intelligententerprise.com/showArticle.jhtml?articleID=192200071

infoq.com/news/Esper--ESP-CEP
Floyd Marinescu on Jul 28, 2006 05:23 PM
Esper: High Volume Event Stream Processing and Correlation in Java

www-128.ibm.com/developerworks/java/library/j-mxbeans/
Using Java platform management beans

[endquote]
"POJO"=plain old java objects