ESPER
[quote]
Esper is open-source software available under the Apache LPGL license.

esper.codehaus.org/using/faq/faq.html
Can I run it with multiple threads? What, if anything, is
multithread-safe?

Esper supports multiple independent Esper engines per Java VM.
Each engine instance itself is NOT multithread-safe. In other words,
for a given engine instance we do not currently guarantee the engine
instance behaves the same in a multithreaded environment as in a
single-threaded environment.

Applications using Esper can synchronize around the sendEvent()
method. No ThreadLocal data representing engine state remains after
the sendEvent() method completes. Thus multiple threads that
synchronize posting events to the same engine instance is supported,
with the following limitations: (1) statement creation through the
EPAdministrator interface, and statement start/stop must be avoided or
synchronized with the posting of events (2) Iterating (pulling) of
statement results must be synchronized as well.

Applications can also segregate work to different engine instances
allocating a thread to each engine instance.

Applications that perform pure filtering of events via event
pattern statement are also thread-safe. In this scenario, the event
pattern statement can only consist of a simple filter and no other
operators. Also, new patterns cannot be created and patterns cannot be
removed while multiple threads are used to send events in for filtering.

We plan to improve this in upcoming versions.


Could you explain the concept of windows for a database programmer?

One could perhaps think of a table with a timestamp column
containing the time when the row was inserted. We could create a view
that sorts by timestamp descending and selects all rows between the
current timestamp and say up to 1 minute prior to now. Every time we
fire a query against this view, the view returns the recent rows added
in the last 1 minute. The rows returned are the contents of a 1 minute
time window. Every time the query is fired we get a new window. Older
rows would seem to leave the window while new rows would seem to enter
the window.


What is the difference between Esper and an in-memory database?

The Esper engine works a bit like a database turned
upside-down. Instead of storing the data and running queries against
stored data, the Esper engine allows applications to store queries and
run the data through. Response from the Esper engine is real-time when
conditions occur that match queries. The execution model is thus
continuous rather then only when a query is submitted.

How does Esper work? How does Esper allow you to search and match
patterns on temporal events?

Esper is an event stream processing (ESP) and event correlation
engine (CEP) written in Java. Basically instead of working as a
database where you put stuff in to later poll it using SQL queries,
Esper works as real time engine that triggers actions when event
conditions occur among event streams. A tailored Event Query Language
(EQL) allows registering queries in the engine, using javabeans to
represent events. A listener class - which is basically a POJO - will
then be called by the engine when the EQL condition is matched as
events come in. The EQL allows expressing complex matching conditions
that include temporal windows, and join different event streams, as
well as filter and sort them.

A simple example could be to compute the average stock price of
the BEA tick on a sliding window of 30 seconds. Given a StockTick
event bean with a price and symbol property and the EQL "select
avg(price) from StockTick.win:time(30) where symbol='BEA'", a POJO
would get notified as tick come in - and in real world millions of
ticks can come in - so there's no way to store them all to later query
them using a classical database architecture. Statements can be much
more complex, and also be combined together with "followed by" conditions.

The internals of Esper are made up of fairly complex algorithms
primarily relying on state machines and delta networks in which only
changes to data are communicated across object boundaries when required.


What's the advantage of using Esper over other products?

Esper can easily be embedded in any Java runtime including J2EE
application servers or standalone Java applications. Events in Esper
can be represented as plain Java objects (POJO) eliminating the need
to format, feed and receive events into a separate possibly remote
engine process. Esper is an entirely free open-source component
available under the LPGL license. The open-source nature of Esper
helps in tailoring the event processing language and other community
driven features.


Who's using Esper?

With the Esper 1.0 release and given it's shorter history most
users are at the evaluation stage. A major financial institution is
looking at Esper for real-time monitoring of proprietary options
trading. A network monitoring application had success with Esper, and
Esper is also playing a role in RFID research. Our user community is
growing rapidly.


What business areas/problems is Esper best suited for?

Esper is best suited for real-time event driven applications.
Typical application areas are business process management and
automation, finance, network and application monitoring and sensor
network applications. Esper take much of the complexity out of
developing applications that detect patterns among events, filter
events, aggregate time or length windows of events, join event
streams, trigger based on absence of events etc.

A primary difference with system relying on classical SQL
databases is that we do not query a repository for events matching
some conditions, but instead trigger customized actions as the flow of
events come in matching event conditions - hence drastically reducing
the latency.


What might be some mis-uses for it?

Esper is not designed for storing and retrieval of fairly static
data - that is better left to conventional databases. In-memory
databases may be better suited to CEP applications than traditional
relational databases as they generally have good query performance.
Yet they are not optimized to provide immediate, real-time query
results required for CEP and event stream analysis.


What is the intended audience and what is their interface?

Esper doesn't have a GUI and access is via API only, currently.
The intended audience is developers of CEP or ESP applications.


How does Esper hook into an existing event-based system?

Today, event based system are often confused with message based
system such as JMS and alike backends or Enterprise Service Bus (ESB).
Messaging system are event based but usually do not correlate events
or process event streams at all. They simply process unitary events -
and it becomes quickly complex if you need to correlate messages.

Esper can hook into any Java based system as a message consumer -
whose main purpose would be to listen efficiently to everything whilst
sending back other events when a matching condition has occured. On
the other hand, hooking Esper in a classic messaging system allows
having it correlate events even if those come from non-Java based
systems. This is just an example. Esper is completely container
agnostic, can run as a standalone component, and the 1.0 release
simply assumes events to be POJO. We are currently working on
providing native XML format support as well.


How would you position Esper next to existing solutions (or hacks) in
the industry?

Esper is the only open source Event Stream Processing out there -
and this is not to be confused with classical rules engines. Compared
to commercial ESP engines implementations, Esper may lack high
availability options in the 1.0 release, but its open source nature is
a driver to get the Event query language right, and get the Java
community up to speed on those concepts - before tackling enterprise
grade requirements in upcoming versions.

Generally Rete-based production rule engines can indeed be used to
address part of what CEP coins. Especially triggering by correlating
events, possibly including a temporal relationship between events.
That said the ESP side - Event Stream Processing - is a different
beast, where what matters is the "S" for Stream. In this side of the
Esper engine we provide several language facilities to build
expressions using time not for temporal relationship (happened before
etc) but for sliding window. As an example, this makes it very
valuable for computing things like volume weighted average price
(VWAP) of ticks - which would possibly be awful using something
Rete-based like a classical rule engine.


What is the concept or philosophy behind the design?

Esper was developed using test-driven development and excellent
automated test coverage. Esper's design evolved by refactoring with
courage towards higher design quality. Favorite patterns are
dependency injection/inversion of control, GOF patterns, Immutable,
Specification.


What is the history? Where did it start and come from?

Thomas Bernhardt is the project founder and project lead. He works
as a software architect for a major financial institution. On a
project in 2004 Thomas had been asked to evaluate different rules
engines for use in an application for monitoring a trading system. The
rules engines proved cumbersome and slow. There was no budget for a
CEP product, thus a custom application to solve the monitoring problem
was developed. The project lead's interest in CEP and ESP technologies
lead to further independent research into the knowledge space followed
by prototypes build for no particular organization or purpose. The
prototypes evolved and the Esper project started.

[endquote]

The reason that ESPER and programs like it are of interest in this
group's discussion is that streams (of data) are generated by sensors
in a robotic system and by the various biological systems in a person.
This group is about technology about computer (technical and
scientific) metaphorical systems, ontologies, metadata, first and
second order metaprogramming topics, and even Humberto Maturana and
Francisco Varela autopoeisis. Tools, such as XML, XSLT2, Protege,
SWRL, RuleML and others will be discussed too.

Advanced systems have multiple simultaneous streams of data activity.
Situation Awareness systems are one way of organizing a collection of
systems into a functioning whole, not just a reactive one, but one
also using deliberation. ESPER type systems can provide a tool to
adequately handle large of amounts of data flow in multiple channels
to some useful end.

The filters in ESPER are one implementation of context, and ESPER can
provide
Event Pattern Matching such as Logical and temporal event correlation.
If you were paying attention to the info above about ESPER then you
saw that it deals with temporal asynchrony as well as the more
obviously processed same channel sequential data streams such as
business processing systems typically use.

A future posting will discuss temporal asynchrony in Situation
Awareness systems and how ESPER may be used there to advantage. Also
pattern guided Aspect Oriented Programming (AOP) will be discussed,
AOP is usually done by a human programmer, what happens when a program
does the AOProgramming instead? SVG Reflection ! and AOP in the Clock
demo will be discussed as a project.