On 10/24/07, Eric Peterson <eric.peterson@...> wrote:
>
> Hey Moe,
>
> Instead of assigning an arbitrary number of points to every page on
> your site, you may want to have a look at the visitor engagement
> calculation I've been talking about for awhile now. You can read the
> most current post here:
>








Definitely thought-provoking, Eric... I'm deep into this issue, although
focused specifically on community sites.

Overall, your approach doesn't work for me on two main counts -- it is too
complicated (and thus unlikely to become any sort of standard) and doesn't
generate a metric that allows different sites to be compared. The latter is
arguable, since standardized weightings could yield comparable numbers, but
I think that's excess complication also.

Is there any ground truth behind this? In case that isn't clear, do you
have any sort of primary market data for engagement that correlates with the
output of your engagement metric?

As I've dug into the issues and our data (about five dozen communities
ranging from very large to very small), I keep coming back to two main
indicators of engagement -- return rates and proactive behavior. If
visitors don't visit regularly and do something other than passive page
viewing, I have a tough time including them in any measurement of community
engagement.

Some point-by-point thoughts...

Click-depth index -- this is a place where ground truth really matters, I
think. I'm not comfortable with the assumption that more clicks per session
means greater engagement. Do we know enough about browsing behavior to know
that this is true? And of course there's the old problem of bad design
resulting in more clicks... but when I consider that issue, I tend to think
that if people show willingness to click through a bad design, maybe that
means they really are engaged! Perhaps we should all include some known bad
design... ;-)

I have pretty much the same questions about duration. Is there good,
objective evidence that session duration correlates to engagement? There
are visitors with long-duration visits who don't visit regularly and don't
do anything proactive... I can't see including them in any measurement of
engagement.

Recency makes perfect sense to me -- the fact that engaged visitors return
often is practically a tautology. I would be very skeptical of calling
anybody engaged if they aren't returning regularly.

Your Brand Index is a great piece of data, but I don't believe it works in a
metric intended to compare sites. Language is too subtle and ambiguous to
infer engagement from search terms. I spent years in the search engine and
related markets, which gave me a great appreciation for the fact that what
sometimes seems obvious about language isn't. When people search on
brand-related terms, it indicates *reach* to me, not engagement. I'm
unwilling to assume anything more than brand awareness. People search on
things they dislike, but that doesn't mean they're engaged with the subject
they're searching. And my data shows that visitors who show many other
indications of engagement actually search *less* often.

Counting brand-related searches makes sense if we're measuring *brand*
engagement.

Counting direct (non-referred) visits makes sense if we're measuring *site*
engagement.

Counting both in the same metric doesn't make sense to me. I don't think we
should even be talking here about ways to measure brand engagement...
because I believe that's well beyond the scope of site analytics. It
requires massive monitoring systems along the lines of BuzzMetrics. (I'm
the primary inventor of one of their systems.)

One more problem with the Brand Index -- people will argue all day long
about what terms are appropriate to include... and there's a strong
incentive for site owners to err on the side of too many terms if their
success is being measured by this metric. For example, you included "web
site measurement hacks" in your list... but that could be a generic term.
Is "Web Analytics Wednesday" really your brand? Or is it the WAA's? I
don't want to argue which it is, just point out the kind of ambiguity that
is inevitable.

Your Feedback Index is a specific instance of what I think of as the general
principle of tracking proactive behaviors -- what you seem to be getting at
in your Interaction Index. In communities, visitors have many such
opportunities -- posting, editing, tagging, voting and so forth. I decided
very early in this work to just give people one point for each such
proactive action, despite the temptation to weight them (which would violate
the need to keep things simple). These are the behaviors that make a
community work; sites that aren't based on user-generated content can exist
without them.

Your session focus really got me thinking. Does it make more sense to count
the number of sessions in which visitors signal engagement or the number of
actual such signals? I think it's close to a toss-up, but so far, our
ground truth suggests the latter -- the number of proactive actions
correlates better to our subjective estimates of engagement... but among our
future tasks is to establish better ground truth. So far, I'm just using
our community manager's collective subjective scoring... but it correlates
quite well to all but our smallest communities.

The subscriber index doesn't work for me because we want to be able to
compare communities regardless of whether or not visitors are able to
subscribe, join, become members or what-have-you. Some of our clients --
e.g., a large professional sports organization -- allow full participation
without any need to sign up. Also, as I'll explain below, I've found a
strong negative correlation between highly active visitors and RSS
subscribers.

Finally, I guess I'll toss out one of the ideas that I'm working with --
segmenting visitors by proactivity.

In several ways, communities (and most web sites, I suspect) have a bimodal
distribution of users. There's typically a relatively large "Core" group
that visits often, looks at lots of pages and does a lot of proactive
stuff. There's a middle ground, which I'm calling "Lingerers," of people
who fall into the 10th to 80th percentiles of such activities. Third and
last, there's a large contingent in the 0th percentile, people who might
have one or two activities in a given time period, which I call the
"Drive-bys." In our communities, the Drive-bys are the largest group, but
the Core usually is a bigger group than the Lingerers. What this says to me
is that people tend to engage a lot or hardly at all -- there isn't much
middle ground. I've been focusing on the Core's relationship to the whole
community for my engagement measurements. That's what seems to correlate
best to what little ground truth we have.

Overall, I've found that the Drive-bys and Lingers exhibit fairly similar
behavior, but the Core is different. The Core visitors post more, search
less and use RSS far less (so much for "subscribing" to RSS as a positive
indicator of engagement!).

This post is getting long... so I'll wrap it up (but ready to discuss
further, of course) by repeating myself. I think any sort of engagement
metric has to be backed up by demonstrating correlation to some kind of
ground truth. Otherwise, it's a mental exercise that runs the risk of
having little relevance to the marketplace.

Nick


--
Nick Arnett
narnett@...
Messages: 408-904-7198


[Non-text portions of this message have been removed]