Now that I’ve been home from San Diego for a few days and had some time to reflect, Xchange 2011 is one of those moments where I realized that I’m just one of many people in the digital measurement community and that I’ve got a TON to learn!  There are so many impressive people out there innovating new measurement techniques and integrating new data sources that when I return to work my program will be 3 years behind where I thought it was…and I’ve only been out of the office for 4 days.  How’s that for a change in perspective?

Coronado Island Marriott Resort and Spa

In terms of a facility to hold a conference, I can’t think of a better location than the Coronado Island Marriott.  At first, I thought the remote location would’ve been a hindrance, since attendees might be tempted to scatter around to see the city and the networking would be limited.  But a big hand goes to the Semphonic crew that set up all of the events, from the cruise of the harbor, to the flamenco night and the pool party to end the conference.  Each of these events allowed me to meet some of the biggest names in the industry and really have some interesting discussions.

Note:  If you’re going to hang out in the hotel lobby/bar and make outrageous claims such as being able to sabre a champagne bottle with a kitchen knife, be sure to have documentation.  Take that, Bob! 

A mix of sharing and therapy

For the most part, the huddles that I attended were centered around making the most of the data you have (“Getting the data to tell you its secrets”, “Distilling complex information into Dashboards”, “Getting valuable insights from less than perfect data”) or when/how to acquire more data (“BIntegration:  What to keep and what to leave out”).  While I felt that most of these discussions were great in terms of outlining problems and then how to work around said problems, I felt at times that the discussions de-volved into therapy sessions.

One of my biggest disappointments in this “therapy” realm was the Dashboard Huddle.  With 40-50 people in the room, I didn’t really hear any solutions to getting beyond imperfect dashboard implementation; to be sure, I heard plenty of ways to diagnose useless dashboards, like turning the distribution off and seeing if anyone notices.  I also heard several people say that they educated their stakeholders better to use the reporting tools directly (eliminating need for automated dashboards) and/or building interactive data architectures for on-demand questions.  But surely, someone in the industry has been able to crack this problem?  We all recognize that tailoring reporting to the audience and stakeholder education are two ways to improve any data distribution, but that’s like saying “If everyone was as smart as I am, then there wouldn’t be a problem.”

Data governance and analytical frameworks still lacking

The other topic that my Huddles tended to cover was around data governance, making sure your data gathering processes are documented and audited and that ultimately, decisions get made due to using this data.  This is one area I was glad that I could contribute to the Huddle topics, suggesting that analysts create a measurement plan document, outlining the following for each analysis they are asked to do:

• Business question (“What do you think the problem is?)
• Proposed site change (“How do you think you’ll solve it?”)
• Data sample and KPI’s (“How will you prove the change worked?”)
• What exceptions or other ideas need to be kept in mind if data looks odd?

By laying out the answers to these 4 simple questions, an analyst can get their stakeholders to starting thinking about their changes in a more rigorous fashion.  Hunting and pecking for insights from a data sample is a sure way to never get anything done, and this framework eliminates that before analysis starts.  I also picked up two questions that I’m going to incorporate into my template:

• What are the possible outcomes from this test?
• For each possible outcome, what decision should be made?

Adding these two questions to a measurement document takes the decision away from being a referendum on someone’s failure (when the test doesn’t go as planned), and puts the focus back on the learning.  The decision becomes one more step in the process before the analysis can be considered “complete”, rather than letting an analysis linger in blamestorming land.

In the end, it’s all about doing something

Like website testing, the value of any conference or training is whether you are inspired to do something with your newly found knowledge.  Xchange 2011 passes this test without question.  At times during the conference, I found myself tuning out the discussion, because I was already thinking about how to incorporate something I heard an hour earlier into my digital measurement program.  There were also times I worked through the break, re-classifying my data or creating a report from a question posed during a huddle.  It was just that sort of conference…being surrounded by your peers, talking passionately about the future of web analytics and measurement was enough to inspire me to get back to work.

And for that I say, “Well done, Semphonic!”
This post is inspired by Gary Angel’s post here about his feelings on Xchange 2011…