By Jefferson Goethals
I posted the other day on the subject of energy monitoring, and one of the points I made was that the key to a good monitoring system was the ability to transform raw data into useful information.
The distribution of smart meters that gather data on energy usage is providing us with mountains of data, which will grow rapidly and exponentially. Just collecting all the data will not help much if we cannot make sense of it. Making sense of it requires two things: analysis and presentation.
Data analysis is hard to do well, but the computing power available now opens up many lines of analysis that were previously unavailable. LOESS regression and other curve-smoothing techniques developed once the computing power was available to make them practical. Now genetic algorithms, used for curve-fitting and optimization, can be executed on a personal computer (though if the analysis is even a little bit complex, it may take 48 hours).
Data presentation may be even harder to do well than analysis. It is very easy to present a technically accurate picture of a data set that is utterly misleading.* It is also very easy to present a picture of a data set that is totally incomprehensible to the intended audience. Meaningful, useful data analysis presentations require creativity, empathy, and an understanding of the implications of any conclusions suggested by the analysis. Essentially, it is sales, which is fundamentally interpersonal.
*Perhaps the best writing on this is from Nate Silver, who comments on political polling. He makes it easy to understand how various methods can make a polling result misleading. His blog is here.
Effective analysis requires mathematical sophistication and interpersonal skill. In the case of data from smart meters, it also requires a foundation in electrical engineering and an understanding of the utility industry. Data is often best understood through visualizations, so artistic skill is also helpful. If the analysis is aimed at encouraging people to change behavior to use less energy, a working knowledge of psychology would help.
The point I am making here is that good analysis is best done in teams. If you are building tools for analysis, like monitoring software, a diverse team is critical. The green energy industry needs more teams like this if the smart meters we are deploying are going to provide the value we hope they will.
Interesting post, Jefferson. I would take it one step farther and say that data presentation isn't an end in itself, but just a means to the true end, which is behavior change on the part of consumers. Somehow, data must be translated into specific behaviors that reduce consumption in a meaningful way, including feedback that lets people know when they're doing it right.
ReplyDeleteI agree that this requires a team effort. As a user experience consultant, I'd be the person testing out possible approaches with real people to find which ones actually work before they get rolled out to millions of customers. If anyone has a project they'd like to collaborate on, I'm at csnyder@snyderconsulting.net