3.12.2007

This is super-boring.

OK, I like statistics. I've always been interested in statistical applications and there's a good reason why. Here's a few of the interesting aspects of the problem of statistical analysis software.

Most people don't understand the complexity of statistical analysis in the first place and those people will never be capable of describing their analysis needs to a tool that starts with raw data manipulation procedures and builds an analysis from there. Business Intelligence is often built-up in a complex heirarchy that many people in an organization do not understand. In many cases, it is not important for them to understand it, yet they still need to be able to produce new analyses and presentations based on those analyses, without the help of a programmer.

Many people do not view their statistical information in the same way. They visualise it differently, and some people do not visualise their statistics in any meaningful way which could apply to a software program. Some people visualise their data in ways which can not be presented accurately by any software program no matter what you do. This means the challenge of presenting statistical data will continue to be interesting in the future, as we invent new presentation technology and require software to support it.

Some people require different modes of presentation to understand the same statistical information. Some people work well with static visual information (graphs, charts, spreadsheets), but others prefer richer presentations such as animations, and interactive charts. Some people may even require audio or tactile information, particularly in the case of people with limited visual ability.

Often these people are attempting to understand the same information at the same moment in time, and they are limited to a single mode of presentation, meaning everyone has a varying degree of understanding, even though they are all looking at the same thing. This means some people may not be able to contribute in some situations even though they may be capable of groundbreaking insight given the proper presentation. It would be nice if we could have a way for multiple people to simultaneously view the same data in their personally preferred manner. This is virtually impossible with current technology unless you have prior knowledge of the type of data which will be presented. It is possible with current software if you have that prior knowledge.

The way that programmers see data and others see data is vastly different and a method must be found to bridge that gap, because in the end, the software must be capable of doing the work the user wants, and the software will be made by a programmer using a programming language. The computer has to bridge the gap between the way it works and the way a human works. This is the basic mission of all software, but with statistics, there are unique aspects to the problem. This leads many to believe that meta-languages like Mathematica, will always be required to do anything but the most basic statistical operations. There is reluctance to get too far away from the silicon, so to speak.

The underlying problem with all of it is that statistics is hard. It's complex to begin with, and so many programmers assume anyone capable of doing the analysis is going to understand a complex programming language as well, so they write such a language and consider the problem solved. Then, the people who the language was supposed to support have to deal with a big learning curve, or hire a programmer to get the problem done, and we're right back to the initial problem - you had to get a freeking programmer to do the thing! It's not because you're not intelligent enough to design the program, it's because in order to use the 'new' language, you have to think like a programmer and visualise things like a programmer, and you're not, you're a statistician!

No comments:

 
hit counter script