Elephant of Improvement

There are certain obvious truths about the wonderful fields of 'data mining', 'machine learning' and 'prediction' as far as their practical applications are concerned. The underlying goals are: transparency and improvement. Exposure of keys that help guide strategically important decisions are all part of the improvement aspect. A prediction and visualization tool is of course of no real use in contexts where common sense is the simplest and best answer. Also, more often than not, proper planning and open face-to-face communication solve a large number of productivity-related hurdles (we ate what we cooked and were convinced of this). But, let's assume that the situations with which we're dealing are non-trivial. Is it really that easy to improve and visualize? Maybe. But not in the way we think. It's great to have the ability to benefit from tools that expose the keys to success and failure; tools that look at the information and say "this is what leads to success/failure", and "if you do x, y and z, then you'll have a success (or failure)". This can be applied to day to day life (some folks have written apps to optimize one's mood, which is pretty cool), and in the workplace.However, this is not as hard as it seems. But neither is it easy in the way we've classically been told. The classic method -- and we love to call it that, because it really is impractical and antiquated in our view -- is to set up a complex set of databases, data cleaners, data mining scripts, presentation engines, analysis layers, and reporting tools. Toss in the the DBAs and statistical or business analysis domain specialists required to get this all going and you're off to the races. Once you've managed to convince the entire C-level to invest, and you've got the first dashboard out, you're an IT and improvement champion standing on the bridge of a starship and well on your way up the ladder, reaping the benefits of 'continuous improvement' and 'intelligence'. It's not that this ambitious final goal is absurd – it is completely feasible – but it's a difficult one to practically achieve within time and budget constraints and impossible without teamwork. The caveat (in the small print) is that everyone should be on board in order to get there. And it's true: teamwork is essential. There are some fantastic large-scale and open source products that will satisfy requirements at every stage. Realizing the advertised benefits from these systems is costly. They're made by and for engineers and scientists. Open source requires an open pocket. The cost of new staff, time spent trying to gain cooperation, time not spent focussing on the primary line of work, and training to name a few, are significant. Time to benefit and ease of use are critical in any tool: applying the “mom” test (asking “would my dear mom be able to understand or care about what I do?”) is a great sanity check. Interface is everything and a bad one can cost a corporation dearly. As for the complex mining and analysis tools: the target audience is not the beneficiary. And this is where views digress: we believe it should be. Just as with cloud computing, the choice of analysis tools and their interfaces, can convert the classical IT manager into a hero. We'll be audicious enough to say it: a lot of existing frameworks, technologies, reasons for consulting and ways of working are meant to ensure job security. Our approach focusses on the interface, to allow decision makers to bypass the technical hurdles one step at a time. Riding the elephant and shaking things up in our little neck of the woods has been fun so far. A huge thanks to our beta group!