Too Much

too much.jpg

I think that the BI world needs a serious shake since I am seeing a well established trend that now it is going entirely against the interest of the organizations being served.



  • Self service tools like Tableau or Spotfire are becoming more and more popular albeit not being based on a strong and controllable semantic model.
  • I see business users trying to learn how to model in DAX or the M language
  • Tools like Alteryx and Lavastorm put the ETL in the hands of the analysts.
  • Data scientists join the business and ask for flat files and consider normal spending the 90% of their time cleansing data.
  • Tools that have never been scrutinized by experts are being used by the business and introduced as a fait accompli.

In general, we are seeing that the analytical functions are sitting more and more into business areas and they are replacing the services offered by traditional specialists. Data manipulation and information management is being popularized and dispersed, as if handling raw data were something that could be mastered by the average knowledge worker. This is happening because traditional solutions, managed by IT, are not supposed to give the users the freedom to pursue their ideas and they do not adapt fast enough to the changing business conditions.

I wonder if I am the only one who is seeing that this trend has made too much headway in corporate culture.

The deep flaw in this "modern" approach that should be blatantly visible to anyone is the fact that now we are building very low efficiency information management systems.

Try to do an exercise. Use activity based costing to evaluate the cost of managing the information in a system like that and compare it with a traditional solution. Try comparing the cost of, say, 200 analysts and data scientist in a "modern" solution with the 20 analysts/DS and 12/15 professionals in a classical configuration (and the professionals are going to be there anyway) and then calculate the difference.

So far, I have done a few of those exercises and the answer was never well received by the people in charge of the money. To avoid some minor complications connected with IT operations and project management, organizations are throwing away enormous quantities of resources, burning Full Time Equivalents in the attempt to manage the way the information is consumed.  A "modern" arrangement may cost three times as much as a traditional one, but as far as the money is coming from different budgets within the organization, everything looks right.

In addition to the costs,  we also have all the traditional issues like, duplication of efforts, lack of a single version of truth, processes out of control, decisions made upon not certified / not certifiable data, wrong data disclosed with legal issues arising etc.

The counter argument to this proposition, is, of course, that the issues connected with a centralized IT approach are still there and the higher cost is just the cost of sorting those problems.

The point is that it is in not a law of physics that IT must be slow to react and it must tie the users' hands. In my experience it is just an organizational problem, and a quite simple one indeed.
In a usual BI unit you have developers, Analysts, BAs, PMs, Architects etc.
Instead of fragmenting the functions like this, try having individuals who are embedded into users groups (I call them BI Officials) and have the responsibility and the full authority, from source systems extraction, through databases to the last of reports, to keep those users happy and you will see the difference.
IT is slow because there are plenty of people who have to do their small piece of job to satisfy a request and an entire management system is in place to coordinate these efforts.
Put everything in the hands of a single skilled professional (eventually with an assistant or two) who has the full authority to take decisions on the spot without the need to consult anyone and, by magic, all these issues disappear.
It is a bit difficult to find such  a professional but it is just because the market goes in another direction. Obviously you will need a manager to coordinate the officers' work and someone to keep servers and databases healthy, but that's all. 

However, even those in the BI industry are starting to ask themselves questions. I had the opportunity to sit in the same room with one of the founders of one of the most successful of these "modern" vendors and he, privately, admitted that they are rushing to add enterprise features because these issues start to emerge, bids are lost because of the lack of a solid enterprise architecture and this is limiting the growth.

So, after all, I am not going against the common wisdom without a piece of reason ...