nextanalytics blog

Seth Grimes published an article about the potential growth of four BI companies. 

“That core software components are free makes open source attractive to users as well as systems integrators and independent software vendors that sell products and services built on open source BI components.” 

Take note: People at least start out expecting products which advertise as Open Source to be free.

The companies he wrote about leverage the words "open source" in their marketing albeit with the word "Commercial" associated with "Open Source". 

I am worried that, to someone who's not "in the know," this distinction is too subtle.  I am thinking of the general press (ie. not BI press) and customers who are new to BI, etc etc. 

They latch onto the words "Open Source," we all know that. They think it's all one and the same, they're talking about True Open Source.

I would worry that might feel as if a bait-and-switch occurred once they learned what's really going on.  To varying degrees, this will eventually detract from the brand and philosophy of "Open Source" (per the OSI definition).  In essence, this is what Roberto Gallopini joked about when he read one of our early days press releases.    BTW, that particular press release was a mistake, some of the wording about Open Source got out of our control and published before we could correct it -- Roberto was quite right to tease us about it :).

Seth said:

 “Commercial open source BI vendors, notably Pentaho and JasperSoft, offer these components in free community editions with open source licenses, and also packaged with non-open source extensions in paid, supported, indemnified editions.

"It's hard to get going without help from the vendor, and anytime I need to upgrade, I get a little bit nervous. (from Seth's article, a quote by Venkat Gaddipati, CTO at online marketplace OnForce)"

“Beyond Compliance harnesses the Palo OLAP Server on the back end and the non-open source Palo Worksheet Server for report distribution.”

To me, these quotes imply that a customer needs closed software to find it useful.   Are these companies really Open Source vendors? I see what I see.   Form your own opinion.

In a separate article by Stephen Swoyer, Vincent Pineau, GM Talend Americas, said:

"We definitely do not feel that open source should equate to free. Yes, we are lower [in price] than our competitors, but we also do not want people to think that cheap in price or cheap in features. We believe that the right features are what people should focus on," Pineau says

Ok, then, we have had a project up at Codeplex for a long time now.  It is true open source.  I'd really prefer you pay for the commercial one (at this site).  Can we now call ourselves Commercial Open Source? Is that now an acceptable term? 

Premise-Promise of Open Source

Seth said “For most open source BI adopters, however, the solution search starts with cost.” 

Folks, from my experience, people seeking Open Source want a FREE, totally free, experience.  They want to pay nothing. Period.

This isn't the case with Commercial Open Source vendors. Customers end up paying $30,000.  And they might spend a month or two of expensive IT time trying to get it working before they resort to these Commercial Open Source vendors. 

Remember, this is something they originally thought would get free.  $30,000+ and several months of work is not free.  One might even argue that "Commercial Open Source" is not in-expensive.

Sometimes it's better to pay and get documentation, support, and quality control. IMO, you will end up spending less.

Our own philosophy is that the stuff you need to be open-source, is. 

For example, our User Interface and programmatic interfaces are all Open Source. Supported, customizable Open Source.

Therefore, one shouldn't divide the BI world into two categories.  There are more than the four "commercial open source vendors" and the monolithic mainstream BI vendors who were mentioned. 

In fact, there are plenty of other vendors who can fulfill BI needs very satisfactorily for less cost than Pentaho and Jaspersoft.

Those are good companies. I know some of the founders and they are great people. I can't quite comment on their product quality since Open Source by its nature can easily have random bugs injected and no one is accountable.  

My Point is, if you're shopping for Commercial Open Source, then also open your minds to other vendors too.  The TCO will be about the same and you might have a better experience. Not everyone charges as much as the big guys, although, I wish :)  ...

Craig Schiff wrote an article at http://www.intelligententerprise.com/showArticle.jhtml?articleID=51201364 some time ago.  Following are some salient points he made:

"Without steady involvement by the business side, IT's dashboard projects are doomed to failure."

"IT thinks in terms of improving access to data, whereas finance focuses on improving its processes (particularly the painful ones)."

"What's most important is the content: that is, what's being measured and displayed on the dashboard."

"confront one of the toughest challenges: How do you get to that short list of agreed-upon key measures?"

"Once you've settled the corporate strategy, chosen the metrics development team, and put a facilitator in place, you can get down to business. The team needs to work down from the strategy to determine the key business drivers."

"the organization may not have the underlying data necessary to perform the calculations to support the KPI. What do you do then? Throw the KPI out? Wrong answer. What most companies do is manually enter the data for the KPI."
"KPIs should offer a good mix of financial as well as operational measures."

I think I've represented some of his key points accurately.  It's enough to tell me (and you) that Craig and I think alike.   His article was written in 2006 and following are my comments on what I think has evolved since then. 

To begin, I've think it's necessary to cut down the reliance on IT in providing the metrics and generating the KPIs because:

1. Managers like to generate their own information whether it be KPIs or metrics. The key evidence of this is the popularity of Excel. They need this to be able to react to fast changing business needs.  Routing new business questions and performance metrics and indicators through IT is too slow.
   
   
2. He suggested that, if a company isn't tracking the data, entering it by hand is risky and an auditable process won't be likely.  For me, this would be a least preferable approach, but at the time it was written, it was a decent & practical one.
   
   Instead, I believe there's a very good chance the data is there in the company, it's just that conventional BI tools are limited to what they can do. 
   
   If you're using a cube, then the data has to be in the one cube or you have to be a grandmaster in MDX to get what you want. If you're using a reporting tool, then the data has to be available from a single SQL query. Relying on this kind of technology and its limitations is just plain in-adequate now that better choices are avaialble. Let's for the moment ignore the fact that they charge way too much money and the IT overhead is a trememendous financial burden.
   
3. Users, not IT, need a way to read in data from multiple heterogenous data sources into the same report.

My work at nextanalytics has been addressing this evolving requirement.   I think we've made great gains and will continue to do so in the future.

Most people don't realize just how much work there is in populating the data for a dashboard or scorecard.  Let’s make a list of some of the things that contribute to the problem.

The Query.  Getting the data

To get useful data, SQL is too difficult and expertise is scarce and expensive.  Moreover, SQL can’t do all the things you need. Often, the solution requires extra tables which are usually undesirable.  MDX & Cubes are an alternative, but they are expensive to design, create and maintain especially if the needs are likely to change.

Once the query has been submitted, too much data returns.  The ODBC layer or DataSet Object uses too much memory and causes failures at the presentation layer.

Putting The Data Into The Chart

Now that you have the data, there's too many labels, and large ones affect how the legends and chart labels appear. It is way too easy to get a chart that looks horrible.  Visualizations fail on large number of rows and columns. Overwriting risers, lines, symbols are all quite common.

Most chart package support some  local data manipulation and calculation operations, but they are not integrated with the query, causing a design & implementation burden for the presentation layer developer.

Crosstabs & Datagrid Objects

Just like with charts, too many and large labels take up too much screen real-estate. There’s no room for numbers and you can’t visualize anything about your data. Too much scrolling.

As with chart packages, local math and sort operations are not integrated with the query, and it’s difficult to save the user’s actions to the same state the next time they open the view.

UI Controls  (Combos, listboxes, etc )

Plainly said, these fail given a normal amount of production data.  UI developers resort to hard-coding but slowly changing data (in the business) makes the dashboard a maintenance nightmare.  Metadata mitigates this, but at a great overhead expense .

Three Part Solution

Reduce the volume, provide subsets as a result of analytical workflows, not just arbitrary hierarchy like OLAP provides.  From the user interface, let the users choose an analyzed subset to look at with their scorecard or dashboard.  Alternatively, use analytics to aggregate lower level detail into upper level views but avoid the overhead of a cube or temporary database tables.

Use analytic workflows to improve the content. Pre-calculate patterns, trends, outliers, comparisons, and whatever else suits your business. 

Provide your users with valuable "already analyzed" numbers.  If you do this, you don’t need to chart as much data, and the data being shown is more useful than raw “database data”.  

Avoid the trap of encouraging them to export to Excel, spreadsheets are an entity that requires auditability per Sarbanes-Oxley. Don't get yourself into a legal mess.

Improve your charts and crosstabs by using programmatic techniques to perform text and numeric substitutions.  This makes your visualizations more readable.

Three easy steps. That's all it takes to make scorecards & dashboards more useful.   

Nextanalytics makes these steps easy.  It offers additional processing on query results and transparently provides the analyzed data to presentation layers.  A simple, elegant solution that is fast and easy to implement.