Pedro Alves on Business Intelligence

PCM13: Pentaho Community Event 2013 - Let's start!

2013-06-14T15:20:00.001+01:00

What

Let's get this going!

#PCM13 is starting to get organized. For the ones that don't know, the Pentaho Community Meetings have been happening since 2008 a bit all over Europe. Last year was in Amsterdam.

It's one of my main objectives to end up with the silly "Enterprise / Community" differentiation. This is an event for everyone with a connection to Pentaho - customers, user, hackers, developers, students. I'll try to bring more of the Pentaho people, but the main spirit will remain the same! Sun, fun, huge amounts of beer and lots of presentations. I'll personally slap hard anyone that even tries to change that!

Where

This year we're going to break 2 rules: We usually don't repeat the event location and it's usually entirely sponsored by the community itself.

When I accepted the role of SVP for Community I put as condition that we'd sponsor this year's event, and that we'd repeat Portugal and make even a better event than the amazing event in Cascais. Location yet to be determined but probably in the amazing Sintra village.

When

So let's start by setting the date in stone. There are 2 options on the table: 27/28 of September or 5/6 of October. I personally prefer the second, but can adapt - the community will chose in the end!

So go to this thread on the Pentaho Foruns and take your pick!

Pentaho London User Group Meeting - Thursday 20th June, 6pm

2013-06-07T11:16:00.001+01:00

Hello everyone!

Exactly 2 months after and one ocean across the Pentaho Community event in Brazil, that had an all time record of about 200 atendees, we're having a User Group meeting in London. Shared points between both events? A common topic, amazing people willing to share their experiences and learn from others and the always fundamental beer and pizza! Unfortunately, the amazing Brazilian weather won't be the same. Most likely it will be cold, dark and wet! Welcome to London! :)

The event will be held at the Skills Matter Exchange in the Clerkenwell area of London on Thursday, June 20, 2013 at 6.00 pm. We're targeting the Pentaho Community, which in my definition includes customers, users, developers, basically anyone that's willing to spend some time helping to make the product better. It's one of my main goals as Senior VP of Community to create conditions for that to happen, and I'm interested in hearing ideas and feedback that anyone is willing to share.

Here's the current agenda:

Matt Casters, creator and architect of PDI / Kettle will lead a demo and discussion on how Pentaho supports Hadoop and big data analytics.
Dave Romano will lead a talk on how big data start-up Causata has been using Pentaho, specifically covering its use of a custom repository, step plug-ins and embedding Kettle
Pedro Alves will present CPK, the Community Plugin Kickstarter, a tool that allow non-developers to create Pentaho Plugins
Simon Raybould will describe the dashboard centric implementation at Found, heavily centered around Ctools and Mondrian

Please note that most of the presentations are technically oriented, mainly of interest to consultants and developers. We invite you to propose discussions, technical presentations, user stories and hosted Q&A sessions to educate and inspire other users.

There will be plenty of time before and after the meetup for informal networking. For more information and to register, please visit this link. On behalf of PLUG organiser Dan Keeley and Pentaho, we hope to see you on June 20^th

Pedro Alves, Senior VP of Community, Pentaho

Ctools Icon Set

2013-05-13T10:13:00.001+01:00

Our UX team, tired of putting up to the speed where we develop new ctools, decided that it would be a good idea to build an official icon set that you can get from our website.

Obviously, we allow it to be available to anyone, so feel free to (ab)use it!

Experimenting and sharing CCC2 charts

2013-05-02T14:41:00.000+01:00

CCC, now on version 2, is a very powerful charting engine, with a lot of customization abilities. Like Thomas Morgner likes to quote, with great power comes great responsibility. While a lot is possible, also a lot is not easy to do, and requires some knowledge.

The CCC website

The team made a huge effort to document CCC2. The website and the CCC jsdocs, complemented by Protovis Jsdocs, provides a great amount of resources for knowing how to best solve a problem

JsFiddle

There's also a great, well known, resource for sharing experiments with javascript, JsFiddle. We can also setup a CCC2 playground that people can use and fork to play with.

Here's a fiddle with a sample of CCC2 barchart with annotations.

A new challenge - Webdetails joins Pentaho

2013-04-22T01:29:00.004+01:00

The Announcement

Now here's a blog post that a while back I wouldn't even think about writing.

I'm very happy to announce that Webdetails will join Pentaho! Here's an excerpt from the press release:

ORLANDO, FLA - April 22, 2013 – Delivering the future of analytics, Pentaho announced today that it has completed the acquisition of its Portugal-based consulting partner Webdetails. Pentaho will benefit from Webdetails’ visual interface development expertise and international consulting services provided by its 20-strong team. Webdetails’ founder Pedro Alves is a high-profile member of Pentaho’s open source community and will take on the new role of Senior VP, Community for Pentaho.

All the team at webdetails feel incredibly proud for the vote of confidence, and very excited to keep working as hard we can but now with the new goal of improving the pentaho platform and the community ecosystem. Oh! - wait, we've been doing that for the past 5 years! :p

The daily work of the now Portuguese pentaho crew won't change much - we'll still operate and provide services under the the name Webdetails (now with an additional tag a pentaho company), have our UX team rock and rolling and keep creating the ctools until we run out of letters on the alphabet.

We'll just have the huge opportunity of doing this things at a much larger scale. The plan to conquer the world advances as planned!

The Community

I had the pleasure to make the announcement first hand in the Pentaho Day 2013, the amazing Brazilian community event that had over 180 people that chose to spend a Saturday on a room with a faulty A/C talking about a subject that unites them.

I'll take the new role of SVP of Community for Pentaho, where I'll try to be "the chief advocate and interface to Pentaho’s active open source developer community". As you may have guessed the bit on italics was copy-pasted directly from the aforementioned press release. I don't even know what that means!

As I had the opportunity to present to the audience in Brazil, the guys at pentaho that thought it would be a good idea to put me on that role have no idea what's going to happen!

This is not me

Building a community is not about going to events on tropical countries, drinking huge amounts of liquor and saying how cool pentaho is (wait, scratch the part about liquor and going to tropical countries, I do like that)

This is me!

A community doesn't exist because a company wants it to. A community is a reflex of the strategy of a company. People will only gather around a common cause - any cause - if they feel connected to it.

I'll be doing a lot of evangelization work, but not externally. It's an inside job, where I'll try to share what the community is doing, feeling, saying and fight to make that information reach the product and the strategy.

I do believe there's a lot to be done, and it's not an easy task to try to achieve a good balance between the Enterprise and Community objectives, but I do believe there's lots of room for improvement!

Please feel free to come to me for any suggestions, complaints, rants, whatever. Pentaho needs to make a good job in allowing people from the community to succeed. It's the only way this works. We're not a bunch of good guys. At least I'm not! Only when people feel they're getting something they'll be willing to give something back.

Community is all about creating Opportunity!

-pedro

New Ctools Releases: 13.03.25

2013-03-27T12:18:00.000Z

Easter release!

CDF

CCC V2 integration!

Full changelog:

* CCC V2 integration

* Added capability to detect session timeout and request new credentials (instead of silently failing)

* Feature: Support for extra options in select component

* Feature: Add clickAction & expand selectors to table component

* Fixed Heatgrid Sample

* Fix: Multiselect component now checks for null values regardless of the plugin used.

* Fix [REDMINE-1822] - Tooltip not showing in select components

* Fix [REDMINE-309] - td.expandingClass now has a dummy click for it to work as intended.

* Fix: Column format in table component now takes hidden columns into account. Ex: If column 0 is hidden: - Now: format for column 1 is colFormats[1] - Previously: format for column 1 was colFormats[0]

CDE

Support for CCC V2

Full changelog:

* Support for CCC V2

* Update siteMap component to accept an url from where to fetch the siteMap json using an ajax call.

* Implemented [Redmine-93] - Close the Popup Component

* Fixed [Redmine-1840] - External resource editing is failing

* Fixed [Redmine-345] - Scrollbars in PopupComponent

* Fixed [Redmine-1874] - Remplate import should now work as intended

CDA

Full changelog:

* Fixed [REDMINE-1851] - Added conditions that protect the TableComponent from empty metadata queries (eg. Select include mode with output options [1,0] and an empty query)

* Fixed [REDMINE-1881] - IntegerArrays not working on sql queries

* Reintroduced deprecated SQLReportDataFactory.setQuery for pentaho 3.9 support

GGG

* Support for CCC v2

* Internal architectural changes - refactored cgg to cgg-core and cgg-pentaho.

CDV

Full changelog:

* There were some edge cases where parameters were not being correctly parsed in cdv's validation dashboard

Pentaho Day 2013 in Brazil, Fortaleza

2013-03-15T11:33:00.002Z

I'm pretty excited about this. The Brazilian pentaho community, one of the most active ones I know of, is organizing the second community event in Brazil, this time in Fortaleza, April 20

It's gonna be as great as all the other ones. Registrations can be done here and discussion space will be in the pentahobrasil facebook page.

The event agenda will be shared later, as we're giving the community the opportunity to talk about their own experiences.

This is a joint organization of all pentaho consulting companies in Brazil in an amazing display of cooperation. I'll be there! :)

CPK - Community Plugin Kickstarter - Request For Comments

2013-02-27T10:52:00.000Z

CPK

One more Ctool in the works. Once again I'm calling out for comments, suggestions, whatever. Let me know what you thing and suggestions for improvements, or even if you think this makes no sense at all.

Motivation

Pentaho is very well known for being a very good Business Analytics software, but is in fact much more than that; Pentaho is a great platform to build on top of.
Using an easy analogy, I see Pentaho acting as an operating sytem where people can build Application on top of

Objective

The goal of CPK is to provide a simple and easy way to develop pentaho plugins that behave like packaged applications, simplifying it's structure.
The UI is built using CDE, with a simple methodology to create new dashboards / pages and a sitemap that provides simple navigation and a default template
There are 3 options for doing server-side code:

Kettle transformations
Javascript server side code execution
Java classes

The first two are recommended, since it's easier to register new endpoints just by dropping code in a directory and no compilation is necessary.
With this approach not only we expect to make it easier and faster to develop plugins, we also hope to lower down the specific technical requirements to build them. The end goal is that business consultants are able to build new plugins, not requiring specific java knowledge.

Structure

This is the resulting plugin structure. Ideally, no compilation is necessary, so everything except maybe the lib directory could be stored in a VCS system.
This is the proposed stub configuration
to be completed

CPK_Plugin
|-- conf
|-- dashboards
|-- endpoints
|-- lib
`-- plugin.xml

CPK administrative features

Besides providing the regular templating for creating new plugins, CPK can have an administrative UI with the following features:

List existing plugins
Detect if the plugins are up to date with the latest version of CPK
Allow the creation of new plugins
Allow to change plugin metadata
List and register new endpoints (UI and code)

Here's a list of stretch goals / nice to have

Import UI from existing dashboards in solution
Allow editing dashboards from this UI
Submit marketplace metadata to Pentaho
Generate distribution zip package

Updates

As the CPK framework or any of it's dependencies gets improved, the plugins themselves can't stay outdated. There will be a version information attached to the CPK plugin version so that it's possible to upgrade to the latest version.

Dependencies

CPK will have as little code as possible, making it as simple as possible to develop plugins. However, it will need a few dependencies:

Pentaho
CPF - Community Plugin Framework, with the common set of code for the plugins
CDE - Community Dashboard Editor
CDF - Community Dashboard Framework
CDA - Community Data Access

Link with Pentaho Marketplace

Once a plugin is developed, and the authors think it's in a state that can be shared, CPK will be able to generate a packaged plugin and metadata information so it can be integrated into Pentaho's marketplace.
Pentaho will then be able to categorize / approve the plugin so that it becomes available to other users through the marketplace

License

This project uses MPLv2

CBF - Build RPMs automatically

2013-02-22T12:23:00.000Z

Motivation

CBF - Community Build Framework - is an amazing tool for being able to maintain several projects, compile pentaho and even doing remote deploys using rsync.

However, there are cases where rsync is not the best way to maintain a server installation. At Mozilla, IT uses Puppet to maintain everything related to automated server maintenance. The way CBF is structured allows us to define rules to automate solution checkouts, but we need to automate the installation of 3 extra bits:

Pentaho webapp
Pentaho solution (the system/ and admin/ directories)
Administration / Enterprise console

Approach

The cluster uses RHEL6 distribution, so we chose to pack the distributions using rpm format. This should work for any rpm-based distribution. If someone wants to do the same for deb or something else, just get in touch or simply fork CBF on github and send a pull request.

Building RPMs through CBF

CBF now has 2 extra options (be sure to grab the latest version from github):

create-rpm-dist - Creates a tgz file with pentaho, pentaho-style and ROOT folders - to be used with a puppet script
create-rpms - Creates rpms based on the specs found on the config/rpm folder for the solution

In order for this to be usable, we need to create a directory called rpm under config, and place there the .spec files for the rpm's we're creating. We will also need a specific makefile.

This is the structure:

project-metrics
├── config
│   ├── build-ee-pedro.properties
│   ├── build-pedro.properties
│   ├── build.properties
│   └── rpm
│       ├── makefile
│       ├── webdetails-pentaho-adminconsole.spec
│       ├── webdetails-pentaho-adminconsole-stage.spec
│       ├── webdetails-pentaho-solution.spec
│       ├── webdetails-pentaho-solution-stage.spec
│       ├── webdetails-pentaho-webapp.spec
│       └── webdetails-pentaho-webapp-stage.spec
├── etl
├── patches
│   ├── project-metrics
│   ├── target-build
│   └── target-dist
├── patches-ee
│   ├── enterprise-console
│   ├── server
│   └── solution
└── solution -> ../project-metrics-solution

You'll see that I'm building here 6 RPMs, for a production and stage environment.

You can download the entire set of files from here.

Here's a sample of the file webdetails-pentaho-solution.spec:


Name:  Webdetails-Pentaho-Webapp
Version: @VERSION@
Release: 1%{?dist}
Summary: Pentaho webapp customized for Mozilla metrics
Group:  Applications/Engineering
License: MPL-1.0
Source:     webdetails-pentaho.tgz


%description

%prep
rm -rf %{buildroot}
%build
tar zxvf ../SOURCES/webdetails-pentaho.tgz 

%install
mkdir %{buildroot}
make DESTDIR=%{buildroot} install

%clean
rm -rf %{buildroot}

%files
%defattr(-,root,root,-)
%doc

/opt/pentaho/pentaho-server/server/webapps/pentaho
/opt/pentaho/pentaho-server/server/webapps/pentaho-style
/opt/pentaho/pentaho-server/server/webapps/ROOT

%changelog

And here's the makefile that is called by the specs. I chose to install the pentaho server under /opt/pentaho/, but this can be changed to any desired path:



all: install
    
install:
 mkdir $(DESTDIR)/opt
 mkdir $(DESTDIR)/opt/pentaho
 mkdir $(DESTDIR)/opt/pentaho/pentaho-server
 mkdir $(DESTDIR)/opt/pentaho/pentaho-server/server/
 mkdir $(DESTDIR)/opt/pentaho/pentaho-server/server/webapps
 cp -r pentaho $(DESTDIR)/opt/pentaho/pentaho-server/server/webapps/pentaho
 cp -r pentaho-style $(DESTDIR)/opt/pentaho/pentaho-server/server/webapps/pentaho-style    
 cp -r ROOT $(DESTDIR)/opt/pentaho/pentaho-server/server/webapps/ROOT    

install-solution:
 mkdir $(DESTDIR)/opt
 mkdir $(DESTDIR)/opt/pentaho
 mkdir $(DESTDIR)/opt/pentaho/pentaho-solution
 cp *.* $(DESTDIR)/opt/pentaho/pentaho-solution/
 cp -r system $(DESTDIR)/opt/pentaho/pentaho-solution/system
 cp -r admin $(DESTDIR)/opt/pentaho/pentaho-solution/admin


install-stage:
 mkdir $(DESTDIR)/opt
 mkdir $(DESTDIR)/opt/pentaho
 mkdir $(DESTDIR)/opt/pentaho/pentaho-server-stage
 mkdir $(DESTDIR)/opt/pentaho/pentaho-server-stage/server  
 mkdir $(DESTDIR)/opt/pentaho/pentaho-server-stage/server/webapps
 cp -r pentaho $(DESTDIR)/opt/pentaho/pentaho-server-stage/server/webapps/pentaho
 cp -r pentaho-style $(DESTDIR)/opt/pentaho/pentaho-server-stage/server/webapps/pentaho-style    
 cp -r ROOT $(DESTDIR)/opt/pentaho/pentaho-server-stage/server/webapps/ROOT    

install-solution-stage:
 mkdir $(DESTDIR)/opt
 mkdir $(DESTDIR)/opt/pentaho
 mkdir $(DESTDIR)/opt/pentaho/pentaho-solution-stage
 cp *.* $(DESTDIR)/opt/pentaho/pentaho-solution-stage/ 
 cp -r system $(DESTDIR)/opt/pentaho/pentaho-solution-stage/system
 cp -r admin $(DESTDIR)/opt/pentaho/pentaho-solution-stage/admin

Dependencies

Note that this is depends on having rpmbuild installed on your system. Would work on linux and mac, and even eventually on windows with cygwin.

After running the create-rpms ant target, you should get the RPMs in the dist/rpm/RPMS/ directory.

Next steps

You probably know what you want to do with the RPMs. You can either install them directly, or if you're using puppet you may want to set up a yum repository. Lots of tutorials on how to do this, here's one.

Packaging Enterprise Edition

By default, this targets know how to handle the CE version. We can change a few options and tell it to pack the Enterprise Edition, just by setting some extra options pointing to a different location where we unpacked the EE version:



######################################
## EE BASED RPM BUILD
######################################

rpm.source.webapp = /home/pedro/tex/pentaho/ee/4.8/biserver-ee/tomcat/webapps/
rpm.source.solution = /home/pedro/tex/pentaho/ee/4.8/biserver-ee/pentaho-solutions/
rpm.source.administration-console = /home/pedro/tex/pentaho/ee/4.8/enterprise-console

New Ctools releases - 13.02.07

2013-02-12T18:37:00.000Z

Here it goes, a new release available:

CDF

Major Change: Async behavior and whole new component lifecycle.

Full changelog:

Introduced Async Behavior and new component life cycle
Initial Storage fetch moved to server side
Added a separate resource for all js shims
Refactored CoreComponents into smaller pieces
Fixed require.js dependencies. Had to reintroduce a resource called CoreComponents so that EE 4.8 still finds it when resolving dependencies
Fix styling issues with widgets
Upgraded jquery sparkline to 2.1
Fix 'notNullMeasure' function in OlapUtils
Added Jasmine testing library
Added support for secondAxis minimum and maximum value (by Andrea Torre)
Add support for arbitrary user data in Views
Set TableComponent to use async ajax
changed deps to newer pentaho version
Implemented legacyProperty support
Changed the way "earliestDate" and "latestDate" are retrieving their value. Either set by a property, or with a default value (no Dashboards.getParamaterValue)

CDE

Major Change: Support for async components and their new lifecycle.

Async behavior for components
Add NewSelector component
Added javascript unit tests
Exposing 4 properties (2 for setting date, 2 for setting offsets) under the date range input component. Removed the "getParameterValue" call for setting up the date, as it looked unpractical.
Added method that allows file/folder deletion
Added support for css url transformation for components from other plugins
Allowed res to work with symlinks

CDA

Added JSONP support to CDA
Added binary exporter
Implemented setMdxDataFactoryBaseConnectionProperties. Mdx quereies, when executed in 4.8, will pass the correct properties that are defined in olap/datasources.xml, namely UseContentChecksum=true;JdbcConnectionUuid=FOOBAR
Support for blobs encoded as java.sql.Blob
CacheActivator: only commit if no exception, rollback otherwise
Upgraded Hazelcast to version 2.4

CDC

Fixed timeToLive bug
Updated Hazelcast to 2.4
Can now properly persist cache contents with stock mondrian;
Added new settings: master, syncCacheOnStart
Fixed CDC after async and component lifecycle changes

Pentaho Bigdata - 101 to a bit more

2013-02-01T20:16:00.000Z

Motivation

Do I need one? Haven't you read the news? It's bigdata, this will make us all rich!

I'm not one of the voices that claim that this is the best invention since the wheel. There's a lot of hype out there regarding bigdata, and vendors desperatly seeking business opportunities are on top of this appear every day.

Having said all that, hadoop is an amazing framework. It's one more tool at hand to be chosen for specific set of problems. It's getting easier and easier for companies to adopt it and really increases the power to do certain type of ad-hoc analysis that would be impossible otherwise. But there's some things that should be said out loud:

Building and maintaining an hadoop cluster is expensive
You need a team that knows what it's doing - more than the cost of big data is the cost of bad data
Not everyone needs it

Before someone jumps at me: cost is obviously relative. We're talking about

Hadoop or BigData?

There's a huge difference between Hadoop and Big Data. Simply put, Hadoop is a framework that provides a reliable shared storage, provided by HDFS, and a processing framework given by MapReduce. As we dig deeper there are other pieces of the puzzle that start to appear but theses are the fundamental ones.

This is just engineering talk. Big Data is what you do with it. And that makes all the difference. Write a check for a cluster, install hadoop on it and you'll end up with a bunch of noisy machines and 0 added value. The real challenge starts there - what you do with the data.

The challenge

At Mozilla, one of the Hadoop clusters of the Metrics team uses is about 60 nodes, and it is used to store / process several different data sources., stored between hdfs, hbase, hive and a bunch of foreign sound words that meant little to me. It was about time
The initial goal sounded relatively easy.

Analyze a bunch of weblogs that are stored in hdfs, process them using geo localization and find out how many users per country saw the web pages

Now... where do I start? I know other engineers from the team that write their own java code for the map reduce jobs. I'm too old for that.

I heard that Pig would also be an option. The last thing that I need right now is having to learn yet another technology - unless absolutely necessary.

I had heard about all the work Pentaho has done with Big Data Analytics but never quite understood what all that was about. But the idea of being able to use an extremely powerful ETL tool that me and my team have been using for ages with very good results is, to say the least, appealing.

But the first step has nothing to do with it. For me is to answer the question: What exactly is hadoop and how does it work anyway?

Hadoop 101

This link proved to be a great resource to get me up and running. I have access to the staging, research and if needed production cluster at Mozilla but using it as an experimentation ground doesn't make me comfortable at all. So I decided to install it locally and try to get it working.

Basic concepts: Hadoop

Hadoop was created by Doug Cutting and Michael J. Cafarella and was originally developed to support distribution for the Nutch search engine project.

There are lots of components on hadoop, but the core is divided into 2 main subprojects:

MapReduce - A framework that schedules and assigns jobs and tasks on the cluster
HDFS - A distributed file system that guarantees scalability and reliability

There are some important services running on the cluster. Mapreduce work is managed by the Job Tracker, running on the master and handed over to the different Task Trackers on the nodes.

On the data side, the master runs a Name Node that keeps a reference to every file and block in the file system, and talks with the different Data Nodes throughtout the slaves in the cluster.

One of the big advantages of mapreduce over the generic concept of grid computing is it's ability to process the data that is stored locally; The planners try as much as possible to reduce bandwidth usage by processing local data.

There's obviously a lot more around, but I'll stop here for the sake of simplicity.

Installing Hadoop

Like what happens in Linux, even though the main project is driven by Apache, there are several different distributions, that ensure that all the independent hadoop sub projects are correctly configured and ready to talk between them.

The main providers are HortonWorks, Cloudera and MapR. At Mozilla we use Cloudera's CDH3, so that's what I chose to install.

Downloading the Virtual Machine

I chose to install a pre-configured virtual machine. I'm not very interested in the small details of configuration, so a one-node cluster is more than enough to get me started.

Cloudera provides pre-built virtual machines you can use, and in different formats. I use Virtual Box, so that's the one I used. This excellent post provides details on how to install it, plus a great overview of hadoop.

In the end, booting the VM will result in something like this:

Almost ready to start playing with your system. There's only some extra network changes we need to ensure communication between host and client.

Network configurations changes

There are some configuration changes that will prove to be important. I don't want to do everything from within the VM, I also want to be able to connect from my host machine to it, and run kettle connected to it.

First step is to configure the network interfaces in virtual box. For vmware or others the instructions may vary. I defined 2 network adapters, one with Nat to allow for outside connections and one Host-only adapter. This will allow a static ip connection between the host and the client:

If this is correctly configured, you should see an extra interface in your host with ip 192.168.56.1, and your client would have 192.168.56.101. For convenience, I chose hadoop-pedro for my machine (lousy name, but my text, my name!), so I changed the following configuration files:

Client:

/etc/sysconfig/network - Adding HOSTNAME=hadoop-pedro
/etc/hosts

$ cat /etc/hosts
127.0.0.1        localhost.localdomain localhost
192.168.56.101   hadoop-pedro hadoop-pedro.local
::1              localhost6.localdomain6 localhost6

Host:

/etc/hosts - Add the following line:

$ cat /etc/hosts | grep hadoop-pedro
192.168.56.101 hadoop-pedro hadoop-pedro.local

This should ensure proper communication between host and VM. You should be able to ping hadoop-pedro and get results from the hosts.

Hadoop configurations changes

CDH's configurations default to the local interfaces, and in order to guarantee that it works flawlessly when called from the hosts, I got better results by changing hadoop's configuration files to attach to the new hostname.

Hadoop is installed in /usr/lib/hadoop, and inside there's a conf/ directory that holds the configuration files.

There are some configuration changes that will prove to be important. The default configuration files makes hadoop's services listen to 0.0.0.0. I got better results by pointing to the specific IP address. So here's the properties I changed:

core-site.xml: Change fs.default.name to hdfs://hadoop-pedro.local:8020
mapred-site.xml: Change mapred.job.tracker to hadoop-pedro.local:8021 and jobtracker.thrift.address to hadoop-pedro.local:9290. Also add the following properties: mapred.map.child.java.opts to -Xmx768m and mapred.reduce.child.java.opts to -Xmx1536m

Reboot the VM for all the changes to have effect. Should be ready to go.

Knowing our way around it

Services

I really feel more comfortable knowing what happens in the system, what is running, how to restart, how to know what's happening. If you wanted to start the services manually, here's what should be run:

$ /etc/init.d/hadoop-0.20-namenode start
$ /etc/init.d/hadoop-0.20-secondarynamenode start
$ /etc/init.d/hadoop-0.20-datanode start
$ /etc/init.d/hadoop-0.20-jobtracker start
$ /etc/init.d/hadoop-0.20-tasktracker start

You'll probably notice this is absolutely coherent with the description of the different hadoop components that I described before. There are a few others, zookeeper and hbase for example, that we won't need for now.

If you want to stop the services... just run the opposite way.

The logs are under /var/log/hadoop/. To know what's going on, simply follow them:

$ tail -F /var/log/hadoop/*

Command line utils

Hadoop comes with a command line executable to interact with the system. You'll find the command hadoop on path (or under the bin/ directory of the hadoop distribution). Execute it without arguments to see how it works. The ones I use more often are hadoop fs to interact with hdfs and more infrequently hadoop job to query job execution.

Web utility ports

There are some important ports to look for:

Namenode / DFS status: http://hadoop-pedro:50070/

Information about the status of our filesystem cluster

Job Tracker: http://hadoop-pedro:50030/

One of the most useful ones. Displays information about running jobs and it's where we can inspect the output of the individual tasks running on the nodes.

Task Tracker: http://hadoop-pedro:50060/

Displays the status of individual tasks.

Hdfs

It's fundamental to know how to interact with hdfs. I use the command line $ hadoop fs . Once again, run without arguments to know what are the different options.

We can either run locally without specifying the hdfs server or remotely specifying the full VFS path:

$ hadoop fs -ls /

$ hadoop fs -ls hdfs://hadoop-pedro:8020/

If not present yet, I recommend creating a home directory for your user on hadoop, on my case... surprise... pedro.

hadoop fs -mkdir hdfs://hadoop-pedro:8020/user/pedro

The most commonly used commands are:

$ hadoop fs -ls : List files
$ hadoop fs -mkdir : Make directory
$ hadoop fs -put : Put local files into hdfs
$ hadoop fs -get : Get files from hdfs
$ hadoop fs -cat : Show the contents of a file in hdfs
$ hadoop fs -rm : Remove a file
$ hadoop fs -rmr : Recursively remove a directory

Pentaho Bigdata

Once I started to get familiarized with the hadoop infrastructure and starting to look at kettle, I was surprised about the level of documentation of Pentaho's Big Data plugin. This is not an easy concept. It's hard to use, hard to debug, lots of stuff to know. So having a Wiki with a good set of documentation aimed more at concrete examples is very good.

My first question was obviously "How do I start? What do I download?". The wiki suggests downloading a stable kettle version and you'd get up and running in no time. But that would be too easy, and we wouldn't understand what was happening behind the hood.

Compiling kettle

I always compile kettle from source. Everyone does that, right? :)

$ svn co svn://source.pentaho.org/svnkettleroot/Kettle/branches/4.4.1
$ cd kettle-4.4.1
$ ant clean distrib

Please note that I'm using the 4.4.1 branch. This is always changing. I don't yet feel confident about using 5.0, so pay attention to the one you should be using.

In the end, we'll get a ready to run kettle in the distrib directory. This doesn't have the bigdata plugin.

Bigdata plugin

Compiling

Next step is to compile the bigdata plugin. Fortunately this one's already on git.

$ git clone https://github.com/pentaho/big-data-plugin.git
$ cd big-data-plugin
$ ant

There's an important detail that made me lose a lot of time and is not obvious at all. I'll describe the details later, but the bigdata plugin prepares a zip of a bunch of jars and dependencies to copy to hadoop, and that's a static bundle. By default, points to TRUNK-SNAPSHOT, which means that will download the latest version of kettle, eventually causing incompatibilities with the kettle version we chose before.

You can edit the file build.properties and change the following line:

$ dependency.kettle.revision=4.4.0-stable

I'm not aware of any artifact that points to a continuous build of 4.x, so I chose the closest version available.

If you compile again you'll get a plugin version ready to use under the dist directory. Unzip in the plugins director.

$ tar -xzf dist/pentaho-big-data-plugin-TRUNK-SNAPSHOT.tar.gz -C ~/.kettle/plugins

Configuring

After installing the plugin, we need to configure it properly. There's an important file that needs to be changed.

$ vim pentaho-big-data-plugin/plugin.properties

You need to change the following properties:

active.hadoop.configuration = cdh3u4
pmr.kettle.dfs.install.dir = /user//pentaho/mapreduce
pmr.kettle.additional.plugins = steps/maxmind

Like I mentioned before, there are several hadoop distributions and each of them has made the modifications they considered necessary in order to ensure everything works well. This is a good thing. The bad thing is that 3rd party integrators have to comply with all the variants.

Pentaho developers did a great approach to try to minimize, to a certain extent, this problem. They developed a shim around the common hadoop code (if you like to mess with source code, you'll find it under the package org.pentaho.hadoop.shim.common) to comply with the variants.

Like I mentioned, I'm using Cloudera's CDH3u4. Luckily it's one of the supported versions. You can see the possible values by looking at the directory pentaho-big-data-plugin/hadoop-configurations/ . Currently the supported versions are:

cdh3u4
cdh4
hadoop-20
mapr

I'm sure this list will increase with time and relevance.

The second fundamental property is pmr.kettle.dfs.install.dir. This is where kettle will be copied to in hdfs in order for mapreduce to be able to find all the dependencies of our jobs/transformations. Due to the way permissions are setup on Mozilla's cluster, I have to use my remote username. So I pointed it to my home dir in /user/pedroalves/pentaho/mapreduce.

The third option is a fundamental one on my case. The bundle file that gets copied to hdfs and run on hadoop has only the core transformations and steps (and bigdata plugin, obviously) . On my case I wanted to add another one. The format is relative to the kettle directory, and my geoip plugin is under steps/maxmind.

Everything is ready to start using. If we now launch spoon, we should see the bigdata steps:

Running mapreduce tasks

Setting up the environment

Back to my initial challenge: Parse and geolocate weblogs. On my case, I wanted to know how many and which snippets were seen on a daily basis by country. Pentaho bigdata wiki has a very detailed example on how to achieve the majority of this, but lacked the geolocation step.

The files are stored in the main cluster, in a hdfs directory. I started by copying a sample of those files to my local vm, simulating the real environment:

$ hadoop fs -get hdfs://:8020/www_weblogs/dir/part-r-00000.gz .
$ hadoop fs -mkdir hdfs://hadoop-pedro:8020/www_weblogs/dir/
$ hadoop fs -put part-r-00000.gz hdfs://hadoop-pedro:8020/www_weblogs/dir/

Preparing the job and the transformation

My job is pretty simple, almost a direct call to pentaho mapreduce step:

We need to fill in some information related to this step:

Cluster information
Map transformation
Reduce transformation (if needed)
Combiner transformation (if needed)
Information about input and output

As usual best practices recommend, I used variables as much as possible. Here are the ones that I'm using, and should be self-explanatory:


# local
SNIPPET_HDFS_HOST=hadoop-pedro.localSNIPPET_HDFS_PORT=8020SNIPPET_JT_HOST=hadoop-pedro.localSNIPPET_JT_PORT=8021SNIPPET_HDFS_INPUT_PATH=/www_weblogs/snippets-stats.mozilla.org/dir/

SNIPPET_HDFS_OUTPUT_PATH=/user/pedro/tests/snippets

The input and output formats are org.apache.hadoop.mapred.TextInputFormat and org.apache.hadoop.mapred.TextOutputFormat. You can see in hadoop documentation the possible values to put here, always with the possibility to write your own. Same for output formats.

This is my transformation, ready to be executed:

Running the job

When I run the job, I immediately see in the logs the following lines:

INFO  01-02 16:49:52,665 - Spoon - Starting job...
INFO  01-02 16:49:52,666 - test_mapreduce_job - Start of job execution
INFO  01-02 16:49:52,668 - test_mapreduce_job - Starting entry [Pentaho MapReduce]
INFO  01-02 16:49:52,708 - test_mapper_with_geoip - Dispatching started for transformation [test_mapper_with_geoip]
INFO  01-02 16:49:52,817 - test_reducer - Dispatching started for transformation [test_reducer]
INFO  01-02 16:49:52,836 - Pentaho MapReduce - Cleaning output path: hdfs://hadoop-pedro.local:8020/user/pedro/tests/snippets
INFO  01-02 16:49:52,841 - Pentaho MapReduce - Installing Kettle to /user/pedroalves/pentaho/mapreduce/4.4.0-TRUNK-SNAPSHOT-cdh3u4

This looks good. However, a few moments (or minutes, depending on where the cluster is), I get a few less motivating messages:

INFO  01-02 16:50:01,687 - Total input paths to process : 1
INFO  01-02 16:50:01,843 - Pentaho MapReduce - Setup Complete: 0.0 Mapper Completion: 0.0 Reducer Completion: 0.0
INFO  01-02 16:50:06,844 - Pentaho MapReduce - Setup Complete: 0.0 Mapper Completion: 0.0 Reducer Completion: 0.0
INFO  01-02 16:50:11,857 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 0.0 Reducer Completion: 0.0
INFO  01-02 16:50:16,861 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 0.0 Reducer Completion: 0.0
INFO  01-02 16:50:21,878 - Pentaho MapReduce - Setup Complete: 100.0 Mapper Completion: 0.0 Reducer Completion: 0.0
ERROR 01-02 16:50:21,920 - Pentaho MapReduce - [FAILED] -- Task: attempt_201301301222_0006_m_000000_0  Attempt: attempt_201301301222_0006_m_000000_0  Event: 1
java.io.IOException: org.pentaho.di.core.exception.KettleException:
We failed to initialize at least one step.  Execution can not begin!


        at org.pentaho.hadoop.mapreduce.PentahoMapRunnable.run(PentahoMapRunnable.java:467)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
        at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: org.pentaho.di.core.exception.KettleException:
We failed to initialize at least one step.  Execution can not begin!


        at org.pentaho.di.trans.Trans.prepareExecution(Trans.java:932)
        at org.pentaho.hadoop.mapreduce.PentahoMapRunnable.run(PentahoMapRunnable.java:354)
        ... 7 more

And loops until I stop the job. From the log messages I would have absolutely no idea what was going on.

Under the hood

I had no option but to try and go deeper into understanding what happens under the hoods. To change that I had to go to the source of the information... literally.

The approach is actually pretty simple, and follows the instructions on practically every hadoop book, but tweaked so that we can execute transformation without the hassle of writing pure java code. Here's the sequence

Detect the shim we chose. This will guarantee later on that the specifics of each distribution is respected
The mapreduce step is processed to get:

The configuration for the mapper
Configurations for the combiner
Configurations for the reducer
Input and Output formats
Cluster information
Input paths
Output paths
User defined configurations
Number of map and reduce tasks

Our set of kettle variables will be passed to the hadoop configuration, ensuring all the environment stays the same
The output path is deleted, if that was the chosen option
Bigdata plugin properties are read to determine the kettle installation directory. This depends on the kettle version, so a single cluster supports the usage of different versions at the same time
Checks if kettle is already installed in hdfs. It does that by seeing if the chosen hdfs directory exists (on my specific case evaluated to: /user/pedroalves/pentaho/mapreduce/4.4.0-TRUNK-SNAPSHOT-cdh3u4 ) and if it has the subdirectories lib and plugins. This is bound to change in the future, as it clearly inefficient and unable to detect changes to the content of those directories
The kettle archive (pentaho-big-data-plugin/pentaho-mapreduce-libraries.zip), bigdata plugin and the extra plugins we specified
Everything is registred in haddop's DistributedCache, for local file access and classpath registration
The job is finally submitted to execution

Debugging the transformation

Once the job is submitted, we will be able to track it's execution in hadoop's Job Tracker at http://hadoop-pedro:50030/

If you follow the link to get more details on the running job, you'll be able to get details on the specifics of the job configuration and the specified tasks. On my case, following the link on the map task I'm able to see the exception thrown by the mapper transformation

If we click on one the task, we'll be able to see all task attempts that have been made. And individually access the task logs. And there is the very familiar kettle output, with a line that clearly states what's going on.

ERROR 31-01 03:10:07,458 - Lookup Country - Error initializing max mind database file location '/usr/local/share/GeoIP/GeoIPCity.dat'
ERROR 31-01 03:10:07,458 - Lookup Country - org.pentaho.di.core.exception.KettleStepException: 
Unable to set up MaxMind database '/usr/local/share/GeoIP/GeoIPCity.dat'
/usr/local/share/GeoIP/GeoIPCity.dat (No such file or directory)

I will spare you of all the pain that we had to go through to fix this. This seems simple but it's not, it was a very very hard task to ensure that the .dat files were available on all the nodes. In the end, Matt Casters and I completely rewrote the Maxmind plugin step, which is now also on github, to support VFS.

Pro tip: Everything happened to me. Apparent thread locks, the system totally hanging with 100% cpu usage and no log output anywhere, that I eventually traced down to memory usage. One trick I managed to use and was very useful was to send a QUIT signal to the task process (with kill -QUIT ). Despite the scary name, this will cause the JVM to do a thread dump, allowing us to spy on what it's doing. This tip is true for any java program.

After several days changing the plugin and debugging the origin of the problem, I finally discovered that by default mapreduce tasks run with a maximum memory of -Xmx200m. Since I was using the city level geolocation, that value was clearly insufficient to run the transformation, that ran into OOM/GC issues, which didn't happen when I used the geo location only at country level. So do yourself a favor - increase the available memory on the cluster.

Pentaho still needs to improve the debugging abilities of pentaho bigdata plugin. Like I wrote on my last post I ended up developing a change to the WriteToLog step to allow displaying only the top N rows of the dataset. Helps a bit until they allow us to do proper debugging from within spoon like any regular transformation.

After all the changes to the maxmind step and increasing cluster memory, I ended up copying the GeoIP files to my hdfs user directory and specified the location using the following variables (the step also supports variable substitution - thanks Matt!)

maxmind.geoip.path = hdfs://hadoop-pedro:8020/user/pedro/geoip/GeoIP.dat
maxmind.geoipcity.path = hdfs://hadoop-pedro:8020/user/pedro/geoip/GeoIPCity.dat

Running the transformation again results in a successful run of both map and reduce tasks!

And we have access to the output

maxmind.geoip.path = hdfs://hadoop-pedro:8020/user/pedro/geoip/GeoIP.dat

$ hadoop fs -cat hdfs://hadoop-pedro:8020/user/pedro/tests/snippets/part-00000 | head -n 40
13/Aug/2012|Afghanistan|1234|31
13/Aug/2012|Afghanistan|2345|7
13/Aug/2012|Aland Islands|3456|16
13/Aug/2012|Aland Islands|4567|1
13/Aug/2012|Albania|5678|7
13/Aug/2012|Albania|7890|5

# get all the files in case there were multiple reducers running

$ hadoop fs -getmerge hdfs://hadoop-pedro:8020/user/pedro/tests/snippets/ result.txt

Final Remarks and Credits

It was a very tough week, but absolutely fundamental for me to understand how things work. Hadoop is an amazing framework, and being able to take full advantage of kettle to run our map reduce analysis is a huge bonus. There's still a lot of user experience improvements around these steps, but considering the alternative is to write java code manually or learn other new languages make this a great start.

This blog post was never meant to be a full pentaho bigdata tutorial. What I hope is that from this point on, understanding in detail how things work and what happens when we press the "run" button allows me to do further development with much more speed, since I know exactly where to look for.

I also got the chance to understand the very basics of how hadoop works, and know what each of the components does. Next step will be digging into hbase and hive.

I was also a bit suspicious of the performance and overhead of executing kettle transformations on hadoop, and how it would compare with pig. Having seen the code and how lightweight the wrapper around kettle is, I have no doubts that using kettle instead of learning new stuff or approaches is, indeed an astonishingly efficient way to run map reduce jobs.

Need to credit a bunch of people that helped me throughout this last week. Mark Reid, Xavier Stevens and Daniel Einspanjer from Mozilla, Doug Moran, Matt Casters and Matt Burgess from Pentaho and Maria Roldan from webdetails. I'm aware I was a royal PITA the last few days :)

Debugging kettle tasks in MapReduce - Sane WriteToLog

2013-01-23T15:48:00.000Z

Finally started to play with bigdata and pentaho. On my specific case, Cloudera CDH3u4. At Mozilla we have a few clusters of over 80 machines that we're using to backup a bunch of services

Debugging mapreduce tasks

It took me a while to get my head around the concepts of how kettle integrated with the mapreducer tasks. When I did, the first thing I noticed is how complex it is to know what's happening. Until Matt Casters and friend get the chance to implement PDI-9148, we need to do things manually - as in inspecting logs, etc.

My first approach was writing to text files. I tested direct output to hdfs, but for some reason didn't work. Using direct file system means that output will be spread through all the cluster nodes. This approach generally sucks.

I also thought about using some hand-made logic in a javascript step, but then looked at the WriteToLog step. This step generally works, but with a great flaw on it; it has no way to limit the output of it. If we have millions of rows, we'll have a huge log generated - and that's not good.

An improved Write To Log step

If it's not there, just do it yourself, the code is open. So I did. I added the ability of specifying a limit to the output of the step. This is very useful to inspect how the dataset is looking inside a map or reduce task. Once I deployed this change to my cluster, this is how my tasktracker log looks like (I ran this with a previous writeToLog version and ended up with a crashed browser and almost half a gigabyte of log files). This shows the first 5 lines of our dataset, with the key and value of our dataset:

I'll work with the kettle team in order to put this into the main code line, hopefully will be in 4.4.1 and 5.0. This is PDI-9195

CDE / CDF components and templates in Pentaho solution directory

2013-01-08T13:06:00.000Z

Someone brought to my attention that this is a very useful, though undocumented feature. TL;DR, you can have templates and components in the solution directory and not in the plugin directory, that gets wiped on upgrades

It's a common requirement having to develop templates or components to extend the capabilities of the Ctools. The normal place is to put it in the solution/system/plugin directory, along the others.

However, that has a huge inconvenience - whenever we upgrade / reinstall the plugin we need to copy the resources back, not forgetting to do a backup before.

That's not actually needed since a while back. If we put the resources directly under the pentaho-solution// all the Ctools will know what to do.

Here's a real life example that can serve as reference:

cde
├── components
│   ├── EmailPrpt
│   │   ├── component.xml
│   │   ├── emailPrpt-implementation.js
│   │   └── emailPrpt.xaction
│   └── VideoGallery
│       ├── ceebox-implementation.js
│       ├── component.xml
│       ├── css
│       │   └── ceebox.css
│       ├── images
│       │   ├── cee-close-btn.png
│       │   ├── cee-next-btn.gif
│       │   ├── cee-next-btn.png
│       │   ├── cee-prev-btn.gif
│       │   ├── cee-prev-btn.png
│       │   └── loader.gif
│       ├── jquery.ceebox.js
│       └── jquery.swfobject.js
├── styles
│   └── Clean.html
├── templates
│   └── index.xml
└── widgets
    ├── IncomeStatementDetailTable.cdfde
    ├── IncomeStatementDetailTable.component.xml
    ├── IncomeStatementDetailTable.wcdf
    ├── index.xml
    ├── sample.cda
    ├── sample.cdfde
    └── sample.wcdf
cdf
├── components
│   ├── jfreechart-cda.xaction
│   └── traffic.xaction
├── includes
│   ├── index.xml
│   └── Operations
│       ├── facilityAccount.cda
│       ├── facilityAccount.cdfde
│       ├── facilityAccount.wcdf
│       └── index.xml
└── index.xml

CDF Async Support

2013-01-04T17:46:00.003Z

Introduction

This is a huge change! Since the beginning, CDF behaved in a synchronous way. The problem is that with several components, performance suffers with it. This change represents a major overhaul of the main CDF code in order to change that. Now, if a priority is specified, all components are executed simultaneously, speeding up the render of a dashboard.

We tried to maintain backward compatibility. Since the new async behavior requires a new - and simpler - way of defining component interaction, by default old dashboards will still render in a "fake synchronous" mode, by applying a specific heuristic where sequential sets of priorities are assigned to components, emulating old behavior.

This blog post (who's contents are also available in CDF's documentation) is a guide to converting old components and dashboards to the new async style, and developing new ones based on asynchronous querying.

This is currently in dev and will soon make it's way into stable releases. CDE support is obviously included

Rationale

The first step to understanding the changes in the async patch is understanding the CDF component lifecycle. When a component is updated, the basic update lifecycle looks like this:

preExecution -> update -> postExecution

Usually, though, there will be a call to a data source, with a subsequent call to postFetch, and only then is the component rendered:

preExecution -> update -> query -> postFetch -> redraw -> postExecution

This is a more typical lifecycle, and one that has some important limitations. First, preExeuction and postExecution are entirely the responsibility of CDF itself, rather than the component. Because CDF has no control over the contents of the update method, it has no way of ensuring that, should the component execute an asynchronous query, postExecution only runs after redraw. In this case, you're likely to see this instead:

preExecution -> update -> postExecution -> query -> postFetch -> redraw

Which breaks the contract for postExecution running after the component is done updating. The solution here is that the component itself must take control of postExecution, while keeping the burden of implementing the lifecycle in CDF rather than passing it to the component developer. On a related topic, postFetch has become a de facto standard part of the lifecycle, yet its implementation was left to the component implenters, which leads to a fairly large amount of boilerplate code.
Our objective here was to retool the base component so as to deal with both of these issues, thus allowing queries to be performed asynchronously while reducing the developer effort involved in creating a component.

Component execution order and Priority

There are no major changes in the way components behave. There is, however an important caveat - since all components (that have been converted) will be executed simultaneously, we can no longer rely on the order of execution.
There's now an additional property named priority. The priority of component execution, defaulting to 5. The lower the number, the higher priority the component has. Components with same priority with be executed simultaneously. Useful in places where we need to give higher priority to filters or other components that need to be executed before other components.
This way there's no longer the need to use dummy parameters and postChange tricks to do, for instance, cascade prompts.

Backward compatibility and changes

We did a big effort in order to maintain backward compatibility, but some care has to be taken. What we do is assume that if components have no priority, we give them a sequential value, trying to emulate the old behavior. It's recommended that proper priorities are set in order to take advantage of the new improvements.
If using CDE, please note that if you edit a dashboard and save it, all components will have a default priority of 5. This may break the old behavior. If you need to change a dashboard, make sure you tweak the priorities, if needed.

Developing Components

Components desiring to use asynchronous queries should inherit from the new UnmanagedComponent, instead of BaseComponent. The UnmanagedComponent base class provides pre-composed methods that implement the core lifecycle, for a variety of different scenarios:

synchronous implements a synchronous lifecycle identical to the core CDF lifecycle.
triggerQuery implements a simple interface to a lifecycle built around Query objects.
triggerAjax implements a simple interface to a lifecycle built around AJAX calls.

Since all these lifecycle methods expect a callback that handles the actual component rendering, it's conventional style to have that callback as a method of the Component, called redraw. It's also considered standard practice to use Function#bind or _.bind to ensure that, inside the redraw callback, this points to the component itself.

Use synchronous if Your Component Doesn't Use External Data

Components that don't use any external data at all can continue subclassing BaseComponent without any change of functionality. However, for the sake of consistency (or because you want querying to be optional -- see the section for details), your can use subclass UnmanagedComponent and use the synchronous lifecycle method to emulate BaseComponent's behaviour:

update: function() {
  this.synchronous(this.redraw);
 }

If you want to pass parameters to redraw, you can pass them as an array to synchronous:


update: function() {
  /* Will call this.redraw(1,2,3) */
  this.synchronous(this.redraw, [1,2,3]);
}

Use triggerQuery when You Want Your Component To Use CDA/Query Objects

If you're using a CDA data source, you probably want to use triggerQuery to handle the component lifecycle for you. triggerQuery expects at a minimum a query definition and a redraw callback to process the query results. The query definition is an object of the form:


{
  dataAccessId: 'myQuery',
 file: '/path/to/my/datasourceDefinition.cda'
}

Typically, if you're using CDE, these properties will be added to one of either this.queryDefinition or this.chartDefinition so you can just use this pattern:


update: function() {
 var redraw = _.bind(this.redraw,this);
 this.triggerQuery(this.queryDefinition, redraw);
}

Alternating Between Static And Query-Based Data

As the lifecycle methods are completely self-contained, you can switch between them at will, deciding on an appropriate lifecycle at runtime. A common pattern (used e.g. in SelectComponent, and the CccComponent family) is exposing a valuesArray property, and using static data if valuesArray is provided, or a query if it is not. Using UnmanagedComponent, this convention would like like this:


update: function() {
 var redraw = _.bind(this.redraw,this);
 if(this.valuesArray && this.valuesArray.length > 0) {
  this.synchronous(redraw,this.valuesArray);
 } else {
  this.triggerQuery(this.queryDefinition,redraw);
 }
}

Rolling Your Own

If you prefer having absolute control over your component, you can eschew the use of any of the lifecycle methods. Instead, you're expected to follow these guidelines:

Call this.preExec() as soon as possible, and bail out if it returns false.
If this.preExec() returned true, call this.block() before any meaningful amount of work is done.
If you called this.block(), make sure to always call this.unblock() as well once all relevant work is done.
If you want to use any sort of AJAX, consider using triggerAjax()
Call this.postExec() once all processing is done
You can override this.block and this.unblock to implement component specific UI blocking. If you override either, you must override the other as well.

New and Changed Features

Component Cloning

If your component holds any references to other components, you need to override the clone method so as to ensure that you don't accidentally clone the target component. For example, if your component has a property named otherComponent pointing at another component, you should override clone using this general template:

clone: function(parameterRemap,componentRemap,htmlRemap) {
 var other = this.otherComponent;
 delete this.otherComponent;
 var that = this.base(parameterRemap,componentRemap,htmlRemap);
 this.otherComponent = that.otherComponent = other;
 return that;
}

New Base Component Class: UnmanagedComponent

UnmanagedComponent is a new base class for components. It provides the base on which all asynchronous components should be built.

Per-Component `isManaged` Flag

Each component should have a member property with isManaged, indicating whether CDF should managed the component's lifecycle. Components where isManaged is false need to implement it's own calls to the lifecycle

Component Stub

Here's an example of a stub to be used whenever you need to use a new component. Just override the redraw function to what you need and define the component as you'd usually do

ExampleComponent = UnmanagedComponent.extend({

  update: function() {
    var redraw = _.bind(this.redraw,this);
    if(this.valuesArray && this.valuesArray.length > 0) {
      this.synchronous(redraw,{resultset: this.valuesArray});
    } else {
      this.triggerQuery(this.queryDefinition,redraw);
    }
  },

  redraw: function(data){

    /* Specific code goes here */
    if(!this.isInitialized){
      this.compiledStr = Mustache.compile(" Got a result set with  {{nrRows}} rows and {{nrCols}} columns 

");
      this.isInitialized = true;
    }
    $("#"+this.htmlObject).html(this.compiledStr({
      nrRows: data.resultset.length||0,
      nrCols: data.resultset[0]?data.resultset[0].length || 0:0
    }));

  }

});

customComponent = 
  {
  name: "regionSelector",
  type: "example",
  parameters:[],
  valuesArray:[["1","Lisbon"],["2","Dusseldorf"]],
  priority: 5,
  htmlObject: "sampleObject",
  executeAtStart: true
};

Debugging lifecycle

In order to be able to more easily track the lifecycle of CDF, we added some extra debugging features. If you use Firefox's Firebug or a recent version of Chrome (I believe >= 24) you'll be able to track the execution in a nice way:

Chartmas gift: CCC 2 released

2012-12-20T11:36:00.002Z

I admit we're really bad at at Seasons' Greetings. I also assume that the worst thing I could do was to start wishing peace on earth with a Jingle bells, as I surely would end up listed in a website like this.

So we decided to do what we do best - deliver software!

The team worked really hard to be able to release, still during 2012, the new version of CCC, our Community Charting Components. As usual, totally opensource and free of charge - The only requirement is that you enjoy it!

If you want to see a little bit of all the changes around it, feel free to browse the demonstration site we put up for you:

http://www.webdetails.pt/ccc2/

Since this requires a lot of testing, we created a separate branch for CDE with CCC2. During January we'll merge it to the main code line, as we feel confident about backward compatibility.

Have fun, and a great 2013!

-pedro

Sharing Mondrian cache in Pentaho

2012-12-14T18:21:00.000Z

Problem

Conceptually, this should be very simple. You have a mondrian cube, Sales, for instance. You build analysis, reports, dashboards, etc, on it. You would expect that issuing the same query though all of them would result in Mondrian getting the cells from cache, right?

Wrong

Why?

The problem is that every connection type does it's own access type to mondrian. Some examples:

Xactions and Analyzer uses MDXConnection
PRD builds it's own connections
CDA uses PRD
Saiku uses XMLA Servlet
Jpivot uses 2 different methods for a single query

If your datawarehouse is small, you may opt to ignore all this. But for large chunks of data this gets important, as all the queries are done several times, impacting performance and memory usage.

Approach

The root of the problem is in mondrian, and the methodology it uses to determine if a connection is new. Basically it builds an internal composite key made of 2 distinct informations:

Key = <is it the same schema?> + <is it the same connection?>

Depending on the origin of the query, each of this informations are different. There's already a way to solve the schema part, by telling mondrian to use the content checksum through the connection flag UseContentChecksum. This may not be the best option if we use Dynamic Schema Processors but it's the best we got.

The second part generated lots of discussion. Finally, Julian Hyde unleashed his hammer of Mondrian architect and decided that the best approach was to implement another property called JdbcConnectionUuid that would uniquely identify that connection. From that point on, it passes to the persons responsible for configuring the system the burden of ensuring the connection is the same. You can follow all the details in this jira issue.

Implementation

The problem with this solution is that we (by we I mean a multi team of pentaho people and community) had not only to change mondrian itself, but all the pieces around it that connected to mondrian. Some samples of some of the jiras involved:

JIRA-1231: Mondrian changes
PRD-4003: PRD changes
BISERVER-7429: Bi server related changes to ensure properties are passed correctly
CDA changes to allow transparently read info from olap/datasources.xml

All of this has been fixed and implemented in Pentaho 4.8 (that includes Mondrian 3.5.0)

How to use it

In order to use this new approach, there are some things that need to be done

BI server

In solution/system/olap/datasources.xml you need to add the 2 properties mentioned above to each catalog's DatasourceInfo, ending with a line like this (example taken from the SteelWheels catalog):

Provider=mondrian;DataSource=SampleData;EnableXmla=true;UseContentChecksum=true;JdbcConnectionUuid=SteelWheels

In summary, one needs to add UseContentChecksum=true to all catalogs and a unique string to each of the catalog's JdbcConnectionUuid=foo.

Also - please vote on BISERVER-7641. I think this should be the default behavior.

Analyzer and Xactions

Just works

Report Designer

Unfortunately, PRD-4004 wasn't resolved. It still ignores the configuration we have in datasources.xml, but we have a way to explicitly set the options on the report. When editing a mondrian datasource in PRD there's a Properties button where we can add them

CDA

Since this commit, CDA is able to read datasources.xml. It's now on trunk and should be included on the first stable release of 2013.

Saiku

We need to make some changes in Saiku, since it ships it's own mondrian version. These are the steps that need to be followed:

Edit saiku/plugin.spring.xml and comment the lines referring to the properties datasourceResolverClass and saikuDatasourceProcessor
Delete mondrian jars from saiku/lib/. (eg: rm -f saiku/lib/mondrian* saiku/lib/olap4j* saiku/lib/eigenbase*)

We're working with the guys from Analytical Labs in order to include in saiku a script called saiku-shareMondrian.sh that would do there steps and should be available in the following days.

Jpivot

This thing is dead. Hacks after hacks to make the connections to mondrian. Use Analyzer or Saiku.

Multiple provider authentication in pentaho

2012-12-10T13:15:00.002Z

Objective

We needed to add support for multiple authentication in Pentaho. Currently, there are several providers that can be used, like LDAP, hibernate and jdbc, but it's not possible to use several at the same time.

Pentaho doesn't allow for that out of the box, but it's simple enough, so we created a project to do just that, than you can find on github

This project allows for that by implementing a bean that will cycle through all the desired providers

Source code

This project is hosted at https://github.com/webdetails/bi-platform-engine-security-multipleuserrole

Also, please vote in this jira issue to get this in the main pentaho source

How to use

Following this steps should get you going:

Compile the project

Just run ant and you shuold be all set

Deploy the jar in Pentaho

Copy the resulting file to pentaho's lib dir, (eg: /opt/pentaho/server/webapps/pentaho/WEB-INF/lib/).

Copy the muiltiple provider spring config files to solution/system

Copy the following files from resources/ to pentaho-solutions/system/:

applicationContext-pentaho-security-multiple.xml
applicationContext-spring-security-multiple.xml

Change pentaho-spring-beans.xml to load the new files

Instead of loading one of the individual files (defaults to hibernate authentication), tell pentaho to instead load the new configuration files.
pentaho-spring-beans.xml should then look something like:


<beans>
  <import resource="pentahosystemconfig.xml" />
  <import resource="adminplugins.xml" />
  <import resource="systemlisteners.xml" />
  <import resource="sessionstartupactions.xml" />
  <import resource="applicationcontext-spring-security.xml" />
  <import resource="applicationcontext-common-authorization.xml" />
  <import resource="applicationcontext-spring-security-multiple.xml" />
  <import resource="applicationcontext-pentaho-security-multiple.xml" />
  <import resource="pentahoobjects.spring.xml" />
</beans>

Note: This snippet is taken from pentaho 4.8, different versions may have files with different content

Change the list of providers in applicationContext-spring-security.xml

In applicationContext-spring-security.xml, look for a bean named authenticationManager, and add the providers you want. If you're using the sample file applicationContext-spring-security-multiple.xml, the 2 referenced beans are called daoAuthenticationProvider and daoAuthenticationProvider2. You're not limited to just 2 providers.

Edit applicationContext-spring-security-multiple.xml and applicationContext-pentaho-security-multiple.xml

Edit applicationContext-spring-security-multiple.xml and applicationContext-pentaho-security-multiple.xml for your case.

This is the part where you configure the types of authentication you want. Even if it seems complicated at first, you'll notice that the 2 configuration files for multiple authentication are simply a concatenations of the individual files provided by pentaho, making sure the bean names don't colide.
On this example case we have hibernate and memory authentication joined together.

Launch the bi-server

Launch the BI server and hopefully you're all set.

Troubleshooting

Of course this won't work out of the box. Pay close attention to the logs. One of the most common errors is bean id collision, which is reported in the pentaho logs.
One other option is setting the spring logs to debug in log4j.xml

Questions / doubts

For any additional questions, feel free to get in touch: ctools at webdetails.pt

Pentaho Marketplace is here

2012-12-05T19:12:00.001Z

This is a huge milestone. Webdetails and Pentaho worked together to implement a Marketplace to the BI server. Will Gorman, VP of Engineering of Pentaho already had the chance to blog about this, but it's such a big deal that we want to do as much noise as possible.

There's also an equivalent effort for the kettle marketplace, but I'll focus on the platform side. The marketplace comes with the released version of Pentaho CE 4.8, available from sourceforge.

Objectives

The goals are simple:

Allow pentaho users to get in contact to what plugins exist
Provide a very simple way to install/maintain plugins
Get more users / feedback around existing plugins
Motivate new community contributions

For someone that is not familiar with the concept of pentaho plugins, he'll be surprised about the quantity - and quality - of some of them.

My hope is even that the marketplace can render ctools-installer obsolete, even if we absolutely plan to maintain it.

How to use it

We've put a lot of effort to make it a straightforward task. Basically run through the UI, and the tasks are what you'd expect

Launch the Marketplace

See the list of available/installed plugins

Check details on specific plugins

Install it

Reboot, and you're done

Report any issues you find

Can't get any simpler. If you find any issues, please report them

Do you want to contribute your own plugin?

Please do! See the instructions on Github for the marketplace-metadata project. As a side note, the Marketplace itself is a plugin, and the code available at github too. You can also get in contact with me or any of the vibrating community that's building this tools.

Installing the Marketplace in a older Pentaho version

Even if you don't have the latest and greatest version of pentaho, you can still benefit from the Marketplace. Just manually download the plugin from CI (choose your favorite flavor, tgz or zip) and drop it in your pentaho-solutions/system directory.

ps: please don't use the comments section for posting issues, use Pentaho forums

Making CDF calls asynchronous

2012-11-19T15:25:00.000Z

Motivation

Since the beginning, CDF has always worked in sync mode. Really not sure if it was a decision based on simplification or simply lack of skill. It's time to change that, though.

Having CDF components making synchronous requests simplifies the lifecycle. However, has a performance penalty, like the image shows:

We can very easily get into a big number of components. 6 selects, 3 charts and a table add up to 10 requests. In the synchronous world, means that we will only make one request after getting the answer back. In asynchronous mode, all requests are done at the same time* and then the dashboard will render much, much faster. There's also another problem; some browsers lock some of it's functionality while the request is being made. In Firefox, for instance, you can't even change tabs. That's bad.

CDF and CDE have tons of very important performance tweaks. All files are minimized and concatenated into one, all caching information is built so that as little requests are made and CDA has a very strong query cache layer, optionally backed up by CDC. Turning the lifecycle into asynchronous is the next step

Challenges

This is a big change, hopefully only for the CDF core and not for end users. Basically, in async mode, the concept of "postExecution" changes; Instead of being executed after the component executes itself, has to be called after the component renders itself. So all components will have to be changed, one by one, to conform to the new lifecycle to be implemented.

Currently all components inherit from BaseComponent. That's where the core lifecycle is. Now we're implementing a new one, called UnmanagedComponent.

We're not converting all the components now. It would be a very big effort. It's something that will happen over time. Our focus right now are the following ones:

Selectors
CCC Charts
Query component
Table component

This should give us a good performance boost to start with.

All this work is being developed as we speak, and you can follow the progress by monitoring the async github branch, .

Examples

Here are some samples of the old and new way of implementing components. This is the classic way of displaying a Hello World component

HelloBaseComponent = BaseComponent.extend({
  update: function() {
    $("#" + this.htmlObject).text("Hello World!");
  }
});

This is on the new, and soon to be recommended approach:

HelloUnmanagedComponent = UnmanagedComponent.extend({
  update: function() {
    var render = _.bind(this.render, this);
    this.synchronous(render);
  },
 
  render: function() {
    $("#" + this.htmlObject).text("Hello World!");
  }
});

This may not seem a big gain. However, if we need data from datasources, it's much easier to use:

HelloQueryBaseComponent = BaseComponent.extend({
  update: function() {
    var myself = this;
    var query = new Query(myself.queryDefinition);
    query.fetchData(myself.parameters, function(values) {
      var changedValues = undefined;
        if((typeof(myself.postFetch)=='function')){
          changedValues = myself.postFetch(values);                
        }
        if (changedValues !== undefined) {
          values = changedValues;
        }
        myself.render(values);
     });
  },
 
  render: function(data) {
    $("#" + this.htmlObject).text(JSON.stringify(data));
  }
});

In the new format, you'll just have to call the triggerQuery method

HelloQueryUnmanagedComponent = UnmanagedComponent.extend({
  update: function() {
    var render = _.bind(this.render,this);
    this.triggerQuery(this.queryDefinition, render);
  },
 
  render: function(data) {
    $("#" + this.htmlObject).text(JSON.stringify(data));
  }
});

Besides triggerQuery there are other useful calls: triggerAjax and synchronous, the first one for standard ajax calls and the second for synchronized calls, like used in the first example

*Caveat

I wrote that "In asynchronous mode, all requests are done at the same time". That's not exactly true as browsers limit the amount of concurrent requests. Depends on the browser, but it's something around 4 simultaneous requests. If we have 10 components, the requests will still be split into 2 or 3 batches.

Just to let you know that we're also thinking about this, but that's for another blog post ;)

New CDF Release 12.11.09 - Compatibility release with 4.8

2012-11-12T14:18:00.002Z

Pentaho 4.8 EE is out! Lot's of cool stuff there.

And obviously we had to make sure our Ctools were compatible with it. (Ok, I'm lying - one of our users tested it and reported some incompatibility :p ).

That incompatibility has been fixed and now there's a new CDF release (the only that had to be changed), that you can get in the usual place.

Ctools Releases: 12.10.17

2012-10-18T01:27:00.000+01:00

CDE:

Changelog:

* Added google analytics componente developed by Sinn Tecnologia

* Added about & documentation buttons to CDE.

* Added new features to CGG export window

* General image update

CDF:

Main features:

* Patches for CCC

* General image refresh

Full changelog:

* [FIX] Fixed linear scale calculation for second axis when all values are null.

* General image Update

* Added pv.Rule#strokeDasharray (with support for SVG only, not IE)

* [FIX] Fixed linear scale calculation for second axis when all values are equal.

* AMD compatibility with current Pentaho release

CDA:

Major upgrades:

* Nasty bugs fixed!

Full Changelog:

* Fixed [REDMINE 1159] - Cache Manager does not work when bi-server is in file based repository

* Fixed [REDMINE 1087] - json output problem when number is "infinity"

* Fixed [REDMINE 1217] - JSON infinity workaround

* Fix some instantiation issues with standalone mode

* Update build to include runtime-lib as a variable rather than a hardcoded value

* Image update

* add datatable filter on post-process; stateless TableModelUtils static instead of singleton

* AbstractKettleExporter: inherit from AbstractExporter

* NaturalOrderComparator: lic header, clean

* sort table model: cast where reflection is used

* CdaContentGenerator: bad outputIndexId not fatal, don't check null name for extension , bool parse

CGG:

Main features:

* Mainly bug fixes

Full changelog:

* Sync CCC with that of the CDF branch

* Updated platform dependency to 4.5 stable

* Added missing copyright for jquery bits

* Added missing output type declaration when no filename was specified

* Workaround for changes in detecting method name in draw

Data Services Pyramid

2012-10-15T23:22:00.001+01:00

Data Services Pyramid

While delivering services on Business Intelligence - or any of the other words you want to make up for it, Business Analytics, Data Science, whatever (I’ve always been more focused on actual content than on the wrapping some people like to put around it) - one very important part that sometimes ends up being a bit overlooked is how you hand out the information.

In the end it’s all about it. All the discussions carried out about technology choices, implementation, hardware, all ends up the same way - giving the information back to the consumers.

This may seem obvious. It’s not, and has a lot of implications especially when we have at our reach a huge amount of technologies to choose from.

This is written from the perspective of someone internal to a company’s metrics / data engineer team. I’m wearing my Mozilla metrics hat now, thinking about all the challenges we face on a daily basis.

Information delivery

One may be tempted to think that the best way to deliver information is the one where the user has the higher degrees of freedom when it comes to interact with that data. It’s not. With great power comes great responsibility, and sometimes less is better.

A metrics person has to guarantee not only the data he delivers is correct, but also that there’s no possibility of that data to be misunderstood. I find the second part harder than the first, especially when we’re dealing with scenarios with people that have been in a specific line of business for a long time.

Ground zero - Raw data

This is the example taken to the extreme. Raw data is, by far, the most valuable source of information, and the one where we can draw more information from. Obviously, makes no sense to anyone to hand out this source to information consumers. Very few hold the keys to the dungeon, and for good reasons.
You need to go up in the stack, progressively converting data language, the zeros and ones, into business terminology, to build your Data Services Pyramid

Data Services Pyramid

While growing our information system, we’ll inevitably want to end up with something like this. But instead of a structured stack, it’s very easy to end up with a disjointed mess of technology and services.

Should be easy to understand what kind of information deliverables sit on each level. They obviously depend on the tools at use, but some examples:

Raw data
- Handing out files
- Direct file system access
Storage area
- SQL
- NoSQL endpoints
- Hadoop / hive access
Ad-hoc layer
- Metadata based tools
- OLAP clients
- MDX
Preformatted deliverables
- Dashboards
- Reports
- Csv/xls exports

The Services challenge

We must have a very clear objective in mind - put users as far up the stack as possible. The goal is not to prevent them from accessing the data. Much more important, it’s preventing them from misinterpreting the data. Every time I see someone asking access to a database or hadoop, I start trembling - I know trouble’s coming.

As we move up in the stack, we’re converting data language into business language. This is a crucial point. As we work on this translation, we need to set in stone our final language as a shared set of dimensions, well documented terminology, that poses no questions to anyone whenever they’re used.

And this translation is hard, moving from data terms to business terms. The less people involved here, the better. That’s why I’ve identified, in the Data Services Pyramid, a danger zone that we should be very cautious with, mostly cause the fore mentioned translation isn’t done yet.

The Technology challenge

I once read a post about the differences between Google and Amazon when it came to inter-departments interaction, and how Bezos always insisted that such interaction had to be done by strict APIs and never in an unstructured way. What initially seemed a huge overhead and caused a lot of despair, the services oriented approach soon turned out to be Amazon’s greatest strength. After fine tuning all inter-department relations, wrapping those up as offerings to the outside world was easy - and that was in the origin of the cloud services offering.

For some reason this keeps popping up to my mind, and I’m the kind of person that can’t remember what had for lunch.

This principle should also apply for the technology stack. We’re in a golden era for IT junkies. Things are moving at the speed of light, every year or less there’s a new best thing in town and it’s getting harder and harder to keep up with everything and separate the hype from the real deal.

Staying frozen on time is just not an option. The complete opposite - always trying new things, new technologies and approaches - is equally dangerous. Gets to a point where too few people are familiar with the systems, they can easily get deprecated internally, the quality of the information stored is progressively harder to validate and comes with associated hardware costs, probably the cheapest of all.

This has to become a two step process. Inside a metrics team, every new technology has to pass through an approval stage. The goal is simple - there has to be a limited set of approved tools and technologies involved. Internally, one may prefer to use R, others perl, python, excel, etc. That’s perfectly fine and recommended, but when it comes to the official set of tools, everyone must be familiar with them. And make the list short, as there’s only so much one can know.

The second step is the link with my initial story about Amazon. If we apply the same principles to the technologies, we’re much less dependent on them, and gets much easier to swap out and optimize individual pieces.

If you think about the technologies and tools you already use and the ones you’re evaluating, they most likely fit into a very specific place inside the data services pyramid. This is where the services approach kicks in. Even though we can have more than one choice sitting at each level of the stack, it’s crucial that each one talks with the layer immediately below and is able to provide end points to the layers above to connect. Al the data translations that I mentioned before must happen only once, at a very well determined place. Data and overall service integrity is at stake here.
It’s now easy to see how changing specific bits gets less painful and error prone - as long as they maintain the same API interface to the enclosing layers.

--------------------------

Feedback appreciated. This is the result of my experience and would be great to have the ability to brainstorm on this issue.

CBF meets Ctools Installer

2012-10-03T11:35:00.003+01:00

Ctools-installer and CBF are very useful projects that usually go hand in hand. And if you've been using them for a while you probably thought "why can't CBF just call ctools installer instead of having to pass the parameters manually?"

Well, Andrea Torre went ahead and sent us a patch to allow exactly that.

There are a few new targets in CBF:


ctools-installer        Install ctools (prompts for modules)
ctools-installer-auto   Install ctools silently

The difference between them is obvious - one does an automated install (passing the -y flag) while the other prompts for which plugins to install.

All you need is to call this target. Here's an example:


ant -Dproject=demo -Denv=pedro ctools-installer

We can control the behavior of this feature. It's possible to specify which branch to install (defaults to stable branch) and specify if you want ctools-installer to run after a CBF build:


ctools.install = true
ctools.branch = dev

Ty it and let us know. You need to upgrade your ctools-installer script (just run it and it will auto upgrade)

On an also important note, CBF is now also on github.

CBF and Versioning - How to develop Pentaho solutions in a team

2012-09-21T12:16:00.001+01:00

This blog post is sponsored by the Antonius BI team

Problem

We're struggling with versioning and deployment. We need some help in managing the development and (especially) deployment process

This is a very well known problem. While versioning is a relevant issue in all areas - and not only development - the correct way to approach it changes drastically, and Pentaho is not different.

We've been working in pentaho projects for over 5 years. Since day 1, on scenarios where we had to manage:

Working on different projects locally
Working with multiple people on the same project
Managing several environments - development, staging, upgrades
Managing platform upgrades

There are more reasons, but this are probably the most pressing ones. It's not up to me to convince you that you need a good approach to this problems: you'll know you need it!

I'll write a collection of my experiences regarding this issue as of today. It can obviously change since we're always trying to optimize our processes.

I've been asked a specific set of question, that I'll introduce while contextualizing the big picture.

Versioning

There are 2 ways to approach this problem:

Simple pentaho-solutions versioning
CBF (Community Build Framework) setup

The first one is a subset of the second. There's no easy way to say it: The first one is a must have. If you're not doing it, you are asking for trouble.

As for the CBF... well, this is something Paul Stoellberger , Saiku author and general BI guru said on the irc channel:

< pstoellberger> i really need to get my cbf out again
< pstoellberger> just done hackish installations recently

Everyone doing something serious on pentaho uses CBF. You may think you don't need it. You may think it's complicated and doesn't worth the effort (since it's not an argument, we've put up a quickstart bundle for you). You're wrong, and you'll know it once you start using it.

VCS infrastructure

But one thing at a time. Before going through the specific workflows, you need to choose your VCS (Version Control System) tool. Once again there are 2 options:

Note: if you think I forgot to include CVS on this list, press Alt-F4 to close your Internet Explorer browser and go back to 1998, we don't want you here!

I'll skip the long arguments about those two. Use Git. It's amazing, handles branches and tag in a very efficient way, allows multiple remote repositories and has great UI tools that will be very handy.

Also bear in mind that Git is not Github. While you can definitely host your solutions in there, you're not forced to. I'd even say most of us would rather keep our files and implementation very securely locked.

So starting with infrastructure; Git doesn't even need a "server". Any shared directory could be used as the central repository. I've even used the "poor man's git server", initializing a repository ( git init --bare myproject/ ) in a dropbox folder. That has proven to be a very error prone approach, since there's no way to guarantee that our repository won't get damaged with the dropbox synchronization. So use a proper system. There are 2 options (this is getting a bit repetitive):

Use an existing one: Github (or something)
Install your own: Gitolite (or something)

Choosing github is a very valid and logical option. And doesn't mean your solutions have to be public, has you can have a payed account that will allow you to have private repositories. There should also be other alternatives on this lines.

We use Gitolite. Once installed, it's very easy to administer (creating repositories, adding / managing people and permissions) and very secure, as it uses ssh connections.

Regardless of what you chose, I'll now assume you have a proper VCS server available.

Versioning pentaho-solutions directory

Originally we included the entire CBF structure in the same repository, as described in the documentation. SVN allows us to checkout a subdirectory of the repository, so we could checkout only the project-client/solution/ folder. In git that's not possible and I could never get my head around submodules, so I simply have 2 different repositories (notice a trend here with the number 2?):

project-client
project-client-solution

The second has all the BI server solution files. The first one has... everything else, from CBF specific structure to ETL. After we clone the project-client repository we either link to the project-client-solution or clone that one inside the project-client directory.

Here's a real world example of one of our projects:

pedro@arpeggio:~/tex/pentaho/project-client (master)$ d
total 68
24 -rw-r--r-- 1 pedro pedro 20859 Feb 21  2011 build.xml.cbf-3.7
 4 drwxr-xr-x 2 pedro pedro  4096 May 21 16:41 config/
 4 drwxr-xr-x 2 pedro pedro  4096 Feb 17  2011 etl/
 4 -rwxr-xr-x 1 pedro pedro   138 Apr 27  2011 importCache.sh*
 4 -rw-rw-r-- 1 pedro pedro  1057 May 21 14:46 kettle.properties.diogo
 4 -rw-rw-r-- 1 pedro pedro  1118 May 21 14:46 kettle.properties.remote
 8 -rw-rw-r-- 1 pedro pedro  4534 May 21 14:46 kettle.properties.server
 4 drwxr-xr-x 4 pedro pedro  4096 Apr 19 12:57 patches/
 4 drwxr-xr-x 5 pedro pedro  4096 Feb 17  2011 patches-ee/
 4 -rwxrwxr-x 1 pedro pedro   261 May 21 14:46 remote_in.sh*
 0 lrwxrwxrwx 1 pedro pedro    29 Sep  6  2011 solution -> ../project-client-solution/

The project-client-solution is simply the pentaho solution folder without the system specific folder, admin, bi-developers, plugin-samples, steel-wheels, system. Tune this exclusion list at will in a file called .gitignore. Here's mine for the project-client-solution:

pedro@arpeggio:~/tex/pentaho/project-stonegate-solution (master)$ cat .gitignore 
admin/
system/
steel-wheels/
bi-developers/
*_tmp*
index*.properties
cde_sample/
.project
plugin-samples/

From this point on we can use the generic VCS techniques. Git can take a while to get used to, but the list of commands we need are very simple. I won't focus a lot on the project-client CBF structure, as there's lots of documentation on the CBF website, but everything still applies.

Moving on to the list of questions

FAQ: Frequently Asked Questions

Q: How do I checkout a project with git?

A: $ git clone git@yourserver.com:project-client-solution

This can be done not only for the development sandboxes (both project-client and project-client-solution) but also, for the latter, on the production and staging machines. That will allow us to manage versioning on the server too.

Q: Should several developers be working on the same development box? How to avoid conflicts?

A: I do not recommend this. It's always a good idea to have a local development sandbox. It's doable if the developers are working on different areas, but you'll get into conflicts that will be harder to isolate

Q: How to check in/check out units of work?

A: Once we have our local repository, we can jump to the most up to date with the command:

$ git pull

Q: How to check in my work?

Unlike svn, when you commit work it doesn't get pushed to the remote repository. You shuold commit early and often (don't even need internet for it) and then push to the central server. You do that with the following:

$ git commit -m 'message'
$ git push

Here's where a visual tool gets useful. Mac users have GitX, linux users have GitG, everyone has gitk and git update. So for the commit part I usual use the visual tool.

Q: Guidelines how to package a new version of a dashboard and migrate it from development to test, and from test to production

The development happens on the main branch, called the master branch. When we're ready to release a certain version, we create a new branch with the name of that version (obviously, feel free to choose what you want). In this example, I'll call it v1.

That branch can then be be checked out on the QA server.

dev$ git branch v1
dev$ git push

qa$ git pull
qa$ git checkout branch

Next step is testing it. If we find a bug that need fixing, we can fix it on that branch. If appropriated, we can merge the fix back on the development branch

qa$ git commit -m 'fixed bug on v1'
qa$ git push

dev$ git checkout master # be sure we change back to master
dev$ git merge v1 # pull the bug fix
dev$ git push # fix integrated

After we're happy with the solution and ready to go to production. This is where the tags come to good use. We can create a tag on it and push that information.

qa$ git tag v1.0
qa$ git push --tags

prod$ git pull
prod$ git checkout v1.0

That would put you on the correct version. Don't forget to update the solution repository.

Q: I was playing with the solution but I don't want to commit any change, just want to wipe the entire thing and get back to the clean state

$ git reset --hard HEAD

Q: I changed a single and I want to have it back / reverted to the last state

$ git checkout

Q: How to avoid overwriting each others work (which happens now if we're not careful)

No git command for this. Basically comes for free. However, if we're working on a bigger change, it's recommended that you create a new branch for it. That way you can work on that with guarantees that a specific feature can be developed independently.

Here's a schematic of how it conceptually works:

The idea is to isolate a specific feature. It's very easy to start:

$ git branch featureX

This will start an isolated development. You can do regular commits, pushes, etc. When finished, you can can merge back to the man branch. You do that by

1) switching to the main branch, usually called master:

$ git checkout master

2) Merge feature x back to the main master

$ git merge featureX

3) If there are any conflicts you'll need to resolve them and commit. After a push, feature X will then be available.

Conclusion

This doesn't aim to be a full tutorial about git. There are tons of great documentation, and it's really a powerful tool. But should provide some best approaches on how to best handle a pentaho implementation.

Any extra questions, just email / comment here and I'll add them

CGG - Putting CCC charts in Pentaho reporting / other tools

2012-09-13T14:32:00.000+01:00

This has been a long standing blog post. CGG has been around for ages now, it's even in Pentaho platform core, and only a few knew about it.

CGG stands for Community Graphics Generator. Although it even has it's own homepage, I never had the chance to blog about it. It's a somewhat hardcore plugin: it's basically able to execute on the server side custom scripts (java / javascript) that outputs images that can be used in external systems.

One of the most useful use cases is to be able to export CCC charts to images (png or svg) - either to allow a user the ability to download it, or to include it in Pentaho Reporting or any other of those tools

Here's an example of how it works. Imagine you develop for your users a great looking dashboard, almost as good looking as one of the UIs we develop :) :

(This was the dashboard we developed in a recent Ctools training course in Orlando, Pentaho headquarters)

As you can see, we went to some extent, exploring CCC capabilities, to fine tune the charts. Now we want to be able to use that line chart to PRD. Even though CGG does not have a UI and doesn't aim to be user friendly, CDE has a way to make the bridge to it, using some hidden features.

Going straight to the point. In CDE, press shift-G. That will open the CGG window (as a side reference, press shift-? to see other very useful keyboard shortcuts):

If you ever used this feature before, you'll notice this screen is a little different. We just added some more features, namely the ability to automatically get the url that generates that image from the outside - even I always struggled to find the right url. This is already available in the dev builds and will be released in the next stable version.

You'll be able to see that you can set some options there, namely the outputType, that currently can be either png or svg and you may need to change the server url if are developing in a sandbox and want to publish it to a server. I actually thought svg support was broken in PRD but just tested it and that seems to be fixed, so take advantage of that feature.

One other thing to note is that you must take care of authentication. Either you pass the extra arguments &userid=joe&password=password or you allow that url to be accessible with no password, or whatever.

If you save your dashboard and try that url, this is what you'll get:

This is what CGG is all about, and there's tons of engineering work underneath to allow this "simple step" to work with just one keystroke.

Now it's gets very obvious what to do. Open PRD, add an image component, and put that url (don't forget authentication)

You'll also want to check the blog post I did a while back about using CDA datasources in PRD. In the meanwhile, in recent versions of PRD (4.5 and above) you don't need to download the CDA datasources, all you need to do is enable the experimental features in Edit -> Preferences -> General -> Enable Experimental Features.

To render this report from the server, you'll need to copy to pentaho/WEB-INF/lib/ the file pentaho-reporting-engine-classic-extensions-cda-*.jar that you'll find in PRD library directory.

There's another feature of CGG. You can pass parameters to the query, by adding &paramParameterName=ParameterValue to the url. And that can be exposed from PRD too.

Just create a prompt the usual way. On my sample, the parameter is called a month, and since the query is already on the dashboard too, I just need to select it to build the prompt in the prpt.

However, in order to make the call with the new parameter, we can't use the image component anymore, and use the image-field instead.

For that, we need to create a formula that will build that url. Here's the sample formula I used:

="http://127.0.0.1:8080/pentaho/content/cgg/Draw?script=/SyncOrlando/Lab12/lineChart.js&outputType=png&userid=joe&password=password&parammonthParameter="&URLENCODE([month])

The URLENCODE function allows us to be sure that our parameters will reach the server properly. In the end, we should have a report looking like this:

Advanced topic: If you make reports with a lot of charts (you can even have cgg rendering a chart per line) you'll soon find out that your report starts to take a really long time to render. There's an explanation for it: PRD does a lot of passes to better determine the final layout. Since CGG doesn't support the HEAD request method, PRD won't get the appropriate info regarding cache, resulting in a bunch of requests for the image. Fortunately, Thomas Morgner allowed is a workaround to this issue, by changing a behavior in libloader. In your loader.properties file (located in WEB-INF/classes for the server or create it under prd/resources dir for the report designer) add the following lines:

# Controls the minimum time between HEAD requests regardless of
# the date -headers given by the response object.
org.pentaho.reporting.libraries.resourceloader.config.url.FixedCacheDelay=500000

# Fixes the date headers by simply using Date.now() as mod-date.
# This will break the HTTP specs and thus it is disabled by default.
org.pentaho.reporting.libraries.resourceloader.config.url.FixBrokenWebServiceDateHeader=true

This particular feature will be available in Pentaho Reporting 3.9.1 (or you'll have to compile your own)

And this is what it looks from the Pentaho BI server:

Cheers

-pedro