Replacing Grunt/Gulp with NPM

Grunt/Gulp are Overkill (for me)

I have come to the conclusion that specialized wrapper/runner systems such as Grunt and Gulp are unnecessary technical debt.  I had previously been using Gulp and all the pertinent Gulp add-ons that I needed and it was a source of complexity.

So I decided to port a codebase into a new data visualization project called dex.js.

Previously, in my node_modules directory I had 54 node packages spread across 31,239 files taking up 343M of disk space.  That’s a lot of overhead for a javascript visualization library!

$ ls -1 node_modules/|wc -l

$ find node_modules -name \* | wc -l

$ du -sh node_modules
343M node_modules

As I considered ripping all of this out with a simple Makefile replacement, I came across an insightful article  “Why we should stop using Grunt & Gulp” written by Keith Cirkel.  He follows up with a prescriptive article “How to us NPM as a build tool”.

Keith points out that adding layers like Grunt/Gulp contribute complexity, unnecessary dependencies and bloat to your project.

By using npm and a well configured package.json, you can reduce the complexity greatly.

So I decided to put it to the test…

Life without Gulp

Things are much simplier now.  I have a package.json that specifies how my application is laid out and defines a few scripts for creating build distributions, testing and creating documentation.


  "name": "dexjs",
  "description": "dexjs : dexvis javascript visualization component library.",
  "version": "0.1.0",
  "license": "Apache-2.0",
  "keywords": [
  "homepage": "",
  "email"   : "",
  "url"     : "",
  "files": [
  "main": "dex",
  "repository": {
    "type": "git",
    "url": ""
  "dependencies": {
    "jquery": "latest",
    "underscore": "latest"
  "devDependencies": {
    "jshint": "latest",
    "jasmine": "latest",
    "browserify": "latest"
  "scripts": {
    "lint": "jshint src/**.js",
    "docs": "jsdoc -c conf.json -t ./node_modules/ink-docstrap/template -R",
    "build": "browserify src/dex.js --standalone dex > dist/dex.js && browserify src/dex.js --standalone dex | uglifyjs > dist/dex.min.js && browserify src/dex.maps.js --standalone dex > dist/dex.maps.js && browserify src/dex.maps.js --standalone dex | uglifyjs > dist/dex.maps.min.js",
    "test": "jasmine SpecRunner.html"

I have my main 4 task defined as scripts. Within WebStorm I can execue individual scripts with a single button click.  It’s minimalistic, though script strings can get unwieldy.  Here are my new set of dependencies.   I now have 7 package dependencies spread across 6,750 files taking up a total of 45M of disk space.  That’s much better than my previous 54 node packages spread across 31,239 files taking up 343M of disk space.

$ ls -1 node_modules/|wc -l

$ find node_modules/ -name \* | wc -l

$ du -sh node_modules
45M node_modules

I also get rid of 6-7 gulpfiles and targets and my build system is a lot more understandable, much less complex with an order of magnitude fewer dependencies.

Where to go from here?

From here, I am going to focus on defining a package.json compliant enough to allow for distribution via npm in addition to the github repository.

Finally, it feels like my build system is working for me rather than creating work for me.

  • Pat
Posted in General | Leave a comment

Gapminder/Vizabi and the story of Burundi

Lately I’ve been droning on about the reusable player and the merits of being able to create frames of data which may be played out across a dimension (typically time).  I think the first time I really saw this in action was watching Hans Rosling present “200 Countries, 200 Years, 4 Minutes – The Joy of Stats”.  To me, it looked like science fiction; at first glance.

Being an engineer, I condensed this in my head as 200 series, consisting of 200 samples with an x and a y value.  Roughly 200 x 200 x 2 datapoints depending on sparseness of the sampling. Roughly 80,000 data values represented by 40,000 data points presented in manner understandable by virtually anyone.  40,000 points on a screen would look like chaos and would be difficult from which to derive anything insightful.  Removing a dimension via framing or partitioning; in this case the dimension of time, reduces the number of points on the screen at any given time to a much more manageable 200; albeit this is still a lot for the human brain to handle.  Series selection and filters come to the rescue.

One of the downsides of partitioning, or framing out a dimension is that you may only view one frame at a time.  Something that Gapminder solved by adding trails to selected series.  Literally providing the best of both worlds.  A tremendous reduction in visual clutter without loss of signal.  The ability to show multiple trails allows for direct comparison between two series.

So the other day, serendipity struck in the form of a tweet which informed me of a pre-alpha project called Vizabi, a Javascript project on Github which leverages D3js to provide those wonderfully insightful Gapminder charts. Aha!  The geniuses at Gapminder are sharing their secrets at long last!  I’m looking forward to peeking under the hood.  Anyone reading this should check it out.

Child Mortality & GDP per capita

This is a compelling view within Gapminder because it relates prosperity to quality of life.  There is no greater common instinct in human beings more powerful than protecting our children.  This is a great view on the overall health and direction of a nation.

Click the image to see for yourself.


So, at a glance of the snapshot of 2015 affairs, we see a great disparity between the light blue and the rest of the world.  Picking a nation, “Burundi”, I enable trail and look for major points of inflection.


In 1945, child mortality starts dropping at a more accelerated pace.  Cross referencing Wikipedia, significant events within this time frame were:

1940 : Classified as a United Nations Trust Territory under Belgian administrative authority.

1943: Powers were split in the legislative division of Burundi’s government between chiefdoms and lower chiefdoms. Chiefdoms were in charge of land, and lower sub-chiefdoms were established. Native authorities also had powers.


In 1960, we start seeing a decline in financial prosperity, while mortality rates fluctuate slightly.  Again, according to Wikipedia, significant events in this time frame were:

1959: On January 20, 1959, Burundi’s ruler Mwami Mwambutsa IV requested Burundi’s independence from Belgium and dissolution of the Ruanda-Urundi union.  In the following months, Burundian political parties began to advocate for the end of Belgian colonial rule and the separation of Rwanda and Burundi.

Don’t read this wrong, these are just the facts Jack!  Perhaps not enough of the facts at that.  I am an advocate of independence.  However, it seems that there was a financial price for Burundi’s independence which took a decade to recover from financially.  Family income declines from $632.00 to $531.00 with a gradual recovery over a 10 year period of time.


In 1974, the trend of increased financial prosperity became a decline until 1991 when things again started to turn around.  Wikipedia, what happened?

1992: The government promulgated a new constitution that provided for a multi-party system.  Civil war broke out.

1993: A first attempt at democracy. June 1993, Melchior Ndadaye won the first democratic election and became the first Hutu head of state, leading a pro-Hutu government before being assassinated in October of 1993.  300,000 people were killed in the following few years, but it looks like very few of them were 5 and under.

With so many unregistered deaths, obviously, a single view is not a complete view.


Finally, 2005 brings what looks to be a decade of decreasing Child mortality rates with an increase in prosperity (though very much poverty level on a global scale).

Trans-formative events around this time frame appear to be:

2000: On 28 August 2000, a transitional government for Burundi was planned as a part of the Arusha Peace and Reconciliation Agreement. The transitional government was placed on a trial basis for five years. After several aborted cease-fires, a 2001 peace plan and power-sharing agreement has been relatively successful

2003: A cease-fire was signed in 2003 between the Tutsi-controlled Burundian government and the largest Hutu rebel group, CNDD-FDD (National Council for the Defense of Democracy-Forces for the Defense of Democracy).  Domitien Ndayizeye (Hutu) is elected president.

2005: Ethnic quotas are formed for determining positions in Burundi’s government.  Pierre Nkurunziza (Hutu), once a leader of a rebel group, elected president.

2006: UN reconstruction efforts seem to take effect.

The Tutsi/Hutu Conflict

I never expected to learn so much about Burundi when I started this post.  I didn’t realize that it was another country riddled with Tutsi/Hutu conflict.  What’s all that about anyway?  So I google:

500-1000 BC : The Hutus, an agricultural people, settle in the Great Lakes region of Central Africa.

1600 AD: The Tutsis, also known as Watutsis, a nomadic people, begin arriving from Ethiopia.  They settle peacefully among the Hutus – adopting their language, beliefs and customs.

Gradually…economic differences form…  Tutsis as cattle herders often being in a position of economic dominant position over the soil-tilling Hutus.  As the Tutsis gain more dominance, the minority begins to rule the majority.  Resentments form…

Over time, the Tutsi and Hutus has evolved into more of an economic difference rather than an ethnic one as explained by Charlayne Hunter Gault, Professor Izangola:

“In Rwanda, the Tutsi and the Hutu are the same people. They are all people–large grouping or communities which go from seven regions of Cameroon to Uganda–all the way to South Africa, in the same culture,” Izangola said. “People used to be Tutsi or Hutu, depending on the proximity to the king. If you were close to the king, you owned wealth, you owned a lot of cattle, you are a Tutsi. If you are far away from the king, you are a cultivator, you don’t own much cattle, you are a Hutu.”

Colonial rule, which began in the late 19th Century, did little to bring the groups together. The Belgians, who ruled what would later become Rwanda and Burundi, forced Hutus and Tutsis to carry ethnic identity cards. The colonial administrators further exacerbated divisions by only allowed Tutsis to attain higher education and hold positions of power.

All I can say is wow…till next time…

  • Pat


Posted in General | Leave a comment

Pushing the Limits of HTML

Previously, I posted a visual of a single time-series against a global heatmap based upon the World Development Indicator dataset from the World Data Bank.

So I wondered, could I display all the series from this dataset?  This is no small task as it consists of over 18 million datapoints spread across 1,345 series for 248 different countries and country aggregations.

Having had my site hacked a number of times, rather than worry about exposing a restful service to the internet, I decided to use an in-memory database.  Sure the initial load will be slow, however, exploration itself would be fast.

Initially, it took me over a day to load the datapoints into sqlite.  However, I didn’t get it right the first time.  As Larry Wall once said, Impatience is an virtue of the developer.  Impatient, I tried to figure out why my insertion rate into sqlite was only 211 entries per second.  Turning autocommit off and committing every 1000 insertions increased my performance by roughly 4000 times.  Once that change was made, my 24 hour load times became 3 minutes.

The schema I settled on looks like:


On my local machine, it takes around 8 seconds to initialize the application.  From there, I can explore all 1,345 development statistics across time in a Gapminder like setting.

For a limited time, I will keep this link active so that others may also give it a try:

However, due to the fact that the sqlite database is over 500 megs (mostly due to indexing), I will likely take this offline in the near future.

-rwxrwxr-x+ 1 Patrick Patrick 423095296 Mar 6 19:49 wdi.db

Do not click this link from a mobile device!  It will surely fail and expect it to take awhile to load! While the banner: “Please wait, the data is loading…” is displayed; THE DATA REALLY IS STILL LOADING.


Posted in General | Leave a comment

Internet Adoption : The Movie


In my last blog article I discussed the reusable Player component.  So I’ve decided to put it to work on a real WDI dataset consisting of 5,166 rows.  This example plays a heatmap of % users on the internet mapped across country.

Images link to live examples.

Tying the chloropleth to the player was quite easy.  In this case, I didn’t even bother to wrap the component in the Dex component framework, though I may do so in the future for consistency.

Next, I threw the same dataset at the C3 based player discussed in the earlier blog.  I also expanded the player to support categorical x-axis.  By framing the data in different ways, we can create very different views of the data and gain deeper insight.

In this example, I frame the data by country.   The player plays the internet adoption across each country from 1989-2014.  Here, we can see Afghanistan’s slow but increased adoption of the internet.


By simply reframing the datset by year, we can see all countries side by side.  The player cycles through time.  There are many labels, so the zoom and pan options of C3 come in handy here.  Same data, but quite a different view.


Anyway, that’s it for now, but I expect that there will be much more to come.

  • Pat
Posted in General | Leave a comment

A Reusable Javascript Player Control

Anyone who has read deeply into my blog knows that I am a huge fan of motion charts.  However, adding that extra interactive dimension tends to add more than it’s share of complexity.

So, with inspiration derived from multiple sources such as Hans Rosling, Mike Bostock, James Curley, Alex de Muller and lastly Steve Wittens.

My idea was, why is a motion chart such as this a special case?


I asked myself, isn’t it really two independent components interacting with one another?  In this case, a time slider driving a simple bubble chart.  Furthermore, why can’t we implement them independently and in a reusable manner so that we can simplify things?

So the thoughts mulled around in my head, almost dying until I saw Alex de Muller’s three dimensional animation:



The controls are simple and serve as a dimensional filter, in this case a time dimension. As you step through the graph, it changes over time.  However, I would also like to support numeric and categorical dimension in addition to time.

The next thing I would like to talk about is a term Mike Bostock introduced me to called Object Constancy.


Object constancy is the practice of allowing the eye to follow the transitions of visual state changes.  It’s the realm of tweens, transitions and easing.  If an element enters or exits the stage, give it some flourish (a dramatic entrance and exit if you will) so that the viewer can better wrap their mind around the changes from frame to frame.  James Curley has been a prolithic producer of interesting visuals; genereously sharing them on  James tends to pay great attention to transitions as this NHL example shows.


Finally, Steve Wittens of Mathbox fame, state:

“I’ve found that, when in doubt, adding an extra dimension always helps. The moment I finally realized that every implicit graph in N dimensions is really just a slice of an explicit one in N+1 dimensions, a ridiculous amount of things clicked together.”

So true…and lead me to think of a design where there might be multiple dimensional filters interacting in tandem.

The Player Component

So without further adieu, let me introduce the Player component.  In the coming weeks or months depending, I will github everything, sooner if there is avid interest.

The old deprecated project of mine DexCharts has evolved into an more general purpose wrapping framework for providing a consistent interface around different browser-based technologies.  It’s called dex.js.  So I implemented the Player in dex.js under the dex.ui module.

Here’s an example of the Player component interacting with C3 components also wrapped in the dex.js framework.


In this example, I kept the dataset to 4000 points in order to respect those viewing the visuals on mobile devices.  However, it scales much higher on more powerful devices.  This one has 11,000 datapoints.


The controls are simple and should be intuitive as they follow a media-player metaphor.

You can move to the first or last frame, step through the frames at a specified delay, or play the frames on a continuous loop.

All links are live so click the images for live versions.

C3 was an ideal choice due to Masayuki Tanaka’s strict adherence to the principles of re-usability and object constancy.

Within dex.js, this chart is pretty simple to describe.  Essentially, the core of this program is as follows:

var player = new dex.ui.jqueryui.Player({
  'parent' : '#toolbar',
  'csv' : csv,
  'delay' : 500

var linechart = new dex.charts.c3.LineChart({
  'parent'      : '#ChartArea',
  'resizable'  : true,
  'color'       : d3.scale.category20c(),
  'linktype' : "line",
  'csv' : player.attr("frames").frames[0]

linechart.subscribe(player, "new-frame", function(msg) {


We create a player with a 500ms delay based on a simple csv dataset.

The player separates the csv into frames based on column zero of the csv, though this is configurable.

Next, we create a linechart which is initialized to the contents of the first frame.

Then we add a pub-sub event where the linechart listens for “new-frame” events from the player.  When received, it will update the csv data to the current frame.  If the dexjs wrapped visual.

Lastly, we render both and the fun begins.

Wrapping Up

I have much more to say on this topic and more demos to come.  I want to demonstrate some multi-dimensional players across multiple dimensions, and want to bring motion to more than just simple area/line/scatter/bubble charts.

For example, here is a motion-chord chart.  It’s pretty much the same code as the first examples only using a chord instead of a C3 chart.

It’s not quite as polished due to my Chord diagram’s lack of object constancy, but you get the idea of where I am going.


That’s it for now…

  • Pat
Posted in General | Leave a comment

Trumps Iowa Speech

Here is a link to Trump’s Iowa speech visualized.  Size is proportional to word frequency and adjacent words are connected to one another.

It uses vivagraph with custom svg text nodes.


Posted in General | Leave a comment

Graphing in 3D with Dex and Vis.js

Here is a video demonstating graphing in 3D and time via Dex and Vis.js.


This capability is available in Dex which is also available here.

Posted in General | 1 Comment