Running JavaFX in a Docker container

There’s lots of talk nowdays about the cloud, containerization, so I decided to see what it would take to run a JavaFX application within a docker container.

It took a bit of work to figure out, but it works great.

Due to some Oracle licensing requirements the version of Oracle Java 8 is not bundled like normal linux packages, but rather has to be downloaded and installed from a special repository called webupd8.  This small amount of additional complexity is also reflected in the Dockerfile we will cover below.

Why Docker?

Why bother bundling an application with Docker?  Well, there are many reasons.

  1. Reproducible installations — Since docker ships a machine with the application, if it runs on the source application container, it’s almost guaranteed to run on the target environment.
  2. Easy installations.
  3. Doesn’t touch your machine directly.
  4. It seems to be the emerging defacto cloud containerization standard.

Without further adieu, here’s my Dockerfile, let’s walk through it.

###
#
# Dex Dockerfile installation:
#
# build:
#
# docker build -t dex .
#
# run:
#
# docker run -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=unix$DISPLAY dex
#
###

# pull base image
FROM debian:jessie

RUN \
    echo "===> add webupd8 repository..."  && \
    echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" | tee /etc/apt/sources.list.d/webupd8team-java.list  && \
    echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" | tee -a /etc/apt/sources.list.d/webupd8team-java.list  && \
    echo "deb http://ftp.de.debian.org/debian jessie main" >> /etc/apt/sources.list && \
    apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886 && \
    apt-get update && \
    apt-get install -y gtk2-engines libxtst6 libxxf86vm1 freeglut3 libxslt1.1 && \
    apt-get update  && \
    \
    echo "===> install Java"  && \
    echo debconf shared/accepted-oracle-license-v1-1 select true | debconf-set-selections  && \
    echo debconf shared/accepted-oracle-license-v1-1 seen true | debconf-set-selections  && \
    DEBIAN_FRONTEND=noninteractive  apt-get install -y --force-yes oracle-java8-installer oracle-java8-set-default  && \
    \
        apt-get install -y git && \
        cd ~ && git clone https://github.com/PatMartin/Dex.git && \
    \
    echo "===> clean up..."  && \
    rm -rf /var/cache/oracle-jdk8-installer && \
    apt-get clean  && \
    rm -rf /var/lib/apt/lists/*

# cd to the Dex directory and execute the jar.
CMD cd ~/Dex && java -jar Dex.jar

Step 1:

In line 16 we see:

FROM debian:jessie

This simply tells docker to download, install and use a base debian installation using the jesse release as our base imate.

Step 2:

Next on lines 18 through 39 we see a pretty intimidating command:

RUN \
 echo "===> add webupd8 repository..." && \
 echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" | tee /etc/apt/sources.list.d/webupd8team-java.list && \
 echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" | tee -a /etc/apt/sources.list.d/webupd8team-java.list && \
 echo "deb http://ftp.de.debian.org/debian jessie main" >> /etc/apt/sources.list && \
 apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886 && \
 apt-get update && \
 apt-get install -y gtk2-engines libxtst6 libxxf86vm1 freeglut3 libxslt1.1 && \
 apt-get update && \
 \
 echo "===> install Java" && \
 echo debconf shared/accepted-oracle-license-v1-1 select true | debconf-set-selections && \
 echo debconf shared/accepted-oracle-license-v1-1 seen true | debconf-set-selections && \
 DEBIAN_FRONTEND=noninteractive apt-get install -y --force-yes oracle-java8-installer oracle-java8-set-default && \
 \
 apt-get install -y git && \
 cd ~ && git clone https://github.com/PatMartin/Dex.git && \
 \
 echo "===> clean up..." && \
 rm -rf /var/cache/oracle-jdk8-installer && \
 apt-get clean && \
 rm -rf /var/lib/apt/lists/*

It would be nice to break it up some, but there’s a good reason that we don’t.  Multiple RUNs within a Dockerfile will cause the creation of multiple layers; one per RUN.  We’re trying to minimize this, and the objective causes some ugliness.  It’s necessary though.

Basically, this command is performing the following task:

  1. Lines 19-23 are setting up some repositories.
  2. Line 24 updates the packages.
  3. Line 25 installs a few necessary dependencies required for JavaFX libprism dependencies.
  4. Lines 29-30 automate the acceptance of conditions and lines 31 performs a headless installation of Java8.
  5. Line 33-34 installs git then uses it to download the Dex repository, code and all to the current directory.
  6. Lines 36-39 perform some cleanup.
  7. Line 42 specifies the command we will run by default when executing this application container.

Building from Dockerfile

Assuming docker is installed and running, drop the Dockerfile described above into an empty directory somewhere.

Issuing the command:

docker build -t dex .

Will create a dex image.  It’ll take a few minutes to run.  Remember, it’s creating a vm image, updating it to spec, installing oracle and dex.  That’s a lot!

Next, we can run it from an X11 capable linux client like this:

xhost +
docker run -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=unix$DISPLAY dex

It should be possible to run like this from Windows or OSX, but I haven’t tried it personally.

Anyway, that’s it for now…Good luck!

  • Pat
Posted in General | 5 Comments

Dex code on github!

Finally, we are truly open source software.  The source code for Dex is now on github.  The repository can be found here.

Also, the living documentation is in gitbook form and can be found here.

Posted in Dex | 2 Comments

Replacing Grunt/Gulp with NPM

Grunt/Gulp are Overkill (for me)

I have come to the conclusion that specialized wrapper/runner systems such as Grunt and Gulp are unnecessary technical debt.  I had previously been using Gulp and all the pertinent Gulp add-ons that I needed and it was a source of complexity.

So I decided to port a codebase into a new data visualization project called dex.js.

Previously, in my node_modules directory I had 54 node packages spread across 31,239 files taking up 343M of disk space.  That’s a lot of overhead for a javascript visualization library!

$ ls -1 node_modules/|wc -l
54

$ find node_modules -name \* | wc -l
31239

$ du -sh node_modules
343M node_modules

As I considered ripping all of this out with a simple Makefile replacement, I came across an insightful article  “Why we should stop using Grunt & Gulp” written by Keith Cirkel.  He follows up with a prescriptive article “How to us NPM as a build tool”.

Keith points out that adding layers like Grunt/Gulp contribute complexity, unnecessary dependencies and bloat to your project.

By using npm and a well configured package.json, you can reduce the complexity greatly.

So I decided to put it to the test…

Life without Gulp

Things are much simplier now.  I have a package.json that specifies how my application is laid out and defines a few scripts for creating build distributions, testing and creating documentation.

package.json:

{
  "name": "dexjs",
  "description": "dexjs : dexvis javascript visualization component library.",
  "version": "0.1.0",
  "license": "Apache-2.0",
  "keywords": [
    "chart",
    "charting",
    "dataviz",
    "visualization",
    "c3",
    "d3"
  ],
  "homepage": "http://dexvis.com",
  "email"   : "patrick145@msn.com",
  "url"     : "https://dexvis.wordpress.com",
  "files": [
    "src/dex.js"
  ],
  "main": "dex",
  "repository": {
    "type": "git",
    "url": "https://www.github.com/PatMartin/dexjs.git"
  },
  "dependencies": {
    "jquery": "latest",
    "underscore": "latest"
  },
  "devDependencies": {
    "jshint": "latest",
    "jasmine": "latest",
    "browserify": "latest"
  },
  "scripts": {
    "lint": "jshint src/**.js",
    "docs": "jsdoc -c conf.json -t ./node_modules/ink-docstrap/template -R README.md",
    "build": "browserify src/dex.js --standalone dex > dist/dex.js && browserify src/dex.js --standalone dex | uglifyjs > dist/dex.min.js && browserify src/dex.maps.js --standalone dex > dist/dex.maps.js && browserify src/dex.maps.js --standalone dex | uglifyjs > dist/dex.maps.min.js",
    "test": "jasmine SpecRunner.html"
  }
}

I have my main 4 task defined as scripts. Within WebStorm I can execue individual scripts with a single button click.  It’s minimalistic, though script strings can get unwieldy.  Here are my new set of dependencies.   I now have 7 package dependencies spread across 6,750 files taking up a total of 45M of disk space.  That’s much better than my previous 54 node packages spread across 31,239 files taking up 343M of disk space.

$ ls -1 node_modules/|wc -l
7

$ find node_modules/ -name \* | wc -l
6760

$ du -sh node_modules
45M node_modules

I also get rid of 6-7 gulpfiles and targets and my build system is a lot more understandable, much less complex with an order of magnitude fewer dependencies.

Where to go from here?

From here, I am going to focus on defining a package.json compliant enough to allow for distribution via npm in addition to the github repository.

Finally, it feels like my build system is working for me rather than creating work for me.

  • Pat
Posted in General | Leave a comment

Gapminder/Vizabi and the story of Burundi

Lately I’ve been droning on about the reusable player and the merits of being able to create frames of data which may be played out across a dimension (typically time).  I think the first time I really saw this in action was watching Hans Rosling present “200 Countries, 200 Years, 4 Minutes – The Joy of Stats”.  To me, it looked like science fiction; at first glance.

Being an engineer, I condensed this in my head as 200 series, consisting of 200 samples with an x and a y value.  Roughly 200 x 200 x 2 datapoints depending on sparseness of the sampling. Roughly 80,000 data values represented by 40,000 data points presented in manner understandable by virtually anyone.  40,000 points on a screen would look like chaos and would be difficult from which to derive anything insightful.  Removing a dimension via framing or partitioning; in this case the dimension of time, reduces the number of points on the screen at any given time to a much more manageable 200; albeit this is still a lot for the human brain to handle.  Series selection and filters come to the rescue.

One of the downsides of partitioning, or framing out a dimension is that you may only view one frame at a time.  Something that Gapminder solved by adding trails to selected series.  Literally providing the best of both worlds.  A tremendous reduction in visual clutter without loss of signal.  The ability to show multiple trails allows for direct comparison between two series.

So the other day, serendipity struck in the form of a tweet which informed me of a pre-alpha project called Vizabi, a Javascript project on Github which leverages D3js to provide those wonderfully insightful Gapminder charts. Aha!  The geniuses at Gapminder are sharing their secrets at long last!  I’m looking forward to peeking under the hood.  Anyone reading this should check it out.

Child Mortality & GDP per capita

This is a compelling view within Gapminder because it relates prosperity to quality of life.  There is no greater common instinct in human beings more powerful than protecting our children.  This is a great view on the overall health and direction of a nation.

Click the image to see for yourself.

ChildMortalityvsGDPpercapita

So, at a glance of the snapshot of 2015 affairs, we see a great disparity between the light blue and the rest of the world.  Picking a nation, “Burundi”, I enable trail and look for major points of inflection.

Gapminder_Burundi_1945

In 1945, child mortality starts dropping at a more accelerated pace.  Cross referencing Wikipedia, significant events within this time frame were:

1940 : Classified as a United Nations Trust Territory under Belgian administrative authority.

1943: Powers were split in the legislative division of Burundi’s government between chiefdoms and lower chiefdoms. Chiefdoms were in charge of land, and lower sub-chiefdoms were established. Native authorities also had powers.

Gapminder_Burundi_1960

In 1960, we start seeing a decline in financial prosperity, while mortality rates fluctuate slightly.  Again, according to Wikipedia, significant events in this time frame were:

1959: On January 20, 1959, Burundi’s ruler Mwami Mwambutsa IV requested Burundi’s independence from Belgium and dissolution of the Ruanda-Urundi union.  In the following months, Burundian political parties began to advocate for the end of Belgian colonial rule and the separation of Rwanda and Burundi.

Don’t read this wrong, these are just the facts Jack!  Perhaps not enough of the facts at that.  I am an advocate of independence.  However, it seems that there was a financial price for Burundi’s independence which took a decade to recover from financially.  Family income declines from $632.00 to $531.00 with a gradual recovery over a 10 year period of time.

Gapminder_Burundi_1991

In 1974, the trend of increased financial prosperity became a decline until 1991 when things again started to turn around.  Wikipedia, what happened?

1992: The government promulgated a new constitution that provided for a multi-party system.  Civil war broke out.

1993: A first attempt at democracy. June 1993, Melchior Ndadaye won the first democratic election and became the first Hutu head of state, leading a pro-Hutu government before being assassinated in October of 1993.  300,000 people were killed in the following few years, but it looks like very few of them were 5 and under.

With so many unregistered deaths, obviously, a single view is not a complete view.

Gapminder_Burundi_2005

Finally, 2005 brings what looks to be a decade of decreasing Child mortality rates with an increase in prosperity (though very much poverty level on a global scale).

Trans-formative events around this time frame appear to be:

2000: On 28 August 2000, a transitional government for Burundi was planned as a part of the Arusha Peace and Reconciliation Agreement. The transitional government was placed on a trial basis for five years. After several aborted cease-fires, a 2001 peace plan and power-sharing agreement has been relatively successful

2003: A cease-fire was signed in 2003 between the Tutsi-controlled Burundian government and the largest Hutu rebel group, CNDD-FDD (National Council for the Defense of Democracy-Forces for the Defense of Democracy).  Domitien Ndayizeye (Hutu) is elected president.

2005: Ethnic quotas are formed for determining positions in Burundi’s government.  Pierre Nkurunziza (Hutu), once a leader of a rebel group, elected president.

2006: UN reconstruction efforts seem to take effect.

The Tutsi/Hutu Conflict

I never expected to learn so much about Burundi when I started this post.  I didn’t realize that it was another country riddled with Tutsi/Hutu conflict.  What’s all that about anyway?  So I google:

500-1000 BC : The Hutus, an agricultural people, settle in the Great Lakes region of Central Africa.

1600 AD: The Tutsis, also known as Watutsis, a nomadic people, begin arriving from Ethiopia.  They settle peacefully among the Hutus – adopting their language, beliefs and customs.

Gradually…economic differences form…  Tutsis as cattle herders often being in a position of economic dominant position over the soil-tilling Hutus.  As the Tutsis gain more dominance, the minority begins to rule the majority.  Resentments form…

Over time, the Tutsi and Hutus has evolved into more of an economic difference rather than an ethnic one as explained by Charlayne Hunter Gault, Professor Izangola:

“In Rwanda, the Tutsi and the Hutu are the same people. They are all people–large grouping or communities which go from seven regions of Cameroon to Uganda–all the way to South Africa, in the same culture,” Izangola said. “People used to be Tutsi or Hutu, depending on the proximity to the king. If you were close to the king, you owned wealth, you owned a lot of cattle, you are a Tutsi. If you are far away from the king, you are a cultivator, you don’t own much cattle, you are a Hutu.”

Colonial rule, which began in the late 19th Century, did little to bring the groups together. The Belgians, who ruled what would later become Rwanda and Burundi, forced Hutus and Tutsis to carry ethnic identity cards. The colonial administrators further exacerbated divisions by only allowed Tutsis to attain higher education and hold positions of power.

All I can say is wow…till next time…

  • Pat

 

Posted in General | Leave a comment

Pushing the Limits of HTML

Previously, I posted a visual of a single time-series against a global heatmap based upon the World Development Indicator dataset from the World Data Bank.

So I wondered, could I display all the series from this dataset?  This is no small task as it consists of over 18 million datapoints spread across 1,345 series for 248 different countries and country aggregations.

Having had my site hacked a number of times, rather than worry about exposing a restful service to the internet, I decided to use an in-memory database.  Sure the initial load will be slow, however, exploration itself would be fast.

Initially, it took me over a day to load the datapoints into sqlite.  However, I didn’t get it right the first time.  As Larry Wall once said, Impatience is an virtue of the developer.  Impatient, I tried to figure out why my insertion rate into sqlite was only 211 entries per second.  Turning autocommit off and committing every 1000 insertions increased my performance by roughly 4000 times.  Once that change was made, my 24 hour load times became 3 minutes.

The schema I settled on looks like:

WDI_Schema

On my local machine, it takes around 8 seconds to initialize the application.  From there, I can explore all 1,345 development statistics across time in a Gapminder like setting.

For a limited time, I will keep this link active so that others may also give it a try:

However, due to the fact that the sqlite database is over 500 megs (mostly due to indexing), I will likely take this offline in the near future.

-rwxrwxr-x+ 1 Patrick Patrick 423095296 Mar 6 19:49 wdi.db

Do not click this link from a mobile device!  It will surely fail and expect it to take awhile to load! While the banner: “Please wait, the data is loading…” is displayed; THE DATA REALLY IS STILL LOADING.

WDI_Gapminder

 

Posted in General | Leave a comment

Internet Adoption : The Movie

 

Player_WDI_InetUsePlotly

 

In my last blog article I discussed the reusable Player component.  So I’ve decided to put it to work on a real WDI dataset consisting of 5,166 rows.  This example plays a heatmap of % users on the internet mapped across country.

Images link to live examples.

Tying the plot.ly chloropleth to the player was quite easy.  In this case, I didn’t even bother to wrap the component in the Dex component framework, though I may do so in the future for consistency.

Next, I threw the same dataset at the C3 based player discussed in the earlier blog.  I also expanded the player to support categorical x-axis.  By framing the data in different ways, we can create very different views of the data and gain deeper insight.

In this example, I frame the data by country.   The player plays the internet adoption across each country from 1989-2014.  Here, we can see Afghanistan’s slow but increased adoption of the internet.

Player_WDI_InetUseFramedByCountry

By simply reframing the datset by year, we can see all countries side by side.  The player cycles through time.  There are many labels, so the zoom and pan options of C3 come in handy here.  Same data, but quite a different view.

Player_WDI_InetUseFramedByYear

Anyway, that’s it for now, but I expect that there will be much more to come.

  • Pat
Posted in General | Leave a comment

A Reusable Javascript Player Control

Anyone who has read deeply into my blog knows that I am a huge fan of motion charts.  However, adding that extra interactive dimension tends to add more than it’s share of complexity.

So, with inspiration derived from multiple sources such as Hans Rosling, Mike Bostock, James Curley, Alex de Muller and lastly Steve Wittens.

My idea was, why is a motion chart such as this a special case?

MotionChart

I asked myself, isn’t it really two independent components interacting with one another?  In this case, a time slider driving a simple bubble chart.  Furthermore, why can’t we implement them independently and in a reusable manner so that we can simplify things?

So the thoughts mulled around in my head, almost dying until I saw Alex de Muller’s three dimensional animation:

 

VisJSGraph3D

The controls are simple and serve as a dimensional filter, in this case a time dimension. As you step through the graph, it changes over time.  However, I would also like to support numeric and categorical dimension in addition to time.

The next thing I would like to talk about is a term Mike Bostock introduced me to called Object Constancy.

ObjectConstancy

Object constancy is the practice of allowing the eye to follow the transitions of visual state changes.  It’s the realm of tweens, transitions and easing.  If an element enters or exits the stage, give it some flourish (a dramatic entrance and exit if you will) so that the viewer can better wrap their mind around the changes from frame to frame.  James Curley has been a prolithic producer of interesting visuals; genereously sharing them on bl.ocks.org.  James tends to pay great attention to transitions as this NHL example shows.

NHLTransitions

Finally, Steve Wittens of Mathbox fame, state:

“I’ve found that, when in doubt, adding an extra dimension always helps. The moment I finally realized that every implicit graph in N dimensions is really just a slice of an explicit one in N+1 dimensions, a ridiculous amount of things clicked together.”

So true…and lead me to think of a design where there might be multiple dimensional filters interacting in tandem.

The Player Component

So without further adieu, let me introduce the Player component.  In the coming weeks or months depending, I will github everything, sooner if there is avid interest.

The old deprecated project of mine DexCharts has evolved into an more general purpose wrapping framework for providing a consistent interface around different browser-based technologies.  It’s called dex.js.  So I implemented the Player in dex.js under the dex.ui module.

Here’s an example of the Player component interacting with C3 components also wrapped in the dex.js framework.

Player1Demo

In this example, I kept the dataset to 4000 points in order to respect those viewing the visuals on mobile devices.  However, it scales much higher on more powerful devices.  This one has 11,000 datapoints.

Player2Demo

The controls are simple and should be intuitive as they follow a media-player metaphor.

You can move to the first or last frame, step through the frames at a specified delay, or play the frames on a continuous loop.

All links are live so click the images for live versions.

C3 was an ideal choice due to Masayuki Tanaka’s strict adherence to the principles of re-usability and object constancy.

Within dex.js, this chart is pretty simple to describe.  Essentially, the core of this program is as follows:

var player = new dex.ui.jqueryui.Player({
  'parent' : '#toolbar',
  'csv' : csv,
  'delay' : 500
});

var linechart = new dex.charts.c3.LineChart({
  'parent'      : '#ChartArea',
  'resizable'  : true,
  'color'       : d3.scale.category20c(),
  'linktype' : "line",
  'csv' : player.attr("frames").frames[0]
});

linechart.subscribe(player, "new-frame", function(msg) {
  linechart.attr('csv', msg.data).update();
});

player.render();
linechart.render();

We create a player with a 500ms delay based on a simple csv dataset.

The player separates the csv into frames based on column zero of the csv, though this is configurable.

Next, we create a linechart which is initialized to the contents of the first frame.

Then we add a pub-sub event where the linechart listens for “new-frame” events from the player.  When received, it will update the csv data to the current frame.  If the dexjs wrapped visual.

Lastly, we render both and the fun begins.

Wrapping Up

I have much more to say on this topic and more demos to come.  I want to demonstrate some multi-dimensional players across multiple dimensions, and want to bring motion to more than just simple area/line/scatter/bubble charts.

For example, here is a motion-chord chart.  It’s pretty much the same code as the first examples only using a chord instead of a C3 chart.

It’s not quite as polished due to my Chord diagram’s lack of object constancy, but you get the idea of where I am going.

PlayerChordDemo

That’s it for now…

  • Pat
Posted in General | Leave a comment