Just saw this picture of the Wellington RFID farecard system at Adam Greenfield’s Flickr stream (CC-by-nc-sa photograph):

It’s called a Snapper card.

Compare this to the London based Oyster card:

And the Hong Kong Octopus card:

This international sea food theme makes me think that we have definitely missed a branding opportunity here (and this for a country of fishermen). Our entire OV-chipkaart system has been grossly underdesigned on all fronts, so no surprises there.

So I’ve got two proposed alternative names for our low countries farecard system:

• Herring card (Haringkaart)

or

• Mussel card (Mosselkaart)

How do we get this change implemented? And anybody care to mockup a concept?

Goal

I want to create a simple visualization tool for Processing so I can input a set of values for each Dutch municipality and then color a map based on those values.

This is harder than it seems because there is no convenient source for the cartographic data for the boundaries of the Dutch municipalities. So the first step is to acquire those boundaries.

OSM

This blogpost in Dutch put me on track for this dataset. OpenStreetMap has a pretty complete picture of the Netherlands and they track municipal boundaries under boundary admin_level=8.

The data dump contains all administrative boundaries on several levels and is something of a mess. The informationfreeway link on the blogpost which should generate an OSM file with only the relations with admin_level=8 but that specific API seems to be down. Also the approach of importing the OSM file back into a PostGIS database before rendering anything struck me as somewhat too cumbersome. So an alternative approach was called for.

There are some readily available dumps at CloudMade both with OSM files (format) and Shapefiles for these administrative boundaries. That seemed to be a useful starting point.

An OSM file is just an XML file with series of nodes, ways and relations in it. It is filterable by the generic processing tool Osmosis with the options that I guessed --way-key-value to filter ways with the admin_level tag and --node-key-value to do the same for nodes. Osmosis does not specify anything for relations which I want to preserve to be able to identify each boundary by the name of the municipality.

Having done that, the resulting OSM file needed to be drawn to the screen. I made a simple XML reader in Processing to display the resulting boundaries. The result did not seem to be completely what I wanted both with missing and unlinked geographical features and I think not everything properly labelled. For this particular application a wiki-map does not seem like the most suitable source of data.

CBS

CBS (the Dutch statistics office) also provides a dataset with administrative boundaries of the Netherlands. It is a bit hard to track down on the site and there’s a reference to the Kadaster which isn’t entirely clear, but the generalized Shapefile is workable.

One problem is that Shapefiles are only understood properly by GIS people and there are hardly any libraries for web developers to work with the data format. Sunlight Labs recently released their ClearMaps library to aid developers wanting to work with Shapefiles, which is a big step in the right direction.

Another problem is that the file on the CBS site is from 2006 and that several municipalities have merged/split, so that adds some problems for correlating it to data. And come to think of it, this municipal rejiggering makes any historical data view of the Netherlands a daunting task. Somebody on Wikipedia has generated a similar map of the Netherlands for 2010 supposedly using data from the CBS, but I can’t find that dataset on the site.

Oddly enough there’s also nothing readily available to draw Shapefiles in Processing. Perusing the forum yields this post which points to Geotools which is a massive set of Java libraries consisting mostly of a huge dependency nightmare mitigated somewhat by Eclipse and Maven.

Geographical data

Viewing the Shapefile in qGIS shows that it does indeed contain the municipal boundaries with correct labeling. Having verified that, we need to extract the geographical data from the file into a format for easier reuse. Linking all the Geotools dependencies to my Processing sketch does not seem like an attractive proposition. Using the Geotools quickstart to setup Eclipse to pull in the libraries and run the Java code, did work pretty conveniently.

Poking around the Shapefile with Java and using the very poor javadocs (the User Guide’s usefulness turned out to be extremely limited) and sources posted online that are available of Geotools yielded something worthwhile after a full day’s work. I also found lots of forum posts of very confused people with few replies and little insight to be gleaned from them. This really seems to be an underdeveloped field.

It turns out the Shapefile read with Geotools contains SimpleFeature classes (UML for those, and the Wikipedia lemma for the OpenGIS standard) of which you can call the getDefaultGeometry() methods.

Geotools also provides a default Drawer.java which you can use to display the features (via LiteShapes) in the Shapefile using Java AWT Graphics. This turned out to be useful mainly for debugging purposes and to verify that Geotools does indeed properly read in the Shapefile. Using a GeomCollectionIterator to walk through the points and extract the coordinates that way turned out to be a dead end (especially because I didn’t get the role of the various Transforms).

Another idea was to generate SVG from the Shapefile but the GenerateSVG class did not seem to be included in my library checkout and fiddling with the maven file seemed risky.

Finally the following piece of code yielded for me the two pieces of data I was looking for, the names of the municipalities and the content of the SimpleFeatures as MULTIPOLYGONs.

  String gemShapefile = "/Users/alper/Documents/projects/muniboundaries/cbs/buurt_2008_gen2/gem_2008_gn2.shp";
  File file = new File(gemShapefile);

  FileDataStore store = FileDataStoreFinder.getDataStore(file);
  FeatureSource featureSource = store.getFeatureSource();

  FeatureCollection features = featureSource.getFeatures();
  FeatureIterator iter = features.features();

  int counter = 0;

  PrintWriter pw = new PrintWriter(new FileWriter("/tmp/geo.txt"));

  while(iter.hasNext()) {
  	SimpleFeature feature = iter.next();

  	Collection props = feature.getProperties();

  	if (counter > 0) {
  		String gemNaam = "";
  		String geoWKT = "";

    	for (Iterator it = props.iterator(); it.hasNext();) {
		Property property = it.next();

		if (property.getName().getLocalPart().equals("the_geom")) {
			geoWKT = property.getValue().toString();
		}

		if (property.getName().getLocalPart().equals("GM_NAAM")) {
			gemNaam = property.getValue().toString();
		}
	}

    	pw.println(gemNaam + "; " + geoWKT);

    	System.out.println(gemNaam);
  	}

  	counter++;
  }

  pw.close();

The funny thing is every feature has a property “the_geom” which contains the geometry data and its toString() method yields the geometry data as Well-known text. That turned out to be all we needed.

It turns out the coordinates in the Shapefile are in Rijksdriehoekscoördinaten which are cartesian coordinates based on a custom projection and associated rectangular grid for the Netherlands. This is easily verified using this web form to convert a pair back to GPS and looking that up in Google Maps.

The above code produces lines such as:
Leek; MULTIPOLYGON ( ( (224599.999985 582499.999985, 224999.999985 581299.999985, […] 223366.621185 581866.621185, 223499.999985 581999.999985, 224440.97068499998 582470.485385, 224499.999985 582499.999985, 224599.999985 582499.999985)))

This is a simple list of coordinates that define the boundaries of the polygons. A MULTIPOLYGON contains one or more POLYGONS. A POLYGON is one list of the boundary with zero or more lists defining any holes within that boundary. I figured that out looking at the specification for GeoJSON (same data model, different markup) which is the format I am going to republish this information in.

Drawing

With these coordinates, it became quite easy to write a Processing sketch to draw these boundaries. I looked up a datasource for the last European elections and hooked that up for the colors.

Making iterative sketches in Processing with Eclipse is somewhat cumbersome because you need to utilize quite a high level of abstraction if you don’t want your classes to interfere with each other but still Eclipse allows me to work quickly and lets you write Java 1.5 level code against the Processing core.jar (that alone is worth the effort).

I’m going to release a generic Processing sketch where you only need to add a data file with colors or values for each name. Publication as these boundaries as both GeoJSON and SVG is also forthcoming. I’m also looking for the more recent 2010 Shapefile from the CBS with all the current municipal boundaries in it. If there’s demand I can also extract the living quarter and neighborhood level administrative boundaries which are in the other Shapefiles.

Update: Some research shows there’s a very promising avenue to do this stuff by converting the entire Shapefile to GeoJSON as explained in this StackOverflow post and then drawing thath using either ProcessingJS or OpenLayers.

Update: Managed to convert the data to GeoJSON and draw it using ProcessingJS:

This opens up a ton of possibilities for interactive visualization and sharing. More to follow.

Yesterday I attended the fifth edition of This Happened in Utrecht city. It was a great new iteration with more polish even though it was also a big step up moving the entire event to a new venue.

See the This Happened site for the presentations of this session.

The Papercakes presentation was interesting, though I would have wished that they’d focused more on the creative process and not so much on the implementation and user testing. It is a very compelling game and I’m now even more seriously considering buying a Bamboo for my desk.

The Love Hate Punch punching bag hit close to home for me as I used to have a bag like that at home and I definitely can relate to the aggression remediating effects of such a bag. It was an epic story of how to create something interesting using mostly home brew and tenacity.

The Swinxs is impossible not to like. The presentation was oddly paced, but still everybody thought it was great and got a great look behind the process of putting a digital toy on the store shelves (quite a feat!).

Daan Roosegaarde went off in all kinds of tangents talking about creating poetic interactions for public space. His installations are in a league of their own and are a custom designed experience for a certain area. I have experienced LiquidSpace but I had a hard time getting much more out of it than its novelty.

This Happens seems to have arrived at a next level with a space that is comfortable and organizers that are comfortable within that space. Let’s see what’s next.

A small post about the Interactive Environments exhibit at my alma mater. I had to be in Rotterdam anyway that day so I decided to drop by.

There’s some confusion at Delft Science Center whether the exhibit is really open to the public, but it isn’t locked, so just head there and take the first door on your right. Maybe because of the confusion, the machines aren’t turned on, so it’s a rather static display of interactive environments.

Some pictures:

A long held dream came true for me with the delivery this week of my first light stand and umbrella combination. I wanted to play with studio photography longer already, but what really brought it home was following Dustin Diaz’s Project 365 where he did an interesting strobist setup for every day of 2009. I learned so much from that.

So now I can shoot portraits with off camera soft directed light. This is just the beginning:

Today at an event called ‘What makes you horny?’ (#wmyh), I gave a presentation about my inspiration: Big Data.

Data coupled with Design is more or less my field of choice, my work and passion.

This is the same thinking that is behind my proposition that websites are not the most important thing online anymore. Websites are important in so far as they get data and facilitate interaction on that data, but both of those are taking place more and more on completely different locations. It is far more useful to think of the web as streams of data and actors that interact with that data.

The talk was well received though (as always) it stands to be improved greatly under the influences of feedback and ripening.

Update: This presentation is now in Slideshare’s top presentations of the day.

One of the more important online projects Wikileaks got hit by a block this weekend but we at TipiT could quickly respond and get them up and running again. Read the write-up over at the TipiT blog.

The hearings for the various European Commissioners were held last week. Our Dutch representative Neelie Kroes (known from her ferocious busting of Microsoft’s balls) was given a hard time mainly due to political getting back from the Christian and Socialist European fractions for the grilling of their candidates.

Kroes is slated to be the European Commissioner for the Digital Agenda. Thought by some to be a lightweight portfolio, we in the internet business know that it is of course at the heart of most important things: media, science, trade, communication.

Baroso’s brief is as follows (in part):

• You will take the lead in preparing the European Digital Agenda, to promote an integrated ICT policy framework, addressing both supply and demand for digital services, products and contents, ensuring that Europe remains at the technological forefront in this area.
• The EU needs high speed internet networks to realise the potential of ICTs. Therefore, I would like you to elaborate a policy framework to promote investment in high speed internet and set up a coordinated spectrum policy.
• I would also like you to establish an integrated single market for the delivery of electronic services. The EU possesses massive creative, cultural and multilingual potential, which efficient ICT tools can help to tap and transform into productivity gains.
• Europe must also invest in the ICT skills of its citizens. Your role will be to avoid a ‘digital divide’ and to give the possibility to all citizens to acquire e-skills.

(Mission Statement)

Her answers to the questions are here in Dutch, you may be able to find documents in other languages at the main site for the hearings.

I’d read about the question session and the next day Dutch public television put this debriefing with her online:

In one interview she kept saying something about ‘ICTs’ (Maarten said it sounded something like the interwebs) and in the above fragment she excells in vagueness. A lot of criticism on her has been indeed focused on her grasp of the field and her plans for the Digital Agenda which were termed weak.
More video here and audio about the second term this week.

The other criticism that she would not be a champion of the consumer but would take side with the big companies put forward by the Euro Socialists is frankly ludicrous (Dennis de Jong is our Dutch socialist —say no— representative), if you look at her record battling Microsoft.

Dutch pride of having at least one Commissioner in the EC is nice and all, but it would be nice for Europe to have a commissioner for the Digital Agenda who would actually use the internet? One who has at least a first hand idea what this stuff is about.

Given the vast out of touch Kafka-esque bureaucratic monstrosity that is the European Union, putting somebody in charge of the internet who both has a clue to the subject matter and can navigate the treacherous passages of European politics may be too much to hope for. Given that, Kroes may be a very good second and if she displays the same fervor to promote an open internet as she did attacking Microsoft, she may not be that bad at all.

Update: Today’s hearings went a lot better and it looks like she will get the post for Euro Commissioner for the Digital Agenda:

Update: And now it is almost a done deal according to this post on NOS.nl where they also mention that her advanced age may not make her the most suitable candidate for an internet portfolio.

Tim Bray speaks out on what those of us without a vested interest in the disfunctional business practices of enterprise software already knew and have been saying for years:

The community of developers whose work you see on the Web […] deploy better systems at less cost in less time at lower risk than we see in the Enterprise.

It’s a sham and it’s crumbling but slowly enough that if you’re mediocre developer you will not want for work for the foreseeable future (but you will not be doing interesting things either).

Sjors Timmer writes a nice piece about the futility of innovation and the inevitability of failure.

He says:

“Who in their right mind would ever want to be sacrificed in the name of innovation, who in full knowledge would accept an seventy hour work week for unhealthy low pay, who educated by the best would spend their days in cheap office space with Spartan furniture.”

Sjors comes to the same conclusion, but I would answer (free after Camus) that both choices are absurd, both being a cog in a cubicle of bureaucracy and the hopeless railing against the windmills of the upstart.

But if both are absurd, you ought to accept that and choose the more glorious one.