Experience Zeeburgereiland

This week I biked past Zeeburgereiland on my bike route from the Eastern Docks to IJburg and all I saw was completely desolation and this billboard. I had wanted to bike along side the IJ but this part of the city is so unfinished larg pieces of it are inaccessible.

This is what it looks like through the fence:
Zeeburgereiland

I completely forgot the text on the billboard and had a hard time finding the place. Google Maps doesn’t even know it yet. Just look at Street View.

My trusty neighborhood council member Jan-Bert Vroege is usually available to answer these kind of questions for me and he pointed me towards the right site.

Zeeburgereiland

It’s going to be the site of some significant development in the near future. It’s odd that in developing IJburg the city skipped over this part and is only now beginning to fill it in.

The site points to a fantastic concept at the site of the current silo’s to build an art house, playground, museum and restaurant with a fabulous view and attractive position: the Annie M.G. Schmidt house.

Looking forward to this, but curious if the exurbs of the city beyond where I live (Diemen + Bijlmer) will ever be (re)developed or if we’ve forsaken those to become multicultural ghettos:

I biked on to IJburg and large swaths of that area still are empty and being built and filled in. Most people won’t remember how long it took for tram 26 to be extended there or when the first supermarket opened (is there one now even?).

Some nice parts too:
IJburg

Concerning that part of urban development we can learn a lot of the prefab cities being erected in the East (take Dan Hill’s account of New Songdo). I know it takes time to fill up residential units and some amount of organic growth/frontier mentality is good, but is it that hard to have basic amenities in place when the first people arrive there?

Have your Surface

Haven’t got mine yet, but yes they’re Magic Tables.

During the run up to the iPad launch most pundits got into a pissing match of naysaying and detracting from the device. “It’s just a bigger iPhone.” Most of which has by now been disproved.

Like BERG say, it’s a consumer ready, performant, portable version of the Microsoft Surface. If you have ever played with a Surface, you know that it’s a horribly bad and slow experience AND it has a €10k price tag attached. The iPad is a Surface that works for a very broad definition of ‘works’.

Microsoft can go back to the drawing board. Apple ate their lunch.

Fresh election results

Interesting to see that there were a ton of election result charting, display and visualization sites that popped up just before the elections. It seems that this is a hot topic.

We just added a preliminary dataset with yesterday’s election results. If you want to see the decimation of the CDA and the blue spread with the rise of the VVD (also compared with the municipal elections earlier this year), see here: http://bit.ly/a7s3l2

Quick entry of 2010 parliamentary election results

There are some outstanding results and some outstanding todos for the map which explain the holes. Those will be fixed shortly when the entire map migrates to the 2009 Shapefile.

Dutchstats — Your personal Atlas of the Netherlands

Last weekend we released Dutchstats, an online interactive datavisualization of election results and local statistics for Dutch municipalities by Monster Swell, an Amsterdam dataviz agency.

What?

Dutchstats — an application to view Dutch election results and statistics collected by the CBS side by side.

http://bit.ly/dutchstats

index.html

A video demo:

Dutchstats from Alper Çugun on Vimeo.

Who?

Dutchstats has been made by Monster Swell with Alper Cugun and Alexander Zeh.

Which data?

The CBS  publishes Shapefiles with the boundaries of the municipalities, burroughs and neighborhoods of the Netherlands with a legend of the statistics they bundle. The tools that the CBS themselves provide to view this data are not very user friendly, see both: Statline and CBS in uw buurt.

The election results for the European Elections of 2009 came from nlverkiezingen.com. We entered the results for the 2010 Municipality Elections ourselves. Parliamentary Election results are provided by the Kiesraad (electoral council).

Why?

This project started as a foray into drawing municipal boundaries on the screen. No easy to use tool to plot statistics at that level existed yet, so we thought that would be an interesting and useful problem to tackle. After some attempts we created a proof of concept in Processing, but we did not think a Java applet is ‘distributable’ enough. We then rewrote the entire thing in Processing.js.

A logical consequence of having an easy way to draw local statistics was to create a tool to visualize the statistics that were already bundled in the provided Shapefile. We think the result is quite interesting.

Wat is so interesting about it?

There are tons of stories in this data waiting to be told just like there used to be in any old atlas. Instead of paging through the graphics and charts in one of those, Dutchstats allows you to compare statistics for the same area side by side. It makes the social geography that the CBS collects directly tangible with as little jargon as possible.

Some interesting views on the data:
The decimation of VVD and PvdA by LPF in 2002
The dissolution of LPF in 2003
The location of the Dutch Bible Belt
The prevalence of PVV compared to the amount of non-Western immigrants
Crop farms compared to cattle farms

But there are dozens more interesting stories waiting to be told in these numbers and that is exactly what makes them so interesting. You can keep clicking through, changing the selections, viewing local results etc. That is exactly what we think a good datavisualization should provide.

What’s next?

The source code is available on github. An open source release geared towards making it easy to plot your own data on the map —even for non-programmers— is forthcoming. Also we only discovered a more recent Shapefile on the CBS site after the public launch of this project. We will integrate that as soon as possible.

Some other stuff we are considering:

  • Speed optimizations, the visualization is near unusable in Firefox. Recent versions of Webkit and especially Chromium perform much better. I would be interested in hearing if this works in IE9 and how it performs.
  • Zooming display, the same statistics are provided on lower levels as well. Adding those shapefiles and enabling drilling down to the very local level could be very interesting.
  • Showing all of the data displays in small multiples.
  • We’re always interested in displaying more data that can be aggregated at the municipal level. Get in touch.

Dead trees

I don’t think I disagree with anything Craig Mod writes.

In his: “Books in the age of the iPad” he argues succinctly for the abolishment of most printed books in favor of those that are properly designed for the medium.

I myself am going to get rid of all of my books save the design, poetry and photography ones. Because:

The convenience of digital text — on demand, lightweight (in file size and physicality), searchable — already far trumps that of traditional printed matter.

It’s good that most people haven’t caught on to that yet. It gives me time to get rid of my book collection before the perceived value of printed paper plummets through the floor.

Ranking our schools for better effect

Last week we had this piece of news in the Netherlands: “Scholenranglijst van Trouw heeft effect” (School ranking by Dutch newspaper Trouw sorts effect):

Trouw has a database on their website with the statistics from our board of education which they editorialize and republish on their Schoolprestaties site. The news piece reports on research that shows this ranking has a real effect both in prospective school choice and in school improvement.

This week BERG release their Schooloscope (formerly Ashdown) site that makes statistics about school quality accessible. The story about how it works is particularly worth reading. The entire thing is a beautiful work of insight.

The Guardian has a profile of BERG’s work and about Schooloscope and it raises the question if there is no end to the goodness that will come from the 4ip funds.

The Trouw school site we have, though it could have been more beautiful, more legible and more open, does get the job done quite well.

But I’m reminded of the concept we made for a school search engine using government data: Schoolvinder (with many similar goals: to remove jargon and to make school information accessible and beautiful). We applied for a grant to further improve that site, but that was awarded to another party from whom we have not seen any results since.

Elections and Campaign Data Visualization

With the upcoming Dutch elections the campaigns are heating up and we will be allowed to choose our parliamentary representatives yet again. It is interesting to see how statistics and their visualization are used to clarify and position issues in our complex world.

Issues with visualization

You cannot release statistics and visualizations without thinking through the ramifications of these actions. Every non-trivial bit of information has biases and values attached to it. You can never know what will happen, but you can at least think about it.

The other day at the Rotterdam Open Data meeting, someone vehemently defended the point of view that we should not publish data because it could be spun in a way that is harmful to society. A wholly subjective and belittling point of course to which we countered, that unfounded claims can be launched already and without authoritative data sources we do not have a good way of debunking them.

A lot can go wrong when using data visualization, just see this video of a presentation by Alex Lundry which is familiar territory, but is brought nice and quickly:

Or this recent example from a so called Dutch quality newspaper about Greec and other European edging towards the brink of financial ruin spotted by the great Sargasso:

This is guilty of the fiddling with origins and axis scales that is so common in juicing up statistics for presentation purposes. Other faults from the video: “sin of ommission”, “correlation is not causation”, “pie-charts suck”. Most of this is treated pretty well in Tufte’s Visual Design of Quantitative Information where he calculates the lie factor of faulty infographics.

US Job Losses

Data visualizations —especially charts of statistics— in elections are also nothing new, but with the increase of open data and data processing tools, we are bound to see more of them coming out and I hope to see more dynamic ones especially.

The Obama Job Chart (below, taken from Creekside Chat) is a very static traditional chart which could have just as easily been punched out of Excel (though the extra visual touches are nice), but the most important part of this chart is how it supports an overall narrative:

I take issue with the poster’s critique because the chart clearly says that it shows “Job Loss” and not absolute unemployment. Any turnaround of the economical situation will be necessarily coupled by a trend as displayed in the chart (losses have to edge back to zero before they can become gains). Also the comparison to the amount of money in a wallet does not really work because money spent is an absolute loss, while the amount of people in the job market is a pool which is in flux.

UK Job Losses

The UK will have their General Elections next Thursday. In the run-up to the elections, Russell Davies spotted this nice interactive chart by Labour to clarify how they helped reduce unemployment.

Jobs Interactive Map | The Labour Party

It remains to be seen how far these kind of more technocratic online methods support the narratives and media plays that an election revolves around. It does not look like it has helped Labour that much in their struggle.

Combined Approaches

So how to combine online more mechanical and easy to ignore material with the mass-media appeal of legwork on the campaign trail?

What’s more likely to be pivotal is the canny use of the latter to leverage the former: ensuring that every casual contact goes into a database, every issue raised by a constituent (or inferred from a pattern of facts on the ground) is captured and tracked, everything that shows up in the gillnet of your feeds is exploited for its propaganda or organizational value. —“Harvey Milk, community development and the digital balance sheet” by Adam Greenfield

As suggested by Adam Greenfield, a combination of both may be the best option, but besides the much praised Obama campaign we haven’t seen much successful work along those lines yet and even the Obama grassroots organization has been underutilized since the inauguration.

The Dutch Situation

One question would be: Where is the Dutch job loss chart at. If I can massage the correct statistics from the CBS, I’ll see if I can whip up something.

Many political organizations in the Netherlands, do not have the budget or maturity in web infrastructure to be able to quickly create and deploy bespoke applications that are situated within their workflows and fit within campaign deadlines.

A small but comprehensive overview of online activities for the Dutch general elections can be found on Spotlight Effect (in Dutch) but small really is the operative word. I am aware of a couple more initiatives due to come out but it’s quite meagre.

Also when talking about the overarching themes, I haven’t spotted the ones that our election is supposed to be about yet. Unless it is whether you envision a divided Netherlands where a discontented white proletariat rules over both foreigners and intellectual elite alike or whether you want a whole country governed by sane and rational people.
Issues such as education, technology, healthcare, immigration, urban and ex-urban planning for a decreasing population, our international position, energy and food security and all of those with a vision of at least 10 years into the future are sorely lacking. This is probably because most of the population is too shuttered inside their blocks and suburbs to be able to look over the rim of the nearest enclosing dyke.

Update:
This seems to be the overarching theme of the elections for the PvdA.
Iedereen telt mee

Update:
Alex Lundry notices that the Obama job loss chart is being updated by the Washington Monthly. Here’s the April version:

And here’s the same chart for the Netherlands albeit a lot less granular (if anybody has that data, I’d greatly appreciate it):
Werkloosheid Nederland

Table Viewer for Music Hackday

This entire weekend was taken up by Amsterdam Music Hackday for which Alex, Dirk and I had planned to build a prototype version of a surface table projector for music discovery.

The functionality we envision helps ad-hoc groups of people who find themselves in the same location/venue/party to compare their music tastes and see where the overlaps and where the holes are. The table would be a turn-taking jukebox with tangible interactions and nice visuals for all users and spectators.

Easier said than done, of course. We spent a great part of the week and most of the weekend hacking, building, eating, drinking coffee, staying up to the wee hours, literally stabbing ourselves with scalpels, cursing a great deal and drinking whisky to get the thing together, when finally on Sunday in the last hour before the presentation we managed to integrate everything to the level that we could shoot a demo video.

Pictures of the proces and demo videos below:

Real men eat meat!

BENQ

Top view

Point to interface

Last.fm

What we built was just an initial step on the way to the jukebox I described above, but it seemed to look promising enough to net us the first prize from last.fm at Music Hackday for which we were very happy.

We like to thank last.fm, the organizers and the participants of Music Hackday. It was a great event and for us it was a great occasion to finally get this project started.

We will develop the table further and build out the functionality we had envisioned to make it a real locus for social music discovery. It should be hanging in one of our studios soon, so get in touch and visit if you want to try it out.

New disciplines for a real-time data world

Some posts that had been sitting in my browser tabs for a while combined together in a brand new job guide for 2010. You can also read this as a follow-up post to my previous post on Why developers are important, this is which developers are important. This post has been lying in my drafts folder for a while, but it has actually only become more relevant.

Some interesting jobs for the coming year(s):

Scalability Engineer

These are already highly sought after ever since Twitter was failwhaling half of the time. Having the competency to keep a website running while it is experiencing massive growth is going to be highly sought after. Some technologies such as Google App Engine promise to make this easy, but they introduce a set of problems of their own. Traditional relational databases are abandoned more and more for the looser often schemaless variety of BigTable-like NOSQL databases that live in the cloud (CouchDB, HSQL, Cassandra, MongoDB, Tokyo Cabinet etc.) or can be scaled at will. If you want to get up to speed on this stuff really fast, there’s a NOSQL conference in London April 20-22nd.

Also knowing your Scala, Tornado, Twisted, NodeJS or other non-blocking framework is increasingly important, since we’re slowly moving out of the request/single response paradigm for the web.

Key skills: Everything command line, functional programming, traditional database management, SQL, virtual machine configuration, puppet

Client Side King

The web based client already was the biggest delivery mechanism for functionality and experience, but it is going to become more and more important. Functionality which you would not have thought possible in a web application, will become available. Some apps may at first be functionally inferior to their native versions, but the fact that they are web native and inherently social will draw people in. After a while either the apps will become more capable or the users won’t care anymore.

Developments in HTML5 bode well for those of us who would like to abolish slow and crashy plugins (yes, Flash). Audio, video, hardware accelerated 3D graphics and much more will soon be native to the web. Just look at Quake II written in JavaScript. There is still a place for adding Silverlight and Flash to certain sites, but the benefits of those technologies are much harder to argue for.

Key skills: JavaScript all variants, styles and frameworks; web-native UI design; marginal IE skills required (since you will not be building for that platform but you should know its limitations); iterative development; guerilla user testing

Algorithm Cook

Ridiculous amounts of data requires strong analytics, very capable navigation and a new sort of editorial proces. These databases draw more and more information from the real world:

“The advent of inexpensive high-bandwidth sensors is transforming every field from data-poor to data-rich,” Edward Lazowska, […] said (NYT) and “Today,” he added, “you have real-time access to the social structuring and restructuring of 100 million Facebook users.” (same source)

Better algorithms will allow us to make better sense of all this data and will provide inputs for the other fields. Everything can have an interestingness in a given context for a given person.

Key skills: multivariate statistics, data wrangling, screen scraping, machine learning, data mining, Excel, SPSS, R/SPlus, Matlab, NumPy, digital signal processing

Visualization Artist

Making sense of all the information requires condensed views with aesthetic qualities. There is simply too much data out there for us to be able to grasp it, so being able to filter and mine the datasets with the help of the other disciplines is essential. But after that step any data needs to be refined, represented and made interactive.

“Decode” ends with “Network,” which examines the interconnections of mobile technologies and the Internet. It also illustrates how digital imagery is helping us to make sense of a frenzied, often confusing world. (NYT)

There are tons of frameworks, tools and libraries in a variety of languages for anybody who wants to try out visualizing stuff. In the end no single one will fit the bill and the best result is achieved combining, mixing and writing something by yourself.

There’s a new O’Reilly book coming out for anybody in the finer arts who’s interested in getting their feet wet with Processing: “Processing for Visual Artists” Then after a while you may be able to produce stuff such as:

Key skills: aesthetic sense, 2D/3D graphics, cognitive psychology, Processing, OpenGL, JavaScript, SVG, design tools, Tufte

And we haven’t even treated the Natural Language Processer, the Urban Information Planner and the Machine Vision Trainer yet but there’s considerable overlap with the above disciplines. If you have any other that we should look at, please suggest them in the comments.

Update: And the New York Times is just looking for somebody with roughly this description.