Local searches on the BBC

We had an informal hackday in the RAD team on Friday, Suzy got the top 10,000 search terms in one day, July 9th, from the BBC search team and we extracted and plotted any UK placenames (see my previous cuckoo map). We're interested in how much local information people search for.

Interesting things at Activate 09

Martin and Roo have posted much more comprehensive write ups of the Activate 09 conference put on by The Guardian yesterday, so I'm just going to add some interesting facts and quotes which show, I think, what a remarkably diverse, interesting and sometimes inspiring conference it was.


Use tech as a way of thinking for rebuilding failed states. Clare Lockhart

"Mainstream media is ADHD, new media is OCD." Arianna Huffington

The Plague of Justinian in the 6th century left 100 million dead, about 50% of the population of Europe. Nick Bostrom

"Context = location + time + history + social" Ed Parsons

Jon Udell is building a local events site that takes inputs from Upcoming, Eventful and a curated delicious account of relevant iCal feeds, aggregates them and publishes them, including onto the local community TV channel. (me: I love the connections going on here.)

Political blogs are Henry Ford's "faster horse". The technologies that don't look like journalism will be those that change government and politics. Tom Steinberg

Amazon's Mechanical Turk made work into a game - you can choose tasks, do them when you want, you get points and you can communicate socially with other workers. Some key dynamics of games: Personal progress, competition, recognition and goals. Andy Baio

(me: I didn't note who said these and I'm paraphrasing...)

Highly usable online credit card transfers made the difference for Amazon and Obama.

"Test it, if it works scale it"

"We need a government where tolerance of failure is accepted"

"Teach people how to use the tools of participation"

"Develop tools that allow us to understand and analyse all the information that is now available on the internet"


(me again) Go and read something from all the people mentioned above, it'll do you good, and if we actually do or make some of these things that would be even better.

The Springwatch Cuckoos

My BBC colleagues, Tom Scott and Roo Reynolds were recently kicking around ideas for what to do with the 12,000 or so comments (a BBC record) that a blog post for the BBC TV show Springwatch recently received after asking if people had heard a cuckoo. They had lots of interesting ideas but I immediately connected this with the recent release of the Yahoo Placemaker API. This service allows you to submit some text and receive a list of recognised places back - the place name, a unique Where-on-Earth ID (WOEID) and a latitude/longitude. So I've written some Python scripts to extract placenames from the cuckoo comments and plot the sightings (hearings?) on a map.

I submit the comment text to the Yahoo Placemaker API in batches because Placemaker has a 50,000 byte limit for posted documents. This discovered 16078 place mentions consisting of 5386 unique places. The Top 20 are...

1. New Forest, England, GB, 99 mentions


2. Scotland, GB, 94


3. Woodland, England, GB, 83


4. Suffolk, England, GB, 80


5. Yorkshire, England, GB, 79


6. Norfolk, England, GB, 77


7. Lake District National Park, England, GB, 77


8. Dartmoor National Park, England, GB, 77


9. Wales, GB, 74


10. Essex, England, GB, 72


11. Sussex, England, GB, 66


12. Surrey, England, GB, 66


13. Kent, England, GB, 59


14. Somerset, England, GB, 51


15. Hampshire, England, GB, 50


16. Cumbria, England, GB, 48


17. Norwich, England, GB, 47


18. Island of Skye, Scotland, GB, 47


19. Perthshire, Scotland, GB, 46


20. Dorset, England, GB, 45

The top places are mainly counties or areas as you'd expect but further down the list are towns and villages. There are some false positives, like "Woodland" above, but by filtering the place names to include "GB" I removed many of these. There is another problem where some comments will generate multiple locations - e.g. "Basingstoke, Hampshire" gets two places extracted.

Then I converted these places into KML, the data format used by Google Earth, and generated a compressed KML file of the sightings that you can download. You can load this into Google Earth and it seems to cope, though it's pretty useless looking at a map full of pins until you zoom in. You can also paste that URL into the Google Maps search box to see the sightings but it will only load 1000 points and only plot about 80 points at one time - zoom in for more detail.



Finally, as originally suggested by Tom, I drew a heatmap-type image using Nodebox and Python to plot translucent dots onto a map of the UK (courtesy of OpenStreetMap). I fiddled around with the size and opacity until I got this (click to enlarge)...



Tom got into contact with Springwatch and I was phoned by Paul, their web producer. It turns out they had already got a data entry company to extract the place names from the comments, using people to do what I'm doing with code, but hopefully getting higher quality results. They are also sharing the data with the British Trust for Ornithology who will work on producing more accurate results from it. I redrew the map using their manual data and they also asked for a couple of closer views of London, Birmingham, Manchester and Scotland which I also generated though these aren't quite as pretty..





It's not scientifically accurate by any means but it is interesting and was a good experiment in extracting useful data from large numbers of comments.

I've just been watching Springwatch right now and they featured a map but I'm not sure it was mine - looks like it was this one from the BTO.

Disclaimer: These are my thoughts and opinions and not those of my employer

Links for 13-03-09

279 Spore skeletons
Using the Spore API and Processing to draw Spore creature skeletons

Eclipse | EcoArtTech
Online artwork that alters images of US National Parks based on the current air pollution. Nice idea though the image corruption could be more interesting

Guardian API Maps
Cute geocoder for Guardian articles, developed by Stamen...

ContentTagger
...and a prototype article tagger built on the new Guardian Open Platform which uses Freebase for lookup and disambiguation. The problem with both of these is that there's no motivation, apart from the goodness of their hearts, for people to tag.

Sinatra
Very lightweight Ruby web framework...

Heroku
...and Rails hosting in the cloud

Wall Street on the Tundra | vanityfair.com
"Iceland’s de facto bankruptcy—its currency (the krona) is kaput, its debt is 850 percent of G.D.P., its people are hoarding food and cash and blowing up their new Range Rovers for the insurance—resulted from a stunning collective madness. What led a tiny fishing nation, population 300,000, to decide, around 2003, to re-invent itself as a global financial power?"

About that Vanity Fair article
Good rebuff from an Icelander pointing out the more fictional parts of that Vanity Fair article.

Queen’s student gives twitterers ‘new voice’
A hacked radio that "...allows fans to listen to Twitter messages posted on the website in real time so they can keep up to date with friends, celebrities and even complete strangers. Mark uses an old fashioned radio to receive the tweets..."

More at http://delicious.com/tristanf

Links for 03-03-09

Haven't done this for a while so this is a somewhat random selection of recent interesting links...

Flickr: stolenstrategies
A Flickr group for photos of "things that have been translated into code. images must be split into inspiration and outcome. real thing on the left, coded thing on the right."

melka » Soundscapes
An ambitious series of visualisations of music from Messiaen to Pink Floyd using the EchoNest API to analyse the MP3 and then translating the results into a PCR/DNA-like diagram

In search of the click track « Music Machinery
Also using the Echo Next, but this time using the remix Python library to work out which drummers use a click track (or can keep time)

Good Radio Club
From Jem and Steve - "Its like, social listening. ie: its a bunch of fellow radio4 fans tweeting along in real time listening to a 30 minute radio show chosen in advance. Its live. We thought about doing Listen Again but went for an old fashioned scheduled broadcast. Its 8.30pm. This Thursday February 26th on BBC Radio 4."

S3 Radio
"You make a bucket on Amazon S3, filled with MP3s. When someone visits your S3 Radio station, it builds a playlist from the contents of the bucket and starts streaming it to the listener, in a random order." - easy internet radio from Tom Taylor.

The Curious Cook - Do You Need All That Water to Boil Pasta? - NYTimes.com
"My rough figuring indicates an energy savings at the stove top of several trillion B.T.U.s. At the power plant, that would mean saving 250,000 to 500,000 barrels of oil, or $10 million to $20 million at current prices"

More at http://delicious.com/tristanf