Tag Archives: Website

The Technology of Westminster Hubble

Posted on by
4

For any­one inter­ested (yes this is mostly for you, @HolyHaddock), West­min­ster Hub­ble is writ­ten in a com­bi­na­tion of PHP and JavaScript, with a MySQL data­base as its backend.

Most of the clever work hap­pens in the back­ground, set off by a num­ber of cron jobs with var­i­ous tasks such as keep­ing the MP list in sync with They­Work­ForYou, polling our blogs, gen­er­at­ing sta­tis­tics on the con­tents of the data­base, and the big one: trawl­ing through all the MPs’ feeds themselves.

The lat­ter is a mam­moth job, and try­ing to keep up has been a con­stant bat­tle against allowed cron inter­vals and PHP time­outs as we can as yet only afford shared host­ing for the site rather than our own ded­i­cated server. We keep a record of the last time an MP’s feeds were checked, and every five min­utes, we pick the 60 old­est ones and check them. 60 is a rough value arrived at through some pretty low-tech test­ing, and there’s still plenty of work to do to opti­mise this. With 650 MPs in total, check­ing 60 every 5 min­utes means we cycle through every­one in about an hour, which isn’t too bad, though this will get much worse once we add in MEPs and mem­bers of the regional Par­lia­ment and Assemblies.

Items that get scraped are added to the cache table in the West­min­ster Hub­ble data­base, from where they’re served at user request with­out hav­ing to re-visit the orig­i­nal feeds. We use Sim­plePie to find and scrape RSS feeds, after my own attempt proved to be more trou­ble than it was worth. Sim­plePie man­ages its own cache as a flat file struc­ture, and uses its own intel­li­gence to try and detect when feeds are unchanged, light­en­ing our server load when scrap­ing feeds that don’t update very often.

There’s cur­rently no expiry con­di­tion for items in our cache. Disk space is not an issue, but load times may prove to be at some point in the future. If and when they do, we will start remov­ing the old­est items from the cache, pos­si­bly with some kind of type bias so that blog posts hang around longer than tweets.


On the user expe­ri­ence side, there’s noth­ing much com­pli­cated going on. jQuery is used exten­sively for pulling in page con­tents so that we can load pages with feeds on quickly. Like­wise, we use jQuery so we can fil­ter feeds, and switch between Search, Map and List on the home page with­out reload­ing, and we use the Auto­com­plete jQuery plu­gin on our search box.

The Map view is pow­ered by the Google Maps API, and we gen­er­ate the data for the pins from TheyWorkForYou’s data­base of con­stituency locations.


All in all it’s not been a tremen­dously dif­fi­cult project — there have been no major hur­dles that have caused me to tear clumps of hair out or affected the stocks of cof­fee pro­duc­ers. Though that said, West­min­ster Hub­ble is still in beta, and there could be many more issues ahead…

Announcing: Westminster Hubble!

Posted on by
1

Ladies and gen­tle­men of the blo­gos­phere, I am proud to announce the release of a new web­site. This is what I have been wind­ing up most of my Twit­ter fol­low­ers for the last two months by refer­ring to it only as “Super Secret Project #1″. Its name is West­min­ster Hub­ble.

Westminster Hubble Logo

A joint ven­ture between myself and Chris Cox, West­min­ster Hub­ble helps you stay in touch with your rep­re­sen­ta­tives in Parliament.

Each MP has their own page, which you can nav­i­gate to by search­ing by name, con­stituency or post­code, or by select­ing from a map of the UK or a list of every­body in the data­base. Here’s an exam­ple for one of Westminster’s most tech-savvy, Tom Wat­son.

On each MP’s page, each item of their online pres­ence is listed: their web­site, blog, Twit­ter account, Face­book page, and so on. These are all editable, so that MPs or benev­o­lent users can help keep their page up to date. MP’s pro­files on They­Work­ForYou, the UK Par­lia­ment web­site and the Tele­graph news­pa­per are all auto­mat­i­cally linked to as well, and if they’re on record, the pages also list MPs’ con­stituency addresses and phone numbers.

Each of the online items is rou­tinely checked for updates by West­min­ster Hub­ble, and from them a feed is cre­ated. This feed forms the body of each MP’s page, and is a fil­ter­able list of all that MP’s activ­ity on all the web­sites we know about.

Of course, you can sub­scribe to an RSS feed for an MP too (or even a sin­gle feed for all 650 of them, if you’re that way inclined). Just click the “Sub­scribe” but­ton at the top of any feed.

So if you want a sin­gle feed of your MP’s speeches in Par­lia­ment, blog posts, tweets and YouTube videos, West­min­ster Hub­ble is a new site that will give you just that. If you want a sin­gle page that’ll give you quick access to all their pro­files across the inter­net, we do that too.

Please, spread the word!

Farewell, Dynamic Democracy

Posted on by
12

Back in April, the Dig­i­tal Econ­omy Bill was rushed through the wash-up pro­ce­dure of the out­go­ing gov­ern­ment with­out the due debate and con­sid­er­a­tion that I and oth­ers believe such a far-reaching bill deserved. My dis­il­lu­sion­ment with the gov­ern­ment decision-making process over the fol­low­ing week led me to set up and announce a new site, called “Dynamic Democ­racy”. It was an exper­i­ment to see what would be dis­cussed if every­one was involved — on an anony­mous basis — rather than just our elected rep­re­sen­ta­tives that often do not do a good job of rep­re­sent­ing us anyway.

The site allowed all users to cre­ate and com­ment on ‘Bills’, encap­su­lated ideas or laws that they would be push­ing for if they were in power. Reg­is­ter­ing gave users the abil­ity to vote bills (and com­ments) up and down, lead­ing to a list of highest-ranked bills that rep­re­sented the users’ favourite poten­tial policies.

Dynamic Democ­racy saw lit­tle suc­cess, pos­si­bly because writ­ing a full, well-thought-out bill rep­re­sented sig­nif­i­cant effort that a casual browser would be unlikely to com­mit. ‘Karma’, the point sys­tem that aimed to encour­age users to sub­mit bills and com­ments, did not prove to be a good enough incen­tive as there were so few users to com­pete with and no direct reward was ever imple­mented for reach­ing high karma levels.

What the site did bring, how­ever, was a num­ber of enquiries from like-minded indi­vid­u­als all over the world, keen to dis­cuss the ideas behind the site and whether or not some­thing like Dynamic Democ­racy could ever be imple­mented as a real gov­ern­ment policy-making tool. One of the more notable con­tacts, Denny de la Haye, stood as a can­di­date for Hack­ney South and Shored­itch in the gen­eral elec­tion and promised to imple­ment a crowd-sourced vot­ing sys­tem sim­i­lar to Dynamic Democ­racy for his con­stituents to voice their opin­ions in Par­lia­ment through him. (Denny, who sadly did not win his seat, now rep­re­sents the UK arm of polit­i­cal party DemoEx.)

I have decided that today is the day to close the Dynamic Democ­racy exper­i­ment, because today the UK gov­ern­ment announced their “Your Free­dom” web­site. While largely focussed on repeal­ing or chang­ing laws rather than the com­plete free­dom to sug­gest any­thing you like, Your Free­dom is cer­tainly in the same vein as Dynamic Democ­racy, with the cru­cial extra fea­ture that is endorsed and used by our gov­ern­ment and thus ideas pro­posed there stand at least some chance of mak­ing it into offi­cial gov­ern­ment policy.

Time will tell whether that really hap­pens, or if like the No. 10 Peti­tions site, sug­ges­tions will be responded to with an e-mail from the Prime Minister’s office explain­ing why thou­sands of users are all wrong. But I do still hold out hope.

Did Dynamic Democ­racy influ­ence the gov­ern­ment in their deci­sion to cre­ate Your Free­dom? Almost cer­tainly not. As my dis­cus­sions with vis­i­tors to the site have shown, I am far from the only per­son to have come up with this idea, and nei­ther am I the only one to have coded up a web­site around it. No — this is sim­ply an idea whose time has come. A vast gulf exists between West­min­ster and the world out­side, just as it always has, but these days the pub­lic are com­ing to ques­tion why that is and if we can do some­thing to cor­rect it. And nowhere is the desire to bridge that gulf stronger than among the tech-savvy youth that have the drive and the abil­ity to use the inter­net to that end. Sites like these will come and go a hun­dred times over the com­ing years and decades, and slowly but surely we’ll reshape our gov­ern­ment into what we want it to be.

So to every­one who con­tributed to Dynamic Democ­racy: thank you, and goodbye.

If you’d like to con­tact me about Dynamic Democ­racy (or any­thing else), you can still do that here. If you’d like to help get the Dig­i­tal Econ­omy Act repealed, please vote up and com­ment on one of these ideas on Your Free­dom. If any­one would like use of dynamicdemocracy.org.uk until my own­er­ship expires in 2012, let me know. Stay tuned for the announce­ment of another project that bridges pol­i­tics and the inter­net in the next few weeks.

An Experiment in Dynamic Democracy

Posted on by
6

Dynamic Democracy

Dynamic Democ­racy

I’ve been an advo­cate of open­ing up our democ­racy and involv­ing the pub­lic in gov­ern­ment decision-making for some time, with­out doing any­thing par­tic­u­larly con­crete about it besides plac­ing my vote. The Dig­i­tal Econ­omy Bill fiasco showed us that, really, we’re not involved with the day-to-day work­ings of gov­ern­ment at all, and born of that is this experiment.

I’d like to know what we, the peo­ple, think our gov­ern­ment should be talk­ing about. I’d like us ordi­nary peo­ple to sub­mit our ideas, vote on other people’s ideas, and come up with some idea of what we really care about. And so here we are:

Dynamic Democ­racy

This is all very exper­i­men­tal at the moment — please sign up, post ideas, vote on other people’s ideas, and if it proves pop­u­lar I’ll take it on as a per­ma­nent project. Let’s do this!