O'Reilly Radar - Insight, analysis, and research about emerging technologies.

Four short links: 3 July 2009

2009-07-03T11:00:00Z

OECD Factbook -- Flash-built impressive data explorer from OECD. Go to Indicators > Load and, in the words of Ben Goldacre, "prepare for nerdgasm". (via bengoldacre on Twitter)
James Boyle is on Twitter -- author of the book The Public Domain.
Sewers and Startups (Pete Warden) -- designing to last, reminds me of Saul Griffith's heirloom design riff. When I joined Apple back in 2003, the central build farm for all projects had both PowerPC and x86 Darwin boxes, and our code had to compile on both. Steve was playing a long game, years before the Intel switch he was obviously planning for it, (though I only caught the significance in retrospect).
Open Data Makes Garbage Collection Sexier, Easier, and Cheaper -- pragmatic use for open government data. For more on the author of this post, see Hello World for Open Data by Tim Bray.

Twitter Approval Matrix - June 2009

2009-07-03T03:28:15Z

Last month I posted the first Twitter Approval Matrix with data that spanned the month of May and different sources such as Hashtag.org, scraping archives, and observations. This month I received some help from Joe Fernandez the CEO of Klout.net and Dan Zarrella the Social & Viral Marketing Scientist for danzarella.com. They provided some great 'hard' data that allowed me to better place more items on the grid this month.

A quick refresher, the matrix shows four quadrants used to describe trends found on Twitter, or related sites such as hashtag.org, tweestats.com, etc. The Y-axis is partly analytical and shows popularity (mostly through scraped numbers) or perceived popularity (in the future nominated by you). The other part of the grid is more curated and subjective. The X-axis has been plotted based on my personal opinion. You may agree or disagree with my placements and that's all good to me. After all, it is about taste. The matrix and plots do not represent a thorough analytical treatment, but rather a view of the trends that could be found in data sources allowing me to plot with some sense of relevance.

For this post, I've limited the data and activity to the month of June. Again, I'll continue with this project as long as I get enough feedback/help. So, if you are interested in contributing, you can comment here, or read the original post to figure out the best way for you to submit your plots.

I hope you enjoy this and see it as a potentially useful tool to monitor trends that your fellow readers are tracking.

Ignite Los Angeles on 7/21! Submit a Talk

2009-07-02T19:35:03Z

Ignite is coming to LA! As always speakers will get 20 slides that auto-advance every 15 seconds. We're going to be holding the geek event at Cinespace in Hollywood on 7/21. Submit a talk now.

This will be the first Ignite in Los Angeles; it is co-hosted by LA Geek Dinner. The LA G33k dinner was kind enough to let us take over their July dinner to host the first Ignite. LA G33k Dinner, founded by Heathervescent brings people with a passion for technology and the internet together over a meal where conversations happen, friendships form, and collaborations on various projects occur. L.A. Geek Dinner is an inclusive event. Find them on Facebook.

The event is free. We're hosting it at Cinespace on July 21:

6:30pm Geek Dinner starts

8pm-9:30 Ignite talks

10pm Cinespace opens to the eneral public for Dim Mak (you're welcome to stay for the band)

While the event is free, you are responsible for paying for your own food/drinks from Cinespace if you want 'em. Please RSVP to the Geek Dinner list on Upcoming.

If you're working on an interesting project, have an unusual skill, or just some interest that would be fun to share with everyone, please submit a proposal to: http://bit.ly/IgniteLA

Ignite LA is being organized by Brady Forrest, Matt Forrest, Dan Gould, and Heathervescent. If you're not familiar with Ignite check out some videos on the Ignite Show.

Four short links: 2 July 2009

2009-07-02T15:00:00Z

UNESCO book: Open Educational Resources -- UNESCO's first openly licensed publication, a collection of papers and reports in the area of Open Educational Resources. (via glynmoody on Twitter)
ETSI 2.0 -- Paul Downey ventures into the belly of the telco beast and gives them both barrels. The whole thing is great--his talk was one of the best overviews of "how we think on the Web" I've seen. I can only imagine the sound it made as it bounced off the thick dinosaur hides of the attendees. I was reminded of the old, apocryphal quote from a Kodak executive dismissing digital cameras and their poor quality with "people love photos", when in reality it's the taking of photos that people love. Sometimes it's hard for an incumbent with large sunk costs and a vested interest in business as usual to foresee and embrace change. Indeed for a telco or large commercial software vendor the best way to predict the future is to prevent it. (via benjaminblack on Twitter)
Asia Pacific FTTH Market Study -- notable for Hong Kong's discovery with fibre-to-the-home customers: Uplink traffic is 3 times of downlink traffic. That link appears dead, but Google has it cached. (via previous link)
Shownar -- tracks blogs and Twitter plus other microblogging services, finds people talking about BBC television and radio, shows trends in appealing ways. Made by Schulze and Webb (and Dopplr's delicious Matt Jones), more detail available that you should read.

Patrick Collison Puts the Squeeze on Wikipedia

2009-07-02T14:00:19Z

Think about Wikipedia, what some consider the most complete general survey of human knowledge we have at the moment. Now imagine squeezing it down to fit comfortably on an 8GB iPhone. Sound daunting? Well, that's just what Patrick Collison's Encylopedia iPhone application does. App Store purchasers of Collison's open source application can browse and search the full text of Wikipedia when stuck in a plane, or trapped in the middle of nowhere (or, as defined by AT&T coverage...) Collison will be presenting a talk on how he did it at OSCON, O'Reilly's Open Source Convention at the end of July, and he spent some time talking to me about it recently.

James Turner: Why don't you start by talking about your background a bit and how you got involved with working with the Wikipedia?

Patrick Collison: I guess I've always been pretty interested in Wikipedia, and I ran my own MediaWiki installations back when I was in school in Ireland. We had our own personal ones and all of the rest. Then in November of 2007, I went to visit my friend in Japan for a month. And in Japan they have all of this incredibly advanced cellular technology and all of the rest. And so because of that, they had very few wireless networks, and my phone didn't work. As a result, I actually had very little access to the Internet. I sort of realized without Wikipedia how little I really knew. And I had just got an iPhone, so I decided to try basically putting a copy of Wikipedia on the phone, so that I'd have it as I was walking around in Japan. Then basically, I spent a significant fraction of my time there in Japan, again, in 2007 writing those applications, say maybe two or three weeks, just firstly trying to decide if it was possible and putting it all together. And then it was released, I think, January of 2008.

James Turner: Now you've also worked on getting it onto the OLPC I understand. How did that occur?

Patrick Collison: I actually didn't do much of the work for this. It was actually a project led by Chris Ball who works both with FreeBSD and with the OLPC project. But I released the code to this application; it was open source from the very start. So it was pretty easy for them to take it and to port it to the OLPC. I mean there are already some applications that allowed you to put a copy of Wikipedia on your computer or something like that, but none had really been optimized for embedded or low power devices or anything like that, which obviously Wikipedia for the iPhone had to be. I think it took about two or three weeks to take the code that ran on the iPhone and then to bring it to the point where it'd run on the OLPC.

James Turner: There are obvious benefits to having Wikipedia on the OLPC, because connectivity is very important in some of those areas. So you'd want to have it local, but outside of the experience that you were just describing, isn't the point of the iPhone that you can just access Wikipedia? What are kind of the advantages of having it locally?

Patrick Collison: I actually find that you spend, or I certainly spend a surprising amount of my time without access to the internet, even with the iPhone. Say for start if you were abroad, I mean everyone knows the horror stories of the data changes AT&T will issue you with if you're roaming. But also just stuff like personally, I find that on a plane or something you have eight hours to not do much. And so I actually end up doing a lot of my Wikipedia browsing there. But even aside from connectivity issues, it actually turns out to be quite a bit faster to use the built-in, cached Wikipedia application as opposed to the website. I mean you can search in real-time with the applications. You just type a couple of characters and tap into your article, rather than firing up Safari or searching for the article in Google; then zooming in so you can tap in, et cetera, et cetera. I and most of the people I know who use the application actually end up using it even when they have internet connectivity. And maybe 20 percent of the time it's pretty useful because it's the only choice.

James Turner: Now just as a point of interest, is this an App Store app or do you have to have a jail-broken phone for it?

Patrick Collison: It was released back when only the jail-broken SDK existed. It was in that initial sort of surge of early applications. I guess the first jail-broken iPhone app, I think, happened in August, and so this was released just under six months later. And then when Apple announced the SDK, I actually originally did not intend to port it to the App store, just because I was just working on other things at the time and my company had just been bought and so it seemed like a lot of work. But then over the summer, I started getting a huge amount of email from people who had upgraded to the new version of the iPhone OS, and were now missing Wikipedia. And I started getting 20 or so emails from people per day saying they love this application and they were really missing it. Or even people saying they were continuing to use the old version of the OS just for this application. And they really hoped that I would port it so they could eventually upgrade. After receiving these emails for a while, I eventually felt too bad about not porting it. So I spent a couple of days porting it and then released it in the App Store. I wrote it and finished the port in August. And then it took about three months to wade through Apple's approval process. Around the end of October, it was released in the App Store.

James Turner: Now most apps that you see in the App Store are relatively small in the couple of megabyte to tens of megabyte ranges. I understand this is about two gigabytes. Does that make it kind of unique or difficult as an App Store?

Patrick Collison: Yeah. I mean when I first went to submit it to the store I had done quite a bit of work getting it down to just marginally under two gigabytes, because two gigabytes was Apple's stated limit. But it actually turned out that Apple's infrastructure and their software was not able to handle two gigabyte applications or anything even close to it. I don't know, but a couple hundred megabytes was the cutoff. That three-month approval process included them having to fix bugs and me having to change how the application worked and all the rest just so I could physically get it into the store. And so the way it actually works today is the application itself is extremely small. I mean just a couple of hundred K. And then you download the application. And then when you first run it, it includes its own sort of embedded downloader thing that allows you to download the Wikipedia from within the application. And it allows you to pause and resume the download and all of the rest. So this actually ended up being the only reliable way of making the download work.

James Turner: And presumably you want to do that on a Wi-Fi network if you don't want to eat up most of your monthly bandwidth from AT&T?

Patrick Collison: Right, right, right. Or I mean I guess if you want, you can really test how honest AT&T are being by saying they'll give you unlimited data.

James Turner: So how big is the uncompressed original source data for the Wikipedia? And how do you cram it down to two gigabytes?

Patrick Collison: So my memory is that it's around 12 or 13 gigabytes uncompressed. The very first thing we do is it comes in this very verbose XML format. We have our own custom format that just includes the bare minimum amount of metadata. And then compacts it in this fairly space efficient binary way. We manage to strip out 20 or 30 percent of the content just by doing that. And then we apply bzip compression, which gets a pretty good compression ratio because, obviously, it's text. Then we also remove some of the content from the applications, the kind of stuff that's not particularly useful on the phone, We strip out the links, for example, to the article in other languages because those links don't work in an offline application. We don't have the other languages. We add links to pictures because we're not storing the pictures. We strip out references because I'm assuming you're not too interested in analyzing the minutia of the references when you're using the phone, and that kind of stuff. And so that, again, saved us another 20 to 30 percent or so.

And really, that's it. I mean what ends up being transferred to the phone is just this huge two gigabyte bzip2 encoded text file that we then index in various ways to allow it to selectively decompress various chunks when a user wants to load an article.

James Turner: So one other question I have is: how about the little things at the top of the article telling you why it's a bad article or things like that? Do you keep that in?

Patrick Collison: Oh, the info boxes? Right. No, we actually don't keep those. And one thing we're considering right now, we're actually working on a fairly substantial update at the moment, is kind of recognizing those -- okay, to backtrack one step, I mean one thing we do not do at the moment is load any templates because it's too computationally expensive on the phone. The way the Wikipedia website works is that you have this article and it has links to all of these other articles which are kind of special articles, you know, templates. And so to load one article might actually load say 20 articles. But loading one individual article on the phone takes about -- well, certainly on the iPhones before the 3GS, takes around five to ten seconds. If you were to do that for ten articles, it would become unusably slow. And so because of that, we don't load any templates. And, therefore, we don't show stuff like the stub message or something like that or the neutrality being contested or that kind of stuff. But what we're working on at the moment is making it, when we're creating that specially prepared dump for the iPhone, that we recognize some set of templates and then include a special flag with the article title or something that notes for the really important info boxes like say neutrality is contested or stub whatever, that we'd then be able to display something on the phone.

James Turner: Wikipedia is obviously a very dynamic thing. It's literally updated second-to-second. How do you deal with that?

Patrick Collison: Right. So, again, this was one of my concerns in the beginning. I mean I was unsure how useful it would be for that exact reason. But actually, it turns out, to be honest, in practice not to matter all that much. I mean the dump I currently have on my phone is actually from August of '08, so it's pretty old. And that causes obvious problems, like if I go to Barack Obama or take a look at something, there's quite a lot that it does not have. But, at the same time, it's pretty rare that I have the experience of wishing that my dump had something that it does not in fact have or something has happened since then. We considered doing updates in the form of deltas, where you could download say 100 megabytes per week to bring your dump totally up-to-date. But that actually ended up not being possible because Wikipedia's own infrastructure can't handle or at least was not able to handle weekly updates. And so they were releasing dumps every couple of months. And so there wasn't really much point in us putting together a really advanced update schedule.

They seem to have kind of improved things over the last couple of months. And so I'm noticing that they are releasing dumps now it seems every week or two. And so we're thinking again about the question of updates and what we could do. But we're hesitant to put too much work into it just because it seems that the usefulness of the application isn't affected that much by whether or not it's six months old or six weeks old. I guess people are pretty used to the idea of any particular reference document not being totally up-to-date. And Wikipedia is really the exception by being up-to-date. But I mean the application for the iPhone, I guess, is conceptually more like a conventional encyclopedia, and that seems to sort of work out okay.

James Turner: How often do you take dumps? Oh, boy. That sounds like the wrong question.

[Laughter]

James Turner: How often do you generate dumps?

Patrick Collison: Right, right. So it varies actually for the different languages. For English, just because it's all so complex, quite infrequently. The latest one available right now is from, I believe, October. And we're actually working on a new one at the moment. For other languages, it's quite a bit more frequent. And I think most of the ones available at the moment are from March or thereabouts. And, like I say, we're hoping to speed that up.

James Turner: You're going to be speaking about this at OSCON in about three weeks. I was curious; is there anything else at OSCON that's really caught your eye or you're interested in?

Patrick Collison: Yeah. So I know that the Cloudera guys are going to be there. And so I'm really looking forward to that. I think they're working on some pretty cool stuff with Hadoop. And so I'm really interested in having a look at that. And then, also, I'm a huge fan of the GitHub and what those guys are doing with kind of changing the dynamic of open source software and interaction and all of the rest. And I'm sort of interested to see -- like for a long time, I feel like open source software generally was held back by SourceForge and its ilk. And sort of the -- well, it's frequently just not a particularly good infrastructure. Like for a long time, we never really moved beyond Source Forge and mailing lists and CVS or, I guess, now subversion. And it's really interesting how Git has, through technology, started to change the sociology or social dynamics of open source software. And GitHub seems to be kind of continuing that with kind of the ecosystem of forks of different projects and allowing those to be later reconciled and all of that kind of stuff. And so I'm pretty excited about that. I'm looking forward to talking to those guys.

James Turner: All right. Well, Patrick Collison, thank you so much for talking to us. And we look forward to seeing you at OSCON.

Patrick Collison: Thank you.

In Defense of Social Media (At Least Some Of It)

2009-07-02T01:55:50Z

Scott Berkun just posted a great rant titled, Calling Bullshit on Social Media. I suggest everyone read it. Berkun raises good points - and I agree the hype around social media warrants taking a critical look. Despite being in general agreement, there are a few areas I can't abide, starting with this statement:

social media is a stupid term. Is there any anti-social media out there? Of course not. All media, by definition, is social in some way.

Railing against the popular lexicon is always a losing bet. Language is formed by collective agreement and it sticks because it resonates and serves a purpose. The words we use to assign to concepts can reveal quite a lot. Rather than dismissing it, we should try and learn from it. I have written before that I believe the term "social" is a new metaphor for understanding how we will transact business and conduct government. As Lakoff and Johnson so aptly pointed out in Metaphors We Live By, metaphors play a crucial role in shaping our very thought and action. We should take the "social" in social media seriously.

Next Berkun writes:

We have always had social networks. Call them families, tribes, clubs, cliques or even towns, cities and nations. You could call throwing a party or telling stories by a fire “social media tools”. If anything has happened recently it’s not the birth of social networks, it’s the popularity of digital tools for social networks, which is something different. These tools may improve how we relate to each other, but at best it will improve upon something we as a species have always done. Never forget social networks are old. The best tools will come from people who recognize, and learn from, the rich 10,000+ year history of social networks.

Well yes and no. The problem is this. Communication is the foundation of economies, government and business. When you scale up communications you change the world. It is that simple. When you radically accelerate or democratize a means of communication (I would include physical transportation in this category too) it is not a change in class (as Berkun argues) it is a change in kind.

By analogy, the railroad did not invent the wheel nor did it invent locomotion or steam power. In fact the train did not create anything particularly new. What it did was massively accelerate the ability to move people and goods across land. That acceleration changed everything… In the U.S. it standardized time, it nationalized commerce. Around the world it broke the lock of power on maritime cities that used to control commerce… and on and on.

Similarly the Internet, and social technologies in particular, do not create much that is new in the way of content (or even human interaction as Berkun notes) but the medium massively accelerates our ability to create, share, connect and collaborate. That acceleration of our innate capacity and desire to be social is exactly what makes social technologies transformative. Where I agree with Berkun's statement above is that the same rules of social etiquette will apply in this media. That is exactly what stuns so many corporations believing they can migrate essentially antisocial behaviors (hack PR blogs, social media gimmick campaigns etc.) into "social" media.

Lastly Berkun writes,

Be suspicious of technologies claimed to change the world. The problem with the world is rarely the lack of technologies, the problem is us. Look, we have trouble following brain dead simple concepts like The Golden Rule.

Agreed. People can really suck. But "change" is a value neutral term. It doesn't imply good or bad and while it is true that many negative human traits will accompany these technologies, it is hard to overstate the magnitude of the changes that are taking place as a direct result of social media - new ways to communicate, stars (including academics finding an audience) born from YouTube, bloggers redefining journalism and science, open source software dethroning traditional players, the demise of established industries like publishing, music and entertainment, with other industries like telecommunications and manufacturing, retailing queuing up for their turn. We see social technologies organizing spontaneous rallies in California, Moldavia and most recently Iran. That is change. I would also argue that the democratic promise of these tools - the promise that people can connect with each other without an intermediary (I know all of the ways that this may not turn out to be the case - but still...) holds the possibility of distributing power more evenly. If there is one root problem in much of this world - it is the concentration of power wielded by a small minority. We should celebrate any technology that lowers barriers to communication.

caveat: Scott Berkun is an O'Reilly Author and in my defense, I owned his book Myths of Innovation long before I joined O'Reilly.

Four short links: 1 July 2009

2009-07-01T20:18:00Z

The Onyas -- New Zealand web design awards launch, from the people behind Webstock and Full Code Press. The name comes from "good on ya", the highest praise that traditionally taciturn New Zealanders are allowed by law to give.
The Year of Business Metrics: Don't make your users run away! -- wrapup of the Velocity conference. AOL: Users who had a slower experience view far fewer pages. Some interesting notes on performance from a Google-Bing study: Notice that as the delays get longer the Time To Click increases at a more extreme rate (1000ms increases by 1900ms). The theory is that the user gets distracted and unengaged in the page. In other words, they've lost the user's full attention and have to get it back. [...] As much as five weeks later, some users, especially those who saw delays greater than 400MS, were still searching less than before. (via timoreilly on Twitter)
Printcasting -- very simple content management system for print magazines that lets anyone start a magazine, add content, sign up contributors, sell ads, and go. Clever!
Pachube Augmented Reality Hack -- sexy hack that pushes all my buttons: computer vision, Arduino, sensor network, ubiquitous computing, pervasive alternate reality cyborg villians with chalk designs hellbent on world domination and the enslavement of the human race to use as meatsack AA batteries for their sex toys. Okay, four out of five ain't bad. (via bruces on Twitter)

Pachube Augmented Reality Demo

Velocity and the Bottom Line

2009-07-01T19:20:56Z

Velocity 2009 took place last week in San Jose, with Jesse Robbins
and I serving as co-chairs. Back in
November 2008, while we were planning Velocity, I said I wanted to highlight "best practices in performance and operations that improve the user experience as well as the company's bottom line." Much of my work focuses on the how of improving performance - tips developers use to create even faster web sites. What's been missing is the why. Why is it important for companies to focus on performance?

That question was answered at Velocity last week by speakers from AOL, Google, Microsoft, and Shopzilla.

Eric Schurman (Bing) and Jake Brutlag (Google Search) co-presented results from latency experiments conducted independently on each site. Bing found that a 2 second slowdown changed queries/user by -1.8% and revenue/user by -4.3%. Google Search found that a 400 millisecond delay resulted in a -0.59% change in searches/user. What's more, even after the delay was removed, these users still had -0.21% fewer searches, indicating that a slower user experience affects long term behavior. (video, slides)
Dave Artz from AOL presented several performance suggestions. He concluded with statistics that show page views drop off as page load times increase. Users in the top decile of page load times view ~7.5 pages/visit. This drops to ~6 pages/visit in the 3rd decile, and bottoms out at ~5 pages/visit for users with the slowest page load times. (slides)
Marissa Mayer shared several performance case studies from Google. One experiment increased the number of search results per page from 10 to 30, with a corresponding increase in page load times from 400 milliseconds to 900 milliseconds. This resulted in a 25% dropoff in first result page searches. Adding the checkout icon (a shopping cart) to search results made the page 2% slower with a corresponding 2% drop in searches/user. (Watch the video to see the clever workaround they found.) Image optimizations in Google Maps made the page 2-3x faster, with significant increase in user interaction with the site. (video, slides)
Phil Dixon, from Shopzilla, had the most takeaway statistics about the impact of performance on the bottom line. A year-long performance redesign resulted in a 5 second speed up (from ~7 seconds to ~2 seconds). This resulted in a 25% increase in page views, a 7-12% increase in revenue, and a 50% reduction in hardware. This last point shows the win-win of performance improvements, increasing revenue while driving down operating costs. (video, slides)

These case studies provide real world numbers that show the benefits of making your site faster. Other Velocity sessions share techniques for implementing performance improvements, including sessions from me, Doug Crockford, and the Facebook and Google frontend teams. But what about the user experience? In his session, Matt Mullenweg (of WordPress fame) makes sure we remember the importance of how the user feels while interacting with our site:

That's why [performance] is important and why we should be obsessed
and not be discouraged when it doesn't change the funnel. My theory here
is when an interface is faster, you feel good. And ultimately what that
comes down to is you feel in control. The web app isn't controlling me,
I'm controlling it. Ultimately that feeling of control translates to
happiness in everyone. In order to increase the happiness in the world,
we all have to keep working on this.

Thanks to the Velocity speakers & their organizations for overcoming the many challenges required to present this data for the first time. We're now equipped with the financial justification, the technical know-how, and the visceral motivation to go out and make the Web a faster place. We'll have more performance success stories next year. Your company could be one of them! Capture your performance improvements and bottom line impact. We'd love to hear from you at Velocity 2010.

Everyblock's Code is Open-Sourced

2009-07-01T16:25:56Z

The code for Adrian Holovaty's Everyblock has been released. The open-sourcing of the site's system were apart of the Knight News Challenge Program. Everyblock is a very impressive site that aggregates and geocodes local data -- news, crime, fire, restaraunt inspections and reviews - and then lets users define their interests down to the block-level.

Adrian made the announcement on 6/30. Here's the list of newly open-sourced, GPL'd goodies found on Everyblock's new Code page:

The main package (probably the thing you're looking for) is the publishing system, known as ebpub.
Second, the packages ebdata and ebgeo contain Python modules for processing data and making maps.
Third, the packages ebinternal and everyblock round out the code that powers EveryBlock.com. They're internal tools and are likely not of general use, but we're including them to be complete.
Finally, ebblog and ebwiki are our blog and wiki software, respectively. Because, dammit, the world needs another Django-powered blogging tool.

Django fans, Python geohackers and anyone who wants to build a local data aggregator are going to be thrilled. Adrian was one of the co-creators of Django and was one of the first Google Maps Mashup creators.

Everyblock has only launched in major US cities. There's plenty of room in the market for locals to create their own version. Everyblock spends a lot of time curating the incoming data feeds so I doubt that anyone will be able to roll out new sites too quickly. One thing to note: the trademark Everyblock is not available. However, the Everyblock team would not mind being acknowledged if you use their code. Personally I get a lot of value of Everyblock in my city. I get a daily email with all the crime, news and errata near my house.

Everyblock is now going to move onto the second stage of its existence. About five months Adrian blogged about the dilemna they would be facing when they open-sourced their software. As he said at the time:

But now we've reached an interesting point in our project's growth: our grant ends on June 30, and, under the terms of our grant, we're open-sourcing the EveryBlock publishing system so that anybody will be able to take the code to create similar sites. That's a Good Thing, in that EveryBlock's philosophies and tools will have the opportunity to spread around the world much faster than we could have done on our own, but it puts the six of us EveryBlockers in an odd spot. How do we sustain our project if our code is free to the world?

At the time I suggested that they try to federate with new everyblocks. After yesterday's announcement I mailed Adrian to ask him for a hint about their future plans, but for now he's keeping mum.

The Hacker Ethic - Harming Developers?

2009-07-01T12:00:00Z

On Monday Neil McAllister posed the question "is the hacker ethic harming American developers?" Slashdot picked it up and Tim forwarded it to the Radar list. As you might expect, it resulted in some spirited discussion.

James Turner kicked things off with this response (it has been slightly edited from its email form). After James lays out his argument I'll reply with my thoughts. Then we hope to hear from you. Let us know what you think.

I've worked in a lot of organizations that thought that the kind of rigid deforesting paradigms that Nayar is referring to were the magic bullet to keeping all three of the variable (dollars, time, features) under control. Without exception, all they did was get in the way and reduce the productivity of the most senior people to the level of the most junior. All of them exhibited some degree of failure, some catastrophic.
The India shops *love* methodologies like UML and the like, specifically because the problems have been reduced to a simplistic enough granularity that they can be doled out to junior-level staff, who may have only been onboard a few weeks because of the massive churn over there.

At least 3 times at 3 different companies, I've seen major pieces of work brought back in-house because the Bangalore team had fallen so far behind or proved so unable to get beyond the literal description of work that they were endangering the project. When you combine the time difference with a tendency to halt dead in their tracks as soon as they hit a stumbling block, it can be a recipe for disaster.

There are certainly some good Indian shops, and I know some outstanding Indian developers (most of whom have come to the States.) But I find Nayar's comments hilarious. It's akin to someone saying that American football players aren't employable in Jamaica because they aren't able to limbo well. Look at the most successful Web 2.0 companies today, most of them started as garage enterprises with a few talented developers, not a 60 person team of UML jockies following some Arthur Anderson project management program. Heck, look at Google Labs.

In huge projects, you obviously need some master planning and coordination to make sure the tracks meet at the right place to drive in the spike, but I don't see any effort being made these days to right-size the amount of project overhead to the needs of the projects. Instead we get a one-size-fits-all approach that smothers anything but the largest project in paperwork. Even some of the original authors of the Agile manifesto, when I've talked to them, point out that part of being Agile is picking and choosing the right components of project management that make sense for a given task.

Nayar's remarks are incredibly self-serving. "We're the best, because we can mindlessly follow some arbitrary and flawed development process." Or is he claiming that Indian projects do better QA? Not in my experience...

This entire debacle is representative of a problem I think is endemic in the industry these days, the inability or unwillingness to engage in rapid prototyping. Every successful project I've ever worked on (and I've worked on some fairly large enterprise-sized projects), we started by designing and coding a quick "throw-away" skeleton of the application, that let us look at how the thing worked, where the unseen warts were, and where the vendors had lied about their APIs, etc. This is the crucial and neglected stage in project design, one that most modern design paradigms ignore or actively discourage. Even Agile tries to jump in and start coding the finished application from the get go, although if project teams were willing to aggressively refactor (a tenant of Agile), early project work could be a rapid prototype (although the story model of scrum really doesn't fit well with this, unless you make the prototype a story...)

This is also something I've never seen an offshored team do particularly well...

Well, I'll be damned if I'm going to jump in and take the side that says hacking is bad for American programmers. First off, I don't need that kind of flame bait and second, I don't believe it. I think approachable programming is hugely important because that's how many people get into the field in the first place. However, my reaction to the article was very different than James' and I might as well try to explain.

I'm not going to take an opposing position, but it's not really an orthogonal position either. Maybe it has a power factor of about .7 or so. Here's my (also edited) response...

When I when I read McAllister's piece, at least some of it resonated with me. Before we were bought by a large firm, we were a small company that grew from nothing to 250 people, about 200 or more of them were programmers. So, a whole lot of my time over three years was spent hiring programmers and building cohesive teams that could deliver to our customers.
In our hiring we aggressively hired hackers into the mix. We wanted outside-our-industry thinking and we thought they brought in creativity. We called it "hiring weird, but not weird weird." Occasionally we pushed it to weird and a half. For our efforts we got creative problem solving and interesting (but frequently weird) dinnertime conversation when we travelled.

However, our pollyanna idea of "disciplined teams catalyzed with a bit of weird" didn't always work out.

That leads to the bit that resonated with me: the sense that hacker = distilled essence of American individualism combined with lots of ISTJ Myer's Brigg's Type Indicator. Individualism is a trait that I hold dearly, but it can make a cohesive team effort difficult if people are unwilling to suborn themselves to the goals of the team. Remember those tee shirts the football team always wore at your university? "There is no I in team?" I sometimes joked that I was going to make a batch that said "I'm the Me in team."

Maybe we were just growing fast and it was going to take more storming and norming than I had patience for, but at times it was a struggle to get everyone to see past their individual biases and focus on what we were trying to achieve, and we couldn't do what we were trying to do with teams of one.

But it really wasn't a hacker problem if hacker means self taught like McAllister implies. We had a lot of people with CS degrees and we used to talk a lot about whether and how their degrees had prepared them for their jobs.

Separate from the individualist approach to development, few of our recent graduates came to us prepared with the terminology and practices of any development approach (or engineering approaches like continuous integration etc.). They knew how to code, but not how teams coded.

At one point I gave a talk on agile software development to about 100 CS students at a university in Philadelphia and I asked them to raise their hands if they had ever done a team project with greater than two people on the team. I don't recall anyone raising a hand. Then I asked if they had ever covered development methodologies in their classes and a few acknowledged they had, but it had been abstract classroom stuff only. That part surprised me.

I'm not sure that the "sanctity of engineering" argument really makes that much difference. I have little faith in McAllister's scheme to do computer engineering instead of computer science coursework.

My undergraduate degree was in Mechanical Engineering and I can only imagine how useless I would have been to a firm that actually did engineering, and for mostly the same reasons. I knew how to take integrals and I still know the packing ratio of a hexagonal close packed material but I didn't know squat about how a complex machine actually got designed in a team setting. It's interesting to note that the Engineer in Training exam I took (a precursor to the professional engineer's exam) didn't probe my knowledge of team practices at all.

Maybe there just isn't time in an undergraduate degree to teach everything that an engineer needs to know. Plus, can you imagine the drop out rate in CS/CE if ITIL was a required course?

Since James mentioned Google I'll switch gears and muse about ecosystems for a moment... I guess I tend to bristle when I hear that everyone should just develop software the way Google does.

Google is to computing what LA was to Aerospace and Electronics in the early 60's. It's gravitational force attracts five sigma talent (probably a bunch of six too) in ways that the rest of us can only envy. More generally, Silicon Valley has had programmer talent flowing into it for the last twenty years the way Hollywood sucks pretty people out of the midwest.

Maybe it's not as obvious because you can't spot brains the way you can spot an oddly beautiful wait staff, but the valley has been the vortex of a talent-laden embarrassment of riches for a long time and, if you work there, you might not even notice it. However, I think that at some level this effects what kinds of processes work when you build software. I think it's at least a little part of the reason why an ERP system in a manufacturing town gets implemented differently than MapReduce (there are other reasons too having to do with software as product vs software as supporting infrastructure). Combine that with the very clear shared vision of "lets do something great and get rich together" thing that valley firms often have, and well, it's easy to see how smart people coalesce to build amazing stuff.

It's easy to denigrate Arthur Anderson's progeny or the offshore firms they compete with, but they do different work, with a different talent pool, for different ends, and with a very different set of personal and organizational incentives. Or, put another way, Kelly Johnson didn't build the SR-71 with General Motor's engineers, and General Motors didn't design the Chevy Cavalier with the Skunkworks' processes. However, even at the Skunkworks, Johnson's brilliant engineers did conform to a process and work together as a team toward a shared vision. And, conversely, I bet a lot of talent is left on the table at General Motors because of processes too restrictive in their attempt to remove all uncertainty.

So,... maybe it's possible that Google's (or the valley's in general) processes are appropriate to an ecosystem that, because of the intellectual environment and potential for riches, is rich in IQ and initiative. So it ends up feeling more "special forces" and less like "infantry regiment." And over there closer to the hump in the normal distribution curve, or in a different cultural environment outside of the valley, a different flavor of processes may be effective?

The counter argument to that, which I'll go ahead and provide, is that I once helped teach a team of engineers at midwestern defense contractor how to do agile development. The effect was amazing and immediate and their productivity and satisfaction went up tremendously; until their management freaked out and shut it down when they "perceived" that it created too much uncertainty in their processes.

Well, it's obvious that I don't know the answers here, so, with that, I'll stop thinking out loud.

What do you think?

The US Online Job Market Was (still) Down Big In June 2009

2009-07-01T10:00:00Z

Updating my post from early June, the U.S. online job market^† still hasn't shown signs of recovering from steady declines that began in September of last year. Compared to the same period last year, there were 50% fewer job postings in June 2009.

An alternate view highlights the start of the downward trend, as well as the smaller than expected seasonal bounce from Dec-08 to Jan/Feb 2009. In a normal year, the number of postings decline in December (as employers table job searches for after the holidays) and recovers sharply the following Jan/Feb. While job postings did bounce back in Jan/Feb 2009, the seasonal bump was less than half of what occurred in previous years.

No geographic region has been exempt from the downturn in online job postings. There have been sharp declines in all states, ranging from -59% in DE, WY, and MN, to -38% in MD, OK, VA.

In closing, we still haven't detected the green shoots that some forecasters have been crowing about over the last few months. If one were to take an optimistic perspective, the worse year-over-year decline occurred in April. OTOH, we are still staring at a 50% decline in June 2009. So while we may have hit the bottom in April, we need a few strong(er) months before we can comfortably announce the arrival of green shoots.

(†) In partnership with SimplyHired and Greenplum, we maintain a data warehouse that contains most U.S. online job postings dating back to mid-2005. Data for this post was through 6/28/2009.

Radical Transparency: The New Federal IT Dashboard

2009-06-30T19:15:35Z

Today, at the Personal Democracy Forum in New York, Vivek Kundra, the US national CIO, unveiled the new IT spending dashboards at usaspending.gov. The dashboards are designed to help Vivek and the CIOs of individual government agencies get a handle on the effectiveness of government IT spending.

At the top level, the dashboards provide a view of spending by major government department, with graphs showing performance against schedule, costs, and the CIO's assessment of how well they are meeting their objectives. For example, here's a stark view of IT performance at the Veteran's Administration (click to expand image):

49% of the VA's IT projects are behind schedule, and Roger Baker, the agency CIO, deems that a full 63% of the projects are in need of serious attention. (Here's a recent article that outlines Baker's tough-love plans for IT at the Veteran's Administration.)

As you drill down, you get to dashboards for individual IT projects (800 projects and approximately $20 billion in budgeted spending). Each project shows the responsible government official, the prime contractors on the project, the CIO's evaluation of its progress against goals, and each month, an update showing an update of that progress. (We'll show one of these later in this article.)

The dashboards are an incredibly ambitious undertaking. In the first place, there has never been a government-wide view like this of all IT spending, and the progress of projects. What's even more remarkable, though, is that the dashboards are being shared with the public. It's a bit like having your performance review posted on the company bulletin board for all to see.

In notes provided to press in advance of the announcement, Vivek Kundra wrote (italics mine):

Over the past several years, we have witnessed numerous public failures of major information technology systems and just last year saw roughly one third of all investments reported as poorly planned or poorly performing. Many of these investments may never deliver on their original promises. With over $75 Billion in annual federal information technology spending, we need a new foundation for management - one built on the values of transparency, accountability, and responsibility....

Data is powerful. It enables monitoring, reporting, and meaningful analysis that leads to better decisions. Yet, in the case of federal information technology, we lack insight into project performance. Poor data quality coupled with infrequent reporting has led to lack of meaningful analysis and bad decisions. Numerous failures and cost overruns may have been avoided with timely access to accurate information.

The Administration is committed to using technology to move past these barriers. In the IT Dashboard, the public has a platform for unprecedented access to useful, unfiltered data regarding the performance of IT investments. Information available includes responsible government officials and contractors as well as project performance data, updated monthly. This enables better decision-making, giving us the ability to turn around poorly performing projects and to divest from those which no longer make sense.

In making this data publicly available, we are providing unfettered access to investment performance to its true owners - the American people.

Vivek explained that last point further in a telephone conversation with me last night. I asked him about the level of buy-in across the government for this kind of radical transparency about the performance of projects. He said:

"It's a cultural transformation, in terms of recognizing that we are in the public square. The work that we do is work that is supposed to be performed in the interest of the American taxpayers. And so making visible how we're performing means fleshing out these complicated issues in the public square. Culturally, making the shift is much better than letting it hide under the veil of secrecy.

Paging through the site, it's clear that not every CIO is on board, or else has not had the time to make the project evaluations. For example, in this project page for the Expeditionary Combat Support System, you can see that the Department of Defense CIO has not yet provided his own evaluation, and even the name of the prime contractor is missing:

According to the Washington Post, some agency CIOs have grumbled about the workload of maintaining the data, but as Vivek noted in that same article:

I talked to the CIO Council and saw the data change overnight. It was cleaned up immediately when people realized it was going to be made public.

That's part of the new media chess game being played out here.
Having the dashboard public creates an urgency to bring the information it contains up to date, and presumably, to respond to what it shows.
In last night's phone interview, Vivek noted:

We're going live in beta, so that we're giving agencies enough time to make sure that the data and evaluations are up to date. They will have up to 30 days to make that journey.

I love that. 30 days is an eye-blink in traditional government time! We're starting to see agile methodologies make their government debut.

The site was built on Drupal. Graphics are in Flash, provided by FusionCharts. Based on a reading of the FAQ, it looks like the initial database was generated from data provided by agencies to the Office of Management and Budget (OMB) on congressionally-mandated reporting forms. Monthly updates will be via a web interface provided as part of the project. There are future plans to allow agencies to provide XML feeds.
(It would be good to see that the site uses the same web services architecture to collect its data as it does to provide it out to citizens.)

This is part of being agile, and one of the first lessons in building an "architecture of participation": rather than having to get complete buy-in from everyone up front, you find a way to bootstrap the project, to get it to what Eric Raymond once called a plausible promise that encourages further participation. What you hope for is the kind of virtuous cascade that gave us successful open source projects like Linux and Apache, and participatory online services like Wikipedia, Craigslist, YouTube, Facebook, Twitter, and, of course, the web itself.

The dashboard platform is primed for citizen participation, not just agency participation. Each page has a "share" icon, which brings up a dialog that lets you tweet a link to that page, share it on Facebook or del.icio.us, or grab the data as an RSS feed or an excel spreadsheet:

It's mashup-ready, with a tool for selecting which fields you want in an XML feed. Unlike the main usaspending.gov site, whose API offers 1000 records at a time and discourages repeated calls to download the whole database, the data behind the IT Dashboard is easily downloaded in its entirety. Kudos for that.

Vivek noted:

This is a fundamental change in the way government is going to be run. In the same way that the President was able to mobilize and organize online communities to drive change, what I want to accomplish here is to tap into that energy, to tap into some of the smartest people in the country, accept the fact that we don't have a monopoly on the best ideas, and make sure that we unleash the online communities of people who are interested in specific areas. Whether it's environment, or healthcare, or energy, we want them to look at the investments that we're making. And we want to make sure that we're getting feedback. A lot of times, the government is not the beneficiary of the best thinking in the country, because something was decided [ten years ago] and we're not asking, what is the innovative path? How can we achieve our objectives in a much faster way, and at a much lower cost?

In this regard, the view of the management objectives by which each project is evaluated is important. For example, the objectives of this behind-schedule biosense project may already be met by non-governmental projects like INSTEDD or the Global Virus Forecasting Initiative - or at the very least, knowledge of this project might allow better cooperation between the governmental and non-governmental projects.

As far as I can tell, feedback right now is limited to a simple comment form. The site would benefit from more structured opportunities for interaction and requests for further information. In particular, we need a better and more detailed view of project objectives for the public to engage properly in oversight and brainstorming. Right now, the project measures what is relatively easy to measure. The next big challenge comes in in saying "what we really need to know in order to be able to decide that the money was misspent is ..." and then changing the reporting law or requirements to get that data.

Overall, the data dashboard is a small but important step. Even though the Federal IT portfolio, at $70 billion, is huge, it's a tiny fraction of overall government spending. But if Vivek Kundra and his team can prove the benefit of radical transparency in this sphere, we can hope that it will spread to other areas of government. We need to see public involvement, so those government agency CIOs who aren't already on board with the program are encouraged to do so by the flow of attention. We also need to see better overall metrics. What contractors are most often behind schedule and over budget? What departments are struggling? Are we doing better over time as a result of the increased scrutiny?

Dashboards are only as good as the people who use them. Ultimately, the goal of instrumenting government spending and its effectiveness depends on the willingness of those in positions of authority to respond to what the dashboards tell them. Vivek Kundra is rolling out a suite of IT management tools - a platform for better government, if you will - of which this is only one installment. Let's hope that as the dashboards are updated each month, we'll see evidence of change, with the data more complete, and with out-of-control projects reined in, improved, or ended.

Four short links: 30 June 2009

2009-06-30T11:00:00Z

Military Open Source Software Conference -- 12-13 August 2009 in Atlanta.
Govloop -- a "Social Network for Gov 2.0". Gov 2.0 could easily become the intersection of talk radio and social media consultant inanity. As with the Web 2.0 lunacy, when everyone who could spell wiki tried to sell one, you should cultivate the art of identifying and sidestepping the bozos, the time-wasters, and the charlatans who use buzzwords as a convenient alternative to thought. (via cheeky_geeky on Twitter)
Introducing the Autom -- a personal robot to help you lose weight. Developed by Initiative Automata as an offshoot from MIT researcher Cory Kidd, Autom has conversations that encourage you to record your diet and exercise. The theory is that the added benefit of interaction will help you stick with the diet longer, increasing the chance that it will stick. Trials showed Autom users stick with their "weight loss regimen" twice as long as pencil-and-paper. (via So, Where's My Robot?)
USA Government IT Dashboard Launches -- Vivek Kundra's latest project, a dashboard giving insight into government spending. Contractors, CIOs, projects, schedules, and data via an API. Built in Drupal!

Bing's Sanaz Ahari on System Feedback (2 of 2)

2009-06-30T10:19:45Z

A couple of weeks ago Bing had a small search summit for analysts, bloggers, SEO experts, entrepreneurs and advertisers. It was held in Bellevue; they put us up in the hotel and fed us. While there we received demos from Bing project teams. I was able to snag an interview with Sanaz Ahari, Lead PM on Bing. She led the team that developed the categories you see on a Bing web search. The interview was based on the slides from her presentation at the event. I have posted the significant images from her slides. The first portion of the interview focuses on how the Bing team handles Query level categorization and some of the problems they face. The second portion focuses on the systems used to generate the categorization.

Disclosure: I was on the MSN Search team (now the Bing team) from 2004- March, 2006. I knew Sanaz at that time.

Brady Forrest: Now on this image, it shows the ranking model and then it shows engagement and measurement.

Sanaz Ahari: Yes.

Brady Forrest: How does engagement and analytics factor into tweaking the ranking, measurement and engagement algorithms?

Sanaz Ahari: So the key thing about engagement is really there's two things: A, how often do people click on the different categories and then B, once they click on it, what do they do after that? So we basically feed that back into figuring out, "Okay. Did we actually put up the right thing? If something lower down is getting clicked on more, does it deserve to be higher? If something is not getting enough engagement, does it need to be bumped down?" And as we really expand the system, I'd have to say for us as a team, this is really the first step towards what we want to do. And, ideally, we want to get to the point for where we have enough understanding about every single query that we can really help you refine your tasks and your categories. So the engagement model can also help us in the future as we go in deeper into queries for helping people. We shouldn't just say, "Seattle, I'm going to Seattle restaurants." You should be able to go to Seattle restaurants and go in really deep and say, "I want restaurants in this neighborhood. I want of this price range, et cetera." So all of the engagement metrics can actually help us figure out what are the follow on tasks that users engage in the most as well.

Brady Forrest: And so what is the second flow chart?

Sanaz Ahari: So the second area, so once we felt that we could deliver intense understanding at a level of quality that we felt comfortable with, then we tackled the second area of problems which is equally difficult, which is really around, okay, how do we know that J Lo is a musician in the first place. And this is really around the query understanding aspect of things. And this is an area where we, again, explored multiple different approaches. We could've done a very kind of clustering on the entire corpus of our quarries. Or we could've said, "We're going to start a little bit more targeted and only go after the domains that we really want to go after." Like we said, "Let's just go after health and see if we can solve a small problem before trying to take on the entire corpus of the web."

For the Bing release, we focused -- and this was just like a principle that we had of the team was we really wanted to start small and see if we can get the level of quality that we wanted before trying to take on a lot more different challenges. And so, in this case, we definitely -- we went after the types of domains that we knew were strategic for us. So, all of the sudden, our corpus of quarries that we were interested in was a lot smaller. And we already have abilities to classify quarries into domains and understand, okay, this query is a music query or this query is health query, et cetera, et cetera. And so the other problems that fall out of that is, okay, when people do do health quarries, what are the categories that fall out of that? Like how do we know that people are going to care about diseases and symptoms, et cetera, et cetera. And then the next problem after that is how do we know that we have a comprehensive understanding of all diseases? So we may be able to understand that there are N different diseases, but how do we know that that’s actually a comprehensive list?

And then lastly, there's a problem about -- and this is one of the fascinating search problems -- is users query for the same thing in many, many different ways. So an example that I had was, for example, health is actually a very complicated one where the ALS disease is also known as Lou Gehrig's disease. And it's also known as one other thing which sounds kind of complicated. I don't even know how to say it. But there's lots of different ways that people basically query for the same thing. And so those were the three different problems that we really had to tackle in the query understanding space. So the two areas that we basically looked at was A, if we are able to identify a C set of quarries in a category, how can we actually really expand that out and be able to understand that we have a comprehensive list expander? Like if we start with N items, are we able to expand it out and get a more comprehensive list of items that are very similar to an existing C set that we started out with. And that's really the query expansion problem.

 Brady Forrest: And what type of numbers are you talking about? Is it 100 or 1,000 or 100,000?

Sanaz Ahari: Oh, for the C sets? It completely varies. It completely varies. There are some categories that are small. There are some that are large. Like if you try to tackle musicians as a whole, that's huge. Whereas if you try to tackle like sports teams or something, that's pretty small. So it varies.

Brady Forrest: And are you pulling category names? Like are you pulling Wikipedia? Like proper nouns in the case of musicians or are you also pulling raw queries from the logs?

Sanaz Ahari: There's definitely both. We use a whole bunch of different features. We do a lot of work from logs. We do a lot of work on document extraction as well. What's very interesting is logs can give you a lot of great information where we have enough information. So it doesn't necessarily help you address the tail with precision. And document extraction can potentially help you with more comprehensiveness. And one of the things I would say is we also realize the good thing about approach on a whole actually, both on the intense extraction side and on query understanding side has been that it was an amazing learning experience for the team to tackle the problems one at a time because we realized there were so many intricacies that there are some things where we can build a generic system and it can help every category. But there were also cases where we would find a lot of intricacies in some categories where we had to do --

Brady Forrest: So what's a query that you're proud of that was like really hard and you feel like -- like an example of a query that really came a long way?

Sanaz Ahari: I actually don't have one at the tip of my -- I do like the experience for Jennifer Lopez because she has a lot of different attributes.

Brady Forrest: What's one that you really want to improve but you didn't want to tweak by hand?

Sanaz Ahari: Actually, the Jaguar one was one (Bing search), the one this morning that we talked about. That was a great query. And in some ways, I actually think we do a lot of positive things with that query. Like in one sense, I would say that we definitely deliver a diversified experience. And we at least capture the different intents. Whereas without the left rail altogether, you get the -- most users don't really go past the third algorithm result. And that in and of itself doesn't really give users enough diverse to creations [word] to say, "Okay. This is really my intent. And this is what I really want to dig down to." So on one hand, I like what we have done. But in the ideal scenario, I envision us being able to enumerate all of the different intents and all of the different tasks that actually fall under every single intent. So ideally, we should be able to call out animal, team, car, et cetera and then call out the individual tasks that the users want to do beneath every single one of them. There is -- the two areas that I really, really want us to improve is one, around that. I think that disambiguation is a pretty hard problem where we've barely scratched the surface. And then the second area is the depths of our coverage. You know, I really want us to have a much deeper experience where if I type in Indian restaurants in Fremont (Bing search), I should be able to still get a categorized experience where I can still dig in deeper.

Brady Forrest: What percentage of queries categorize the experience?

Sanaz Ahari: So today, 20 percent of our queries have a categorized experience. And the team is actively working on our next release where we are working on increasing both the quality and the coverage and specifically going more into longer queries.

Brady Forrest: Okay. Well, thank you very much, Sanaz.

Sanaz Ahari: Thank you.

Personal Democracy Forum conference: initial themes

2009-06-29T23:20:52Z

"So what's this conference you're going to?" asked my friends, not
braced for an explanation that usually took me more than ten minutes.
Ultimately, though, they all ended up excited about the ideas behind

Personal Democracy Forum.

These friends care about politics. They take sides on and argue over
the issues, and at some level they notice the processes. And although
some know what an API was and a few even understood the concept of
mash-ups, it's remarkable how completely they had been bypassed by the
current movement toward open government, whose importance to the Obama
administration was signaled by his release of a

memorandum on transparency and open government
on his first full day in office.

I hooked my friends through the idea of an irreversible political
shift. Not a regulatory regime that could be dismantled like the
agencies responsible for civil rights, or a mandate that could be
defunded like federal housing initiatives--no, in this case a movement
integrating the public into government functioning, and that therefore
creates an external constituency that helps to perpetuate the system;
an ecosystem of non-governmental organizations that will react
precipitously and aggressively if the government tries to shut them
out.

Digging for themes

PDF is appropriately held in New York City, a culturally open
megalopolis that is ethnically and politically uncategorizeable. Free
speech holds forth on the subways where the exhortations of the
homeless prove that the great art of oratory is still alive.

A thousand people signed up for the conference (leading, of course, to
more than a thousand Twitterers). At the gorgeous Jazz at Lincoln
Center location, the Rose auditorium was totally filled, and the
hallway was choked as attendees strove to reach pitifully undersized
rooms for breakout sessions.

As a conference with a contemporary, tech-oriented bent, PDF ripples
off into all kinds of online resources. At several points the keynotes
were held against a

real-time twitter feed,
goading on the feeding frenzy by showing the accounts of the people
who tweeted the most. This focus on immediate response--and on
quantity of response--had a specific effect on the consciousness of
the audience. The twitter feed reinforced through highlighting and
repetition the most provocative sound bites and the statements most
clearly relating to current issues at the top of attendees' minds

This is a useful function to play, but the provocative utterance and
timely issue is only one superficial level of conference
engagement. We all need to take away what we've experienced, sit with
it a bit, and look for underlying themes that represent a significant
trends that can guide us.

Give a few hours for reflection, I'll use this blog to synthesize
three recurring themes I heard during the first day. I'm sure more
ideas will settle out as I spend even more time thinking through these
two days of meetings.

The prerequisite: the power for change lies with the public

The platform for democracy: infrastructure we all need

Time to tune in: we can't tolerate static

Miscellaneous insights from speakers and participants

The prerequisite: the power for change lies with the public

It's scary being a politician, let alone the an agency head. These
people may seem indescribably powerful to the rest of us, but they
live in fear of public pillory triggered by their own missteps.

Jeff Jarvis listed, as one of his four key elements of change, the
ability for government to fail without risk of recrimination. David
Weinberger approached the same theme from a different direction,
talking about how all wisdom is provisional, emerging, and
scattered. Vivek Kundra and Beth Noveck--who will be speaking
tomorrow--have repeatedly made similar statements in the context of
bringing the innovation culture of the Silicon Valley to the area
around Foggy Bottom.

In my

first ramp-up blog for PDF
I talked about a four-part cycle for successful public/government
collaboration.
Perhaps we need to start the cycle earlier, or add some kind of
parallel cycle, to recognize that the public has to make the
commitment asked by Jarvis: the promise to show forbearance when the
government fails and to grant it a mandate to do innovation.

The platform for democracy: infrastructure we all need

If one engages in some deep listening, you can hear beneath all the
celebrations of transparency a recognition that success depends on
several elements of infrastructure. Early experiments in open
government may produce exemplary and even spectacular successes, but
the culture won't take hold until this platform is in place.

Computing networking and computer technology are the most obvious
requirement. Mark McKinnon, a Republican communications strategist,
called for universal broadband during his keynote.

But as audience members pointed out, literacy is another requirement:
basic literacy as well as media-savvy literacy and knowledge of the
tools that let one participate.

Ethnologist dana boyd took the discussion to the next level by pointing
out that even when people do go online and do use social media, they
self-segregate by race, class, and educational status. Her case study
for this claim was limited (the demographics of MySpace users versus
Facebook users) but the statements she culled from young people showed
that the digital divide is possibly even deeper online than these
social divisions are offline.

I believe that a predilection for different forums and ways of
interacting online doesn't have to prevent different races and classes
from coming together on issues of common interest, such as health
care. But boyd's point that people set up online barriers that make it
harder for them communicate across these barriers is salient. She
pointed out that we need to recognize that the sites we visit are not
the same sites everyone visits, to spend time on the sites of people
we want to influence or collaborate with, and to embrace different
modes of interaction among different social groups.

Finally, open discussion requires a tolerant environment. Recent events in Iran, as well as the introduction of Internet filtering software in China, show that governments can choke off civil society online; the technology was described as a cat-and-mouse game where both the side of information dissemination and the side of repression learn how to increase their power.

Time to tune in: we can't tolerate static

The last theme I'll highlight from the first day is the sense that we
can't stand still. Americans (and particularly young Americans) expect
more and more that we can have a say, that we can move quickly and
have choices, that we can contribute to decisions and their
implementation. We've already seen how many businesses (not all, of
course) that fail to keep pace with these expectations are shrinking.
If governments don't meet the expectations, people won't be able to
replace it the way they replace businesses, but there could be
increased feelings of alienation and increased social dissatisfaction.

Miscellaneous insights from speakers and participants

ChallengePost
announced today a site that brings together people with needs and
problem-solvers, using a challenge model similar to the
Netflix prize or the
TopCoder
software firm. In publishing a challenge, someone can offer money or
just recruit people to offer thanks. Respondents may be motivated to
solve the challenge by intangible rewards as well as
money. ChallengePost offers advice on how formulate a good challenge
and judge it expertly, but the form of each challenge is the
prerogative of those who post it.

The

Digital Literacy Contest
tries to develop a generation of problem-solvers who can analyze the
streams of government data coming online. They will run contests in
high schools and colleges that start with test problems and then move
to questions to which they do not have the answers. When several
students converge on the same solution, it is published for the public
benefit.

Morley Winograd of
NDN
briefly analyzed Ron Paul's failure in the presidential election
despite his sophisticated use of social media. If I understood
Winograd, the medium--which is well constituted for bringing groups
together--contrasted too much with the message or individualistic
libertarianism.

In a forum on participatory medicine, Esther Dyson said of the current
health care debate, "We're focusing too much on health care and not
enough on health, just as one might complain that the government
focuses too much on laws and not enough on getting people to do good
things." This was the start of a session that discussed ways patients
and doctors could use information sharing to improve outcomes and
lower costs.

New York Mayor Michael Bloomberg called in over Skype instead of coming to the conference. Over his call he announced an expansion of the famous 311 service and various initiatives to accept public complaints and provide public data online. I was glad Skype was available for the call, but I find it odd for the government to be using commercial services (Kundra moving staff to Google Docs, YouTube hosting White House videos, agencies going on Facebook, etc.). I can see why the government wants to use available social media for convenience, and it provides a familiar access method for constituents. But eventually governments should develop their own public-domain software, tailored to government needs and open to all.

Blair Levin, who is designing a national broadband plan at the FCC, started out buttering up the audience by making fun of incumbent telephone companies, then gave us a "homework assignment" of reviewing and making improvements to its presentation at the the July 2nd FCC meeting, material for a set of staff workshops in August, and plans to be make in the Fall to do research. A panel following Levin's presentation--matching up a much-applauded representative from Free Press with representatives from the cable and telco industries--looked at the issue of speed. Is it fair to set a single target for speeds? Will the FCC define broadband to more closely match more advanced
countries?