Problems in the middle

So I'm on this consulting gig which involves writing a website strategy for a large-ish organization. During a presentation last week one of the project stakeholders pipes up and says "we need an over-arching vision." (Cue giant sucking sound.)

Anyway, my response was basically "sure, but the problems you have are in the short- and middle-term... and you don't need a big vision to solve them." Which reminded of something Karl Fast posted to AIfIA-members a few months ago about Google, information retrieval and information architecture. I've posted the a longer excerpt below, but this is essence of Karl's post:

We tend to think that the hard problems are the big ones. So we believe that searching the Web is hard because it's so huge. But I've been thinking lately that the really hard problems are actually the ones in the middle. In the middle, many algorithms don't work that well with moderate document sets, context becomes much more important, interaction is critical, and you can't get the user "in the ballpark" anymore--you have to get them to right to the thing they're looking for.

I think Karl's nailed this. Middle problems (of the information architecture/interaction design sort) create a healthy creative tension between users, context, design, technology and business goals which can't be resolved by one discipline alone. And there's less chance that user needs or optimal IA and design choices will be sacrificed to an untameable multi-headed enterprise software product.

Of course I see the value in standards, scalability, interoperability, long-term strategy and, yes, even an over-arching vision. But I'm finding more value in solving those middle problems really well. Sometimes big picture thinking is a distraction.

Here's a longer excerpt from Karl's note (which was in response to another post):

You partly disagreed with my point about structure, noting that "engines like Google are still applying patterns and structure.

Indeed they are and I wouldn't mean to suggest otherwise. Google is all about structure. But Google's relationship towards the structure of a document space is quite different from classic LIS.

Our standard meme is "order out of chaos." Classic LIS believes that to achieve order and facilitate retrieval a minimum amount of structure must be *imposed* on the document space. The Web falls well below the necessary baseline that classic LIS would say is necessary.

What Google does is *derive* higher orders of structure from a document space that is a chaotic mess when viewed from the perspective of classic LIS principles. So when librarians attempted to catalogue the web based on AACR2 it was a massive failure. It was too big, too chaotic, and too dynamic. Your only option is to derive structure.

And that is what Google did. Their insight was to embrace the structure of the Web and figure out how it could be exploited to facilitate retrieval. Where LIS said "how can we make the Web like our classic systems?" Google said "we can't change the Web, but maybe we can find some useful properties and exploit them to build a better retrieval system."

I'm not arguing against structure. I'm simply pointing out that the question of structure is actually several questions: What form of structure? How much structure? How does it get created? And, how can we use this structure to facilitate retrieval?

Google certainly relies on structure, but it accepts the structure of the Web and then derives higher orders of structure. The classic LIS approach has been to create/impose a certain type and minimum level of structure first.

Now, you finished up by noting that:

I'd be lost without Google but there is always a place for more structure in helping to find things.

I agree 100%.

We should note Google is useful at working with structure on a global scale and getting you into the right ballpark. It doesn't do so well at local structure.

This is where IA comes in. In small and medium-scale systems we don't have good methods for deriving structure. This is where the classic LIS techniques have huge value.

We tend to think that the hard problems are the big ones. So we believe that searching the Web is hard because it's so huge. But I've been thinking lately that the really hard problems are actually the ones in the middle. In the middle, many algorithms don't work that well with moderate document sets, context becomes much more important, interaction is critical, and you can't get the user "in the ballpark" anymore--you have to get them to right to the thing they're looking for.

IA is about solving the problems in the middle.

And there is plenty of room in the middle (with apologies to Richard Feynman).