redesigning evidence based practice for wicked problem solving

conference presentations, ebl, evidence No Responses »
Jul 052011

I spent the better part of last week at the Sixth International Evidence Based Library and Information Practice Conference (EBLIP6) in Manchester, where I presented a paper co-authored with the wonderful Zaana Howard. Our paper was a conceptual one – it presented an untested model for evidence based practice (EBP), which is essentially a hybrid of design thinking and EBP models. This paper was born out of a discussion (or several discussions really) that Zaana and I had about the instances where EBP doesn’t cut it as a model for dealing with a problem – for example, situations where there is no existing literature, or situations that require innovation. We also had some doubts about particular aspects of the EBP model, including its reliance on published literature.

This presentation was a lot of fun to work on, not the least because it gave me an opportunity to get more familiar with Zaana’s field of design thinking, which I think has a lot to offer information practitioners. As always, it was a pleasure to work with Zaana, too!

Our presentation was well received – we got interesting questions during the session, and were pleasantly surprised to win both the delegates’ choice and committee’s choice best paper awards for the conference.

You can see our slides and references over at my other site, katedavis.info. We’d love to hear your thoughts on the hybrid model, too – we plan to write this up as an article, so all feedback is very welcome. You can either leave comments here or email us (davis.kate@gmail.com – I’ll forward feedback to Zaana).

Tagged with:

more on reflective practice

blogeverydayofjune, ebl, evaluation No Responses »
Jun 272010

I posted recently about the importance of publishing, and in that post I talked about reflection and how important I think that process is. And in my last post, I focused on evaluation – and reflection is a key tool in the personal evaluation toolkit.

So I was interested today to see a tweet about a new article on reflection and Evidence Based Library and Information Practice (EBLIP). EBLIP rock star Andrew Booth has just published an article called Upon reflection: five mirrors of evidence based practice.

Whether you’re for or against (or perhaps ambivalent about) EBLIP, this article is well worth a read.

via @dkouf

Tagged with:

making time for metrics

blogeverydayofjune, ebl, evaluation 2 Responses »
Jun 272010

I have lots of soap boxes, a number of which I’ve posted about here before. Evaluation (otherwise known as assessment) is probably my favourite soap box of all.

It’s also something that I don’t believe we take as seriously as we should. Librarians are really good at collecting statistics – perhaps *too* good at it. The issue, I think, is that we focus on the collecting with such intent that we sometimes miss out on the analysing. We measure everything that’s measurable, and some other stuff too, but I’m not entirely sure that we work with the data we collect in a way that tells us something meaningful about what we’re doing.

Do we actually change how we do things or how promote our services because of what the data tells us? And on an even more fundamental level, do we even take a considered look at the data we collect, or do we just send it straight up the line to meet our reporting requirements?

Evaluation of what we do on a day to day basis is critical. It’s critical because we owe it to our customers to be offering services that meet their needs and that are used. It’s critical because we are constantly asked to do more with less resources, and we need to be rational and thoughtful in deciding where to invest our energies. It’s critical because without meaningful evaluation, how can we ever improve?

Evaluation is also a critical phase in managing any project, and in my experience, it is a phase that is perhaps undervalued (or at least, under-emphasised) in what some might call 2.0 projects. I have observed that, in implementing online services that make use of freely available web based tools, practitioners tend to focus on agility. I’ve seen this result in a lack of planning, in terms of service design, project management and evaluation, and it’s my contention (and a profound one at that!) that in order to be meaningful, evaluation must be planned and aligned with the aims and objectives of the project. You need to know in advance what you’re going to measure in order to collect the right data along the way, and to ensure that you have adequate time allocated to evaluation.

I’m thinking about evaluation at the moment because I’m working on materials for a new unit that I’m running next semester, and working on a plan to evaluate how the unit performs in terms of allowing students to realise the unit’s learning objectives. I’m also thinking about the two units I taught in first semester, and reflecting on both my performance and how the unit structure and content worked. There are many inputs in terms of data: student feedback on my performance, and my personal scores on the Learning Experience Survey (LEX); student feedback on the units, and the unit scores on the LEX; student performance; informal feedback received from students; and students’ reflections (students write one reflection for each assessment item). Of course, working with all of this data will take time, but I think the evaluative process is critical, so I’m setting aside the time to do it.

Do you give evaluation the time, energy and thought power it deserves?

Tagged with:

do we want to know whether library blogs are succeeding in the big, bad web world?

ebl, evidence, library2.0, soapbox 3 Responses »
Mar 142008

There’s been plenty of talk around the Library 2.0 theme on the idea of evaluation or assessment. At Information Wants to be Free, Meredith Farkas says what she wanted to see come out of Library 2.0 was a greater focus on assessment. I certainly want to see libraries have a greater focus on assessment, too, and I want to see them publishing about it. (Particularly public libraries. We just don’t publish enough.)

Why aren’t we (libraries in general) publishing about the success (or failure) of our 2.0 projects? Why is there virtually no data to be found that quantifies some of the outcomes of 2.0 projects? We’ve been on this 2.0 bandwagon long enough for studies and assessments and evaluations to have been undertaken.  For a movement that’s intrinsically tied up with quick publishing channels like blogs and wikis, it seems strange that there is a real dearth of published studies on 2.0 projects. Why is that?

Walt Crawford had this to say in a recent post on his two blog survey books:

Maybe there’s a clear desire not to know how library blogs are doing in the real world, other than a few cherry-picked examples. I’d like to think that’s not the case. It would be unprofessional to tell people about how wonderful library blogs are, and encourage them to create such blogs, without giving them honest and broad-ranging information on what’s actually happening with such blogs.

I’d like to think that’s not the case, too. But I wonder. I wonder a few things:

  • Is the lack of publishing indicative of a lack of success? (And a fear of talking about it?)
  • Is the lack of publishing indicative of a perceived lack of success, a perception that might be formed because we’re not collecting the right data? (eg. How are we measuring ROI? Do we just count comments on blog posts? Or do we look at exit links, time spent on the page, holds on titles blogged about, impact on online resource usage stats…? I certainly hope all of these metrics and more are informing libraries’ evaluations of their blogs, because if we’re just relying on comments to measure user engagement, then we’re not seeing the full picture.)
  • Is the lack of publishing indicative of a lack of evaluation? (And if so, why aren’t we evaluating? Because we don’t know how? Because we don’t have time? Because we don’t want to know?)
  • Or, is it just that we’re not publishing about our evaluations?

I’ve got a blogging project in the pipeline at mpow. It’s germinating quite slowly, because I want to see it well planned. We want a well planned implementation, but also a well planned, multi-faceted evaluation. If it works, I want to know about it, and I want us to be able to reflect on what we did and make links to what worked. If it doesn’t work, I want to know about it just as much (if not more), because I want to be able to reflect on what we did, look for ways we could improve, and ultimately, pull the pin if that’s what we need to do.

Blogs (and all things shiny and 2.0) are just great. They’re fun for staff to work on, and have huge potential to engage our users. But none of us have time to run services that don’t work. If we don’t evaluate, we have no ability to know whether

We know that “because we always did it that way” is not a good reason to keep doing the things we’ve always done, whether they work or not. But neither should a failure to evaluate be the reason we keep on keeping on with our 2.0 services.

If you have evaluated your 2.0 service, publish about it! And if you have published, I’d love to receive some links.

Tagged with:

evidence…

ebl, evidence, soapbox No Responses »
Sep 042007

A brief but pertinent post from Lorcan Dempsey today on the use of evidence – or rather, the non-use of evidence – to back up discussions on bibliographic data and catalogues.

I’ve got evidence on the brain at the moment. This area of librarianship is not alone in the fact that its discourse sometimes lacks grounding in evidence. But the future of the catalogue is certainly a dominant topic of conversation at the moment – a conversation that could only benefit from the existence of and reference to a solid evidence base.

[Aside: While I'm talking about the future of the catalogue... I'll be very interested to see the papers from the Australian Committee on Cataloguing seminar that's on this Friday in Brisbane. Aptly titled Promise for the future, or legacy of the past? : cataloguing in a changing world the program includes papers on a range of topics from educating future cataloguers to tagging to RDA. Should be some good food for thought there.]

© 2012 virtually a librarian Suffusion theme by Sayontan Sinha