evaluation » virtually a librarian

towards a research-led profession (or, how to become a practitioner-researcher)

ebl, evaluation, evidence, research No Responses »

Feb 212012

At LibraryCamp Australia 2012, there was a discussion about building a culture of research in library and information practice. Lots of really practical ideas were shared during the session, and I wanted to follow up here with a few of my ideas on how professionals can get involved in doing research. So here goes!

Every project is an opportunity for research

Starting out a new project with clear, measurable objectives is not only good project management, but it also provides you with an opportunity to publish. Setting objectives and putting strategies in place to collect the data you need to measure achievement of those objectives sets you up to evaluate the success of your project once you hit the finish line. Publishing on the results of your project and your evaluation of its success is a great way to help build the evidence base.

Collaborate

If you’re new to research (or even if you’re not!), collaborating with your colleagues can be a great way to break down some of the fear you might have about doing research and getting published. Collaborate within your institution and beyond it. Connect with people that have similar professional interests and just start talking about what interests you. You’ll be surprised by how many ideas you come up with. Also consider collaborating with academics – we’re always looking for opportunities to work with practitioners on real world projects. Your professional expertise and their methodological expertise will complement each other.

Get a mentor

There are many, many active practitioner-researchers in the library and information professions. Find someone whose work you admire and ask them if they’ll mentor you through the process of undertaking your first research project and looking for publication opportunities. You might already have a professional mentor, but your research mentor is probably a different person: someone who has experience in designing and seeing through research projects and getting published.

Can’t find a research mentor? Drop a comment here and I’ll see if I can help you.

Look at alternative channels

Journals and conferences are probably the two most obvious places to publish the results of your research, but have you thought beyond that? Blogging is a great way to document your research journey and a really valuable way to disseminate your research. Academic publishing timelines are laggy and can mean that your cutting edge project isn’t so cutting edge any more by the time your journal article gets published. A series of blog posts on your findings can help you disseminate any time-critical aspects of your research as you progress.

Get lots of traction out of a single project

Remember, one project does not necessarily equal one publication. You can get much more traction than that out of your research! By publishing some of your findings on a blog, for example, you don’t preclude yourself from publishing about that research in a journal or presenting at a conference. There are always a variety of angles you can use for publications about a single research project. A project I led last year has so far resulted in three publications, and I’ll be publishing again out of this project this year (more than once). Each of these publications focuses on a different aspect of the research and was designed for a different audience. Even after five or so publications, there will still be data (and different views of the data) that I haven’t published on.

But that’s okay, because the other thing you need to remember about research is your data has a longer shelf life than you might imagine. When I first made the switch to academia, I was always in a rush to publish while my data was fresh, but I learned very quickly that I actually have quite a big window of time to disseminate the results of my research. Don’t publish one surface-level view of your research in your haste to get your findings out there. Instead, take a narrower but deeper view of your research in a series of publications over a longer period of time. You’ll do your research more justice this way and your publications will be more useful to others.

Start with a literature review

We’re librarians. We know how important it is to review the literature before we start a project. But instead of just writing up a short literature review in a publication on your findings, consider writing a more holistic literature review before you get started on your research, and publish it independently. Our professional literature could definitely use more of this type of publication. The process will help you focus your research on the gaps in the literature and your review will benefit other researchers. You might consider, for example, publishing a literature review in the new review section of the EBLIP journal.

Do a research degree

If you’re keen to get involved in research, you might like to consider getting a research degree. As librarians and information professionals move into research support, research training is going to be increasingly relevant for practitioners. At VALA a couple of weeks back, research support was probably the hottest topic on the program. Increasingly, information professionals are taking a really practical approach to research support and contributing to research in really tangible ways, through activities like literature searching, grant writing, and advising on publication opportunities.

If your professional qualification is a Graduate Diploma, you might be thinking about upgrading to a Masters. Instead of upgrading by completing more coursework, consider a research degree. The benefits of this in terms of your skills and knowledge are obvious, but the benefits for your wallet shouldn’t be discounted. Masters by research programs are fee-free.

We have an awesome research community at QUT… Come and play with us!

Just do it!

Stop thinking about it, and just do it! You’ve probably got at least one project on the go right now that you could be publishing about. So do it! The Research Applications in Information and Library Studies seminar is a perfect venue to publish research in practice, because the focus of this event is on connecting researchers and practitioners and fostering a culture of research in the profession.

Abstract submission for RAILS8 closes this Thursday, 23 February. You’ve got plenty of time between now and then to prepare and submit a 300 word abstract! The Organising Committee eagerly await your contribution!

more on reflective practice

blogeverydayofjune, ebl, evaluation No Responses »

Jun 272010

I posted recently about the importance of publishing, and in that post I talked about reflection and how important I think that process is. And in my last post, I focused on evaluation – and reflection is a key tool in the personal evaluation toolkit.

So I was interested today to see a tweet about a new article on reflection and Evidence Based Library and Information Practice (EBLIP). EBLIP rock star Andrew Booth has just published an article called Upon reflection: five mirrors of evidence based practice.

Whether you’re for or against (or perhaps ambivalent about) EBLIP, this article is well worth a read.

via @dkouf

making time for metrics

blogeverydayofjune, ebl, evaluation 2 Responses »

Jun 272010

I have lots of soap boxes, a number of which I’ve posted about here before. Evaluation (otherwise known as assessment) is probably my favourite soap box of all.

It’s also something that I don’t believe we take as seriously as we should. Librarians are really good at collecting statistics – perhaps *too* good at it. The issue, I think, is that we focus on the collecting with such intent that we sometimes miss out on the analysing. We measure everything that’s measurable, and some other stuff too, but I’m not entirely sure that we work with the data we collect in a way that tells us something meaningful about what we’re doing.

Do we actually change how we do things or how promote our services because of what the data tells us? And on an even more fundamental level, do we even take a considered look at the data we collect, or do we just send it straight up the line to meet our reporting requirements?

Evaluation of what we do on a day to day basis is critical. It’s critical because we owe it to our customers to be offering services that meet their needs and that are used. It’s critical because we are constantly asked to do more with less resources, and we need to be rational and thoughtful in deciding where to invest our energies. It’s critical because without meaningful evaluation, how can we ever improve?

Evaluation is also a critical phase in managing any project, and in my experience, it is a phase that is perhaps undervalued (or at least, under-emphasised) in what some might call 2.0 projects. I have observed that, in implementing online services that make use of freely available web based tools, practitioners tend to focus on agility. I’ve seen this result in a lack of planning, in terms of service design, project management and evaluation, and it’s my contention (and a profound one at that!) that in order to be meaningful, evaluation must be planned and aligned with the aims and objectives of the project. You need to know in advance what you’re going to measure in order to collect the right data along the way, and to ensure that you have adequate time allocated to evaluation.

I’m thinking about evaluation at the moment because I’m working on materials for a new unit that I’m running next semester, and working on a plan to evaluate how the unit performs in terms of allowing students to realise the unit’s learning objectives. I’m also thinking about the two units I taught in first semester, and reflecting on both my performance and how the unit structure and content worked. There are many inputs in terms of data: student feedback on my performance, and my personal scores on the Learning Experience Survey (LEX); student feedback on the units, and the unit scores on the LEX; student performance; informal feedback received from students; and students’ reflections (students write one reflection for each assessment item). Of course, working with all of this data will take time, but I think the evaluative process is critical, so I’m setting aside the time to do it.

Do you give evaluation the time, energy and thought power it deserves?

Walt Crawford's Public Library Blogs: essential reading if your library is (even thinking about) blogging

evaluation, evidence 2 Responses »

Dec 132008

This weekend, I’m working on the bordering-on-mythological paper on mpow’s blog pilot. To that end, I’ve been trawling the interwebs looking for blog posts, articles… anything documenting libraries’ and the corporate world’s strategies for evaluating the success or otherwise of blogging projects. I’ve been trawling for a while, but I live in hope that it’s just my search skills letting me down, and I’m going to miraculously find the very article I need at the eleventh hour.

As I’ve lamented earlier, libraries are not publicly documenting their evaluations of blogging projects, which is a problem because it makes benchmarking near impossible. Sure, you can still come up with a bunch of metrics and work out a number for each, but how do you know if the number you’re getting it good or bad?

Luckily, through a serendipitous Twitter experience, I’ve managed to track down a couple of people who were willing to share their data. But I need more. More, I say!

Enter Walt Crawford’s Public Library Blogs: 252 Examples. I have to confess, I’ve been meaning to buy and read this book for ages. I finally did buy and read it today, and I wish I’d read it a while back. If you’re responsible for a blogging project, you need to read this study. Especially useful is the grouping of the examples by population served – this allows you to compare your library’s blog’s performance for key metrics against like-sized libraries. I think it’s also a useful tool in setting realistic expectations, especially when it comes to converstatinal intensity (or the number of comments libraries tend to get per post). I wish I’d bought this and shared it with our staff ahead of the pilot.

While Crawford didn’t have access to usage stats, this is still an incredibly useful book.

I do wish, though, that someone would go begging to the library community at large to supply site visits, page views, links clicked and a few other metrics besides, so that we could have a ginormous study alla Walt Crawford’s that includes the kind of statistics he unfortunately didn’t have access to. PhD thesis, anyone? Even to do it for a grouping of libraries (say, public libraries serving populations between 400,000 and 500,000 – obviously, my reasons are entirely altruistic) would be incredibly useful, and you could extrapolate for libraries of different sizes.

As an aside, I love that I can buy a book online and be reading it within two minutes. I wonder if publishers will ever sort out the DRM debate and get pricing right on eBooks so we can do this with anything we might want to read.

Right, enough procrastinating.

a blink, a metric and some thinking out loud

evaluation, evidence 4 Responses »

Nov 092008

So, I blinked… and suddenly six months went by without a peep on this blog. Life has been doing what it does best – barreling along while I try to keep up. Where have I been? Right here, but buried under a pile of projects, a gaggle of committees, and a series of attempts at getting non-library, non-technology related hobbies.

Life’s not easing up but blogging has been on my mind lately, in more ways than one.

In my last post, I raised some questions measuring the success or otherwise of 2.0ish projects. I’m still thinking about this stuff – constantly. The time for me to evaluate mpow’s new blogging project is rapidly approaching, and I’m starting to look in earnest for literature on other libraries’ evaluations of similar projects. I’m still not turning up a lot.

I’m spending a fair bit of time thinking about what success for a project like this looks like. It’s a difficult thing to conceptualise. And obviously it’s something that needs to be conceptualised before you can figure out what sort of data you need to measure that success.

Of course, success for this project will be measured against our aims and objectives for the project – some of which relate to topics quite apart from the level of usage the blog has garnered, including aims like trialling blogging as a service delivery platform, and providing staff with an opportunity to get familiar and comfortable with blogging in the public domain (and with the technology itself).

But what other things should we be looking at? What, in general terms, makes a blog successful?

Conversational intensity

This is something I think a lot of bloggers get hung up on, so it gets its own sub heading. To what extent is success in blogging about “conversational intensity”?

We’re not getting a great deal of comments (I’ve got a theory about why that is, which I’ll probably blog about later), and I’m not particularly phased by that at this point. I had a chat with a colleague about the appropriateness of using blogs without being too concerned about generating conversation. She indicated she thought that a blog without multi-way conversation (ie with little commenting) misses the point of blogging. Her feeling is that conversation is a fundamental element of blogging.

I think I agree, to a certain extent, but I’m not convinced that blogs that exist without active commenting don’t have their own role to play. After all, we know that there are lots of different types of participants in this participatory web: consumers of information; occasional content producers (commenters); active content producers; and so on. (And this doesn’t even take into account the idea of using a blog as a CMS of sorts – people do great stuff with WordPress-driven websites. But that’s a little different.) Does it really matter if you don’t get a whole host of comments? Is there a ratio of comments to page views* that indicates a blog is successful in facilitating conversation?

In my opinion, level of conversation is a measure you should get hung up on only if it’s a primary aim for your blogging project.

Attributes of a successful blog

So, if the success of a blog does not hinge on conversational intensity, on what does it hinge (other than the blog’s aims and objectives)?

According to Asterisk, a successful blog is:

Well written
Frequently updated
Consistent
Open
Responsive
Well designed
Aware of its audience
Varied in topic
Personal
Thick skinned
Honest
Accountable
Funny

Are these the kind of success measures against which libraries should be assessing their blogs? Are subjective measures like these valid? And how do we measure against them? These measures do appeal to me. Or some of them do – some are obviously personal preference things, and dependent on the type of blog (like funniness, for eg) but others could be useful.

An obvious thing to measure is readership – subscribers, site visits, post views and so forth. But how do you decide whether the level of readership is enough to mean success? And, in the case of our project, whether the level of readership is enough to warrant a transition from pilot to permanent service? Do we work with a ratio of site views for the blog versus site views for the library’s website? And if so, do we compare to page views for the library’s home page, or do we look at page views for the young people’s page on our website and compare to that? Do we look at site views for the blog versus population in our region for the target audience?

Is their any validity in looking at participation in polls? What if we created polls for the express purpose of getting a feel for the number of people who might participate? What would that tell us?

Clearly, my thoughts on evaluating blogging projects are still fairly nebulous, and I know I need to do some research outside the library field to see how (or even if) other industries are evaluating the success of customer facing, service oriented blogs. But I do want to here from other library-types on this, and it appears I aint gonna find what I need in the literature.

So now that that I’ve just suggested that conversation may not mean much in an assessment of a blog’s success, I’m going to try and start one. Tell me, readers (if there are any of you left, after my six month hiatus): what do you think the markers of a successful library-land blog might be? What are the attributes of a successful blog? If you were evaluating a blogging project, what data would you be collecting and what would you be comparing to? How would you decide if your blog is a success?

* Conversational intensity is often measured by dividing total comments by total posts, but I think another useful metric would be to get a feel for the number of visitors who feel compelled to get involved in a conversation.

virtually a librarian

towards a research-led profession (or, how to become a practitioner-researcher)