Monday, March 1, 2010

Scanning mentions of the library - Twitter, Google alerts & more

By now many corporations including libraries routinely scan for mentions of themselves on the internet. The number one free tool used is of course Google alerts, but with the rise of real time searches and microblogging, attention has turned towards scanning for Tweets.

In this post, I will share with you some of my experiences doing so called "environmental scanning"  using free tools.

The examples will be more impressive if I actually showed some real examples, but I felt that while such tweets are public, I am still  uncomfortable putting a spotlight on them, so I will not link to those tweets.

Your experience using the ideas here will vary depending on size of institution, penetration rate of Twitter etc .

For comparison purposes, NUS Libraries serves about 36K FTE (students and staff), and the most popular twitter account has over 500 followers. The official name is National University of Singapore libraries but is often abbreviated to just NUS Library or NUS libraries.

Tweet scanning

Many libraries now have Twitter account (more than 600 libraries), and use Twitter for diverse purposes including pushing out information to followers, as well as answering tweets sent directly to the account. But many librarians are also scanning for tweets about their libraries which are not directly address to them,  with the aim of collecting compliments and doing service recovery for complaints.

Google alerts are a lot faster these days in picking up tweets (used to be a lag time of months), but in my experience it's still unreliable, in any case, why not just go to the source and use the built-in Twitter search.

Talking about Google alerts vs tweet alerts, one thing I noticed is that scanning for tweets has a really low false positive rate compared to Google alerts. This makes sense, in a 140 character tweet, space is at a premium, any keyword appearing is obviously not incidental. 

By comparison google alerts pulls in lots of irrelevant hits, where say NUS and library is mentioned in passing on a long webpage.

After playing around with the Twitter search, you will notice that while it's pretty capable (you can use boolean operators), it doesn't do wildcards or truncation and unlike Google it's not smart enough to realise that "library" and "libraries" are different forms of the same root word.

Therefore tweets that say NUS libraries are not the same as tweets that say NUS library . So you should search for NUS libraries OR library . In fact, I recently realized that many   shorten library even further to lib , so perhaps NUS libraries OR library OR lib is more to the point!

Other terms you might want to scan for include librarian for obvious reasons. You can also scan for full names e.g. National University of Singapore libraries OR library but as you might expect this usually gets no hits (remember the 140 character limit means abbreviations are the norm).

Andrew Burkhardt the emerging tech librarian at Champlain College in Burlington, VT (I'm sure his google alerts will pick this up! ),  suggests we also scan for tweets with words, endnote, cite, research and  need AND book OR article OR books OR articles

I would add terms like database OR eresource OR journal OR catalogue (remember plurals are not automatically picked up) . If you have library branded names add those too. E.g we call our library catalogue LINC, our federated search is called Infogate

Of course it depends on your local situation, but so far for me, I find eresource and database being tweeted a few times (e.g complaining eresource is down, wondering what database to use for market reports), I've also gone with the broader book OR article OR books OR articles (without the word " need") There are a few false hits (people happy about getting paper published) but its not overwhelming. 

Incidentally, I had to split these searches into several different search queries rather than combining them all into one, because the maximum length of any twitter search is 140 characters,  

In all the above cases, I also do a AND NUS, so we are pretty sure we are talking about our books, articles etc.

But what if NUS (or your institution) is not mentioned in the tweet? 

I can think of two ways.

Tweet with geolocation scanning

As I mentioned in a blog post last year , tweets are starting to get geotagged, so one could do a search for the terms above and restrict it to tweets made within a certain radius of your library.

Just do near:"  longitude, latitude"  within:1km libraries OR library or lib OR librarian , this search query is surprisingly powerful. You can do the same for other search terms.

While not false hits, for me, the more general search (with institution name)  tend to alert on users who are doing checkins into foursquare and Gowalla (rare). So far, I haven't bothered to try to filter them out. (We have about 30-50 checkins per week in foursquare)

Tweet with scanning of users in a list

So far though, geotagged tweets are still rare as it is an option that needs to be turned on and not many clients support it yet, is there a third method to detect tweets about your library even though your institutional name isn't mentioned? This just recently occurred to me, if you know the person tweeting is from your institution, this provides yet another marker. 

So you need to create a list of all your users (say followers of your twitter account), then only scan those tweets for the required keywords. The only service I know of that does this is , which sends an email alert (no RSS though).

I find quite a few tweets talking about the library (or libraries or lib) by users but aren't picked up as they don't mention NUS , are tweeted outside NUS, or are not geotagged. 

I've tried this to prove it works, and it works very well indeed, but I'm a bit uncomfortable with using this though, and have decided to stop this.

What do you guys think?

Given that you have 3 types of searches, Normal with institution name, Searches without institution name but exploit location of tweet and scanning by tweeter, you might wonder if there are a lot of duplicates.

Strangely, I don't notice that many duplicates (actually I just realized why). A interesting option is to filter tweets by "positive", "negative" or try to ask Twitter to detect if a Tweet is a question, but I'm not confident of the accuracy to try this. Besides the volume isn't that high yet anyway.

Google alerts vs Bing

I won't share much here, mainly because my approach is getting a lot of false hits, so I'm trying to figure this part out. I will add though Google alerts have one huge weakness, while they recently started to index Facebook pages (this is so new that a few days ago, a google search restricted to gave zero results!), but not public status updates! 

Given the recent moves by Facebook to make status updates public by default, this means Google misses a big bunch of information! But is there a Bing alert? Turns out it didn't.
But I found this hidden trick, that allows you to convert any Bing search to a RSS feed. Simply do the Bing search you want and then add &format=rss to the end of the url!

You only want facebook results, so do a bing search nus library , grab the url, and add &format=rss to the end of the url

Scanning of specific social media services - Plurk , Friendster etc

A somewhat minor trick is to use the social search engine Socialmention . This is a very powerful search engine that indexes most of the more popular and obscure social media services, and more importantly allows you to restrict searches to specific social media services, and generate a RSS feed from the search.

 I basically use this to monitor Plurk and Friendster , which is fairly popular here.

Real time or near real-time notifications 

So far, I talked about doing different searches and all the method allow you to generate RSS feeds directly (except which is email only), but how do you put them all together?

So far, greatest successes at environment scanning came from users who were amazed at the speed of response. One user raved to his followers (who in term Retweeted to theirs, estimated reach of 3,000 ), about how we picked up on his ranting and within minutes fixed the problem.

This was possible only with real-time scanning via a Twitter client. I personally use TweetDeck which pops up alerts very quickly. The main problem here is that for some reason , Tweetdeck doesn't support searches that take into account locations. I tried a couple of popular Twitter clients, both web-based and desktop, and almost all had the same problem.

Anyone know of any twitter client that supports alerts of twitter searches with locations?

RSS feeds these days can be much faster with support of  pubsubhubbub (Google reader supports this)
or SUP (Friendfeed) , so maybe it isn't too bad.

There are many RSS feed readers out there, but I think GoogleReader and Friendfeed are both good choices, they allow you to store everything in one searchable place and both have mechanisms in place that supposedly allow "instant" RSS updates

GoogleReader has the advantage of greater support and a more certain future than Friendfeed, where a recent sale to Facebook has created lots of doubts of it's future.

On the other hand, Googlereader doesn't yet have a build-in service that automatically popups to alert you that a feed was updated , though no doubt addons exist that do this.

Here's how I do it.

First, I use Tweetdeck to pick up the normal tweets that aren't based on location, this is very fast within minutes (adjust the 

In addition, I setup a private Friendfeed group, toss all the RSS feeds into it including

  1. Tweets based on location, Tweets with institution name
  2. Google alerts in RSS feed
  3. Bing searches in RSS feed
  4. Social media in RSS feed
 I set it up to IM myself, so I get IM alerts when ever  any of the scans come up with something. And of course I have email alerts via to detect relevant tweets based on person tweeting.

Overcomplicated? Maybe. But since doing this, I realize that our users are talking about libraries a lot (both positive and negative), sometimes a lot more than a simple google alert, or twitter search would indicate.

But is such scanning going too far? Sure the tweets and information are all public, but would users object if they knew libraries were doing such things? My impression is many users have a "there is so much stuff on the net, they won't notice me" mentality.... right until they realize organizations are doing such scans and searches.  

In any case, the aim of libraries in doing this is to be able to help and not to police users, so any users reading this should not be alarmed!

Thus far, twitter has helped to quickly detect problems (systems down) or enable librarians to proactively form help when needed (database needed for assignment) and seems to be worth the investment of time. What do you guys think?

Which of the following do you use to scan for mentions of your library on the web

How do you monitor your alerts for mentions of your library?

blog comments powered by Disqus

Share this!

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.
Related Posts Plugin for WordPress, Blogger...