Monday, August 9, 2010

EndNote, Mendeley, Zotero, WizFolio - comparing import functions

Edit : I edited quite a bit based on comments below. Clearly, there's a lot I don't know about these reference managers, and there's possibly some PBKC problems involved! So the results shouldn't be used to claim X is better than Y.  Also since I blogged, a lot of improvements has being made so the data here should be considered outdated.

I'm somewhat bewildered by the different methods one can import citations into Reference Managers, so this is my attempt to organize them in my mind. The four reference managers I'm comparing are EndNoteMendeley , Zotero and WizFolio.

EndNote Mendeley Zotero WizFolio
Import from PDF Yes Yes Yes Yes
Import from text file (RIS but also sometimes Refer,
/BIBIX, BIB TeX, Tab limited, other formats etc)
Yes Yes Yes Yes
Import from bookmarklet No Yes No Yes
Import from clipboard No No Yes Yes
Import directly from RIS without saving to text file Yes No Yes No
Search by identifiers DOI, PMID via Online search Arxiv, DOI, PMID ISBN, DOI, PMID Locate bibliography searches PubMed
Others "Online search", Browser toolbar (EndNote Web) Google Scholar, drag and drop links Zotero translator, Links with CiteUlike "Intergreted search" of PubMed, ScholarsPortal, Google
Books, CiteULike, Bing

Firstly, all four reference managers can now extract metadata from PDFs with varying degrees of accuracy (See last blog post comparing this feature). This is a nice feature to have, since all you need to do is to download the PDF, point the pdf to the reference manager and it tries to figure out the citation.

Still, this method is extremely unreliable.

The second method, is perhaps most reliable. You search the database, export the citation in one of the various standards (RIS, BIB Tex etc). But this method is quite troublesome. Typically you save the file in text format. Then you open your reference manager, then browse for the file and select it. With EndNote you even have to select the right filter sometimes. And of course even after you have done this, you must add the PDF file.

I'm not quite sure if this is a seperate method, but in many databases, you can select "export to RIS (Reference Manager, ProCite, EndNote) and it will automatically pull in the reference into EndNote by opening it instead of saving it.  Zotero has something similar by turning on "Use Zotero for downloading RIS/Refer files" in preferences.

The main advantage of this is that when you export a database in RIS (which is by far the most common option), EndNote,Zotero automatically pull it in. With other reference managers while they support RIS, you will need to save the file on your hard-disk first, open the reference manager, select import file and browse to the file. This is a lot slower. I tried to associate RIS to Mendeley and open the RIS file (isn't of saving) to Mendeley but it didn't work.

There are a couple of other methods that do not involve RIS.

Mendeley , WizFolio allow you to set up a bookmarklet (a normal bookmark with javascript) called Mendeley bookmarklet and WizAdd, on supported sites, you just click on the bookmark and it will ask you to select the citations you want to import. In certain supported sites, you will be able to get the PDF as well!

Zotero does not have a bookmarklet, it uses it's own translators which are similar. Basically when you go to any database that is supported, the firefox location bar will show an icon that you click to import citations. For supported sites it brings in associated PDF as well, so in someways it's comparable to bookmarklets.

Another method, involves copying and pasting citations directly from clipboard.
WizFolioZotero will try to interprete and format it nicely into relevant fields. This can be helpful if you are tracing references backwards from  articles that are known to be relevant. You can just copy and paste the references from the bibliography of said article.

Sometimes the citation or reference you get is incomplete.  Mendeley or Zotero allow you to search by identifers such as PMID, DOI to pull in complete details of the article. (This can also used either manually or automatically to round up information if other methods such as extraction of metadata from PDF leads to incomplete information or when cutting and pasting citations directly isn't sufficient)

Somewhat similar, you can do a search within the reference manager for specific databases. The idea here is instead of going out to the database to search, you can search specific databases directly within the reference manager using certain search fields, like article, source, author etc.

WizFolio does Google books, PubMed, Scholar Portal etc, while EndNote does everything from hundreds of library catalogues to selected databases like Web of Science, WilsonWeb, CSA, ProQuest, EBSCO, Gale, OVIDSP databases (using z39.50?).

Comparing import methods

That's quite a number of methods each with their pros and cons. In terms of support for databases, all of them at the very least support import of RIS files (i.e Save as a file, Open reference manager and select file).  So based on that they are all probably capable of importing citations from 99% of databases. However this method is slow and clumsy, since it involves a two step process and is very disruptive to the research workflow. Similarly entering dois, Pmids are not really that convenient when all you want to do is a) Do a search B) Select the articles you want C) Import the citations with a couple of clicks without leaving the browser D) Ideally get the associated pdf 

Instead in the test below, I will try out only the following methods

1. Wizfolio - via Bookmarklet
2. Mendeley - via Bookmarklet
3. Zotero - via Zotero translator and with direct export to RIS
4. EndNote - via Direct export to RIS

I don't teach Zotero, Mendeley or Wizfolio, so I'm not sure what the primary method of importing recommended, given that each reference manager has more than one method.

But I believe the methods tested above are basically the quickest, "one click" import methods and only if they don't work, then you save the file as RIS text file and manually bring it in.

The databases I chose to test were mostly from the major electronic resources listed on our library portal. They span a range of disciplines from Medical to Social Sciences, and are a fair test I think. I also included our library catalogue, webpac Pro from innovative interfaces.

I used essentially the same setup as the earlier blog post, I used Firefox 3.6 (of course due to Zotero) also turned off the popup blocker just in case.

I did a search for wikipedia in all the databases, on the results page (though I noted some oddities, where the bookmarklet didn't work on the results page, but when you clicked on individual article page it worked?), I ran the bookmarklet or clicked on the icon in the firefox location bar. In the case of EndNote and Zotero , I selected export to RIS to see if they could automatically pull in the results.

If it automatically pulled in the associated PDF, I noted it too.

As usual, I don't claim this test to be scientific, it's just to get a feel. 

EndNote Mendeley Zotero Wizfolio
ABI/Inform Yes No Yes Yes
BIOSIS Yes Yes Yes* Yes
Business Source Premier (EBSCOHost) Yes Yes No, translator error, Yes Directly via RIS No
Compendex Yes No Yes Yes*
EconLIT (OVIDSP) Yes No No No
EngineeringVillage 2 Yes No Yes Yes
Expanded Academic ASAP No* No No No
Factiva No No Yes Yes
IEEE Xplore Yes No (Yes
without proxy)
No, translator error, Yes Directly via RIS Yes
Jstor Yes Yes Yes, with PDF Yes*
MathSciNet No No Yes No
PsycINFO (OVIDSP) Yes No No No
PubMed No Yes Yes Yes
ScienceDirect  Yes No (Yes
without proxy)
No, translator error, Yes Directly via RIS Yes
Scopus   Yes No No, translator error Yes
Web of Science  Yes Yes Yes Yes
Google Scholar  Yes Yes with PDF Yes,
with PDF
Yes , with
Library catalogue - WebPac Pro No No Yes No
Total 13 7 (5 with proxy) 14 (12 via translator) 12

Overall based on this test, EndNote did the best with 13 successes . But it's not surprising, since RIS is the most supported standard and EndNote is the oldest reference manager of the batch. It is able to directly pull in citations using this method, means it supports traditional databases the best. On the other hand compared to other methods it can't get the associated PDF automatically.

Zotero and Wizfolio were tied next with 12. (According to comments below, Zotero should work with Sciencedirect (translator) , Academic ASAP (RIS) , IEEE (RIS) & Factiva (Translator) which would give it a score of 15 but I can't verify all but the last two)  Wizfolio's bookmarklet seem to work a lot better than Mendeley's bookmarklet (at least on my system). But it managed to bring in only one associated PDF (Google Scholar) even though I think it was supposed to be able to do it for others like JSTOR? Then again Zotero's translator which was supposed to be able to pull in associated PDFs did so only for Google Scholar as well.

I'm somewhat disappointed and puzzled by this failure to bring in PDFs since it's the main selling point of such methods. Granted, I didn't test too many full text databases, but failure to pull in PDFS for Sciencedirect and JSTOR is strange (at least for Mendeley).  I wonder if it has anything to do with our ezproxy implementation?

Also Zotero's translators seem to be broken quite a bit , Scopus has never worked as far as I can remember, and now I see the same for ScienceDirect, IEEEexplorer and Business Source Premier (via EBSCO) . If we based it just on translators it would rank 3rd with 12 successes . But it also allows you to "Use Zotero for downloading RIS/Refer files"which is similar to the direct export in EndNote , which puts it back to 12.

Mendeley's bookmarklet seems to be worst of all. It occurs to me that as the CEO of Wizfolio is actually an adjunct professor of my institution (though currently the library officially supports only EndNote in terms of training), so he might have spent quite a bit of effort ensuring  Wizfolio's bookmarklet works for our specific profiles (I assume he needs to use it himself!).

Still Mendeley results are disappointing. In fact initially when I tried it, there were only 5 successes. I later accidentally noticed that if I did the search without EZproxy it improved to 7 (due to ScienceDirect & IEEE explore). I'm not sure if it's something specific to my use of EZproxy. 

Of course, this is just a snapshot, and if I used a different set of databases, I would get different results (particularly I suspect EndNote is weaker on non-academic sources) but it does support my suspicion, that in terms of database support EndNote being the market leader is better supported then the rest.

Also as I understand it bookmarklets and Zotero's translators are highly depend on the layout of the page, so the database interfaces that are changing rapidly tend to break them.

On the other hand, it's not a big deal I guess, if bookmarklets and translators don't work reliably, if you can rely on simply downloading the RIS file as text file and import. However, In the comment to my earlier blog post , someone who I think is a representative or at least supporter of Zotero implied that translators were the main method to be used.

I'm learning about this as I go along, so I would appreciate comments by more knowledgeable people, on whether bookmarklets/translators not working reliably is a big issue or not. 

blog comments powered by Disqus

Share this!

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.
Related Posts Plugin for WordPress, Blogger...