In other cases, Summon yielded less results than Google Scholar with the exact same search statement but at a big decrease in recall.
Attempts to use the more advanced search features in Summon to include wildcards and longer search statements not possible in Google Scholar, actually exploded the search even further.
Even though I am not a medical librarian, I have read with interest the recent paper "Google Scholar as replacement for systematic literature searches: good relative recall and precision are not enough" by Martin Boeker, Werner Vach and Edith Motschall.
- translated search strategies used to find relevant papers in past systematic reviews into Google Scholar equivalent search statements (as close as possible anyway)
- Checked how many relevant papers were found (the papers found in the original systematic review is the "gold standard" of what is considered relevant)
- Calculated the recall and precision of using Google Scholar as compared to traditional systematic review methods of searching multiple databases (typically Medline, Web of Science, Cochraine Library etc)
The results aren't particularly surprising, as argued by many other papers and blog posts , despite Google Scholar's large nearly comprehensive coverage of studies that allows it to pick up the papers using just one source (93% recall in this paper), Google Scholar has many weaknesses making it unsuitable for use in systematic reviews alone. In particular the lack of precision due to lack of advanced search features is a big one.
As I read through the paper, which is the most comprehensive one I have seen detailing the various weaknesses of Google Scholar for systematic reviews, I couldn't help but think how many of the critiques in there would parallel that for Summon.
In the past, I have blogged about how How Google is different from traditional databases and later I mused about How library web scale discovery services in particular Summon are closer to Google and Google Scholar , but not quite there yet.
On one-hand Summon has many of the same characteristics as Google Scholar. With breath unmatched by traditional databases, it was designed also to maximise recall at the cost of precision with features like auto-stemming which makes it feel google like.
But on the hand Summon does have more advanced search features (though a bit well hidden) and stability of results and more transparent sources.
So how does Summon stack up? Let me go through the critiques against Google Scholar and see if they apply to Summon.
Here's a short summary of some issues in Google Scholar.
- Maximum 1,000 results, 20 results per page - Summon same limitations, 50 results per page
- No bulk export - Summon same limitation, Zotero allows export of results by page for both
- Lack of search history - Summon same limitation.
- Limited advanced search interface - Summon 1.0 same, Summon 2.0 is better
- Lack of truncation and advanced field searches - Summon is better
- Inability to nest logical operators more than one level - Summon is better
- Limited query length to 256 characters - Summon does not have this limitation
- Autostemming leads to lack of control - Summon has same limitation
- Reliability and stability of index - Summon is better with more transparent listing of sources.
Overall : Reasons to believe one could better translate traditional complicated search strategies to Summon which might result in better recall and precision (assuming the full index of Summon is comparable to Google Scholar), but need an actual study to confirm, which will take a bit of expertise to translate search strategies and even more time to look through the results.
But similar to Google Scholar, limitations like maximum 1,000 results, lack of bulk export might make this moot anyway.
For more detail, read on!
Let's start with graphical interface features.
Quotes below are from provisional PDF of "Google Scholar as replacement for systematic literature searches: good relative recall and precision are not enough" by Martin Boeker, Werner Vach and Edith Motschall allowed under BioMed Central Open Access license agreement.
"Not more than 1000 results of the complete result set can be displayed in steps of
maximum 20 results per page."
Many people are surprised to know that regardless of the number of results Google Scholar finds, you can see at best 1,000 results and it won't show more results. (Google is similar though it's variable in terms of where it stops showing results).
In Summon 1.0, one can increase maximum number of results to 50 per page compared to 20 per page for Google Scholar. But you can't get more than 1,000 results.
In Summon 2.0, there is no concept of pages, with the so called "infinite scroll" feature, I can't tell if this limits to 1,000 but might be moot (see below)
Frankly if there is any one reason not to use Google Scholar, this one alone would be sufficient, since many searches done would have >1,000 results. Still, let's press on.
"No bulk export of results is available. Results can only be exported into reference management software (e.g. ZOTERO)"
Yet another killer, since you need to mass export results you get, ideally all results with one export.
With Google Scholar, you can go export item by item, for mass export the only option is to pair it with Zotero so you can bulk upload page by page of results. Unfortunately with Google Scholar you can see only a maximum of 20 per page so it may take a while to export everything. (Wild idea use Publish or Perish software to get everything in one shot? Though limited to 1,000 results plus search limitations?)
Summon 1.0 is exactly the same with no bulk export. You can use it with Zotero, exactly as in Google Scholar, so you can bulk export all the results in one page. Here it works slightly better than with Google Scholar, since you can set the page to display up to 50 results per page as mentioned.
Summon 2.0 has mentioned has "infinite scroll". I think this feature pretty much kills the possibility of quick bulk export. Or would I research, keep scrolling down...until the end, and then exports by bulk with Zotero?
Kinda moot now cos zotero does not work with 2.0 currently.
Lack of "a history function which temporarily stores retrieval results for incremental refinement of search strategies"
"It is not achievable to construct all possible expressions in the advanced search interface
due to the limited number of available entry fields. Only one field for each type of
expression (conjunction, disjunction and conjunction of phrases) is available"
Summon 1.0 advanced search screen
- publication title
- subject term etc
- publication title
- subject term
"Search expressions were limited to a length of 230 characters due to the restriction of a
total of 256 characters"
This is a barrier to creating complicated search strategies in Google Scholar as it has a character limit for length of search queries. This is a big deal because many search strategies needed for systematic reviews are extremely long and complicated.
The paper notes the median length of the Medline searches are 777.5 characters. But because of the character limit in Google Scholar, they had to simplify searches and in the study the median length of the "translated" Google Scholar ones had a median length of only 187.5!
This is a big limitation of course.
As far as I can tell by some testing, Summon does not have the same limitation as Google Scholar. If the query is long enough, browser limitations on processing long URLs (typically >2000 characters) will start to come into play, but this isn't a limitation of Summon per se.
"Terms in Google Scholar are complete single words (truncation is not possible)"
Based on support files, Summon allows truncation (but not within quotes) and proximity operators (without taking into account order).
"Google Scholar applies automatic stemming to terms where the stem is recognizable for Google Scholar. However, this mechanism might not be reliable for domain specific language (e.g. the medical language)."
The paper also warns that "correct interpretation of logical connectors" still needs to be improved, with Google Scholar often giving illogical number of results.
As far as I can test, for Summon, you can nest boolean operators to more than one level. Still, there are indications in the support files that imply complicated nested boolean searches might sometimes give odd results and to report them if seen.
A well known example was where adding quotes would occasionally give MORE results, as reported in this article. So for example
sheep dip flies
would give you less results than
“sheep dip” flies
My understanding of the issue was that Summon by default would do an implied proximity matching (within 200 words) for three or more searches terms if they were found in full text in an attempt to filter out totally irrelevant results where words were extremely far apart in the full-text, but would switch this off when quotes were used.
In any case in the latest versions, this issue is resolved.
"The currency of Google Scholar may not be very high for some resources. The update
period for certain resources is up to nine months. Although research results indicate
very high coverage of Google Scholar, the exact coverage is not known. Google itself
states that it does not index journals, only articles, and does not claim to be exhaustive."
- Multiple level of nesting
- Search queries >256 characters
- More fields for searching including subject terms, abstract (Summon 2.0)
- Filtering by content types (eg journal articles), discipline (Medical, Economics etc)
- Lack of truncation support in Google Scholar
- Google Scholar's limited query length restriction
- Maximum 1,000 results in Summon 1.0 (infinite scroll in 2.0?)
- No Search History
- No bulk export