Results 1 to 10 of 10

Thread: The new QueryPic -- visualising newspaper queries in your browser

  1. #1
    Trove user wragge's Avatar
    Join Date
    Jun 2010
    Location
    Canberra
    Posts
    63

    The new QueryPic -- visualising newspaper queries in your browser

    Just letting you know about the latest version of QueryPic, a little online tool I've developed that enables you to graph historical newspaper searches over time. It's very easy to use. Just go to:

    http://wraggelabs.com/shed/querypic/

    Enter a word or phrase and click 'show'. QueryPic builds a graph that shows you the number of results matching your query over time. You can create comparisons by simply adding additional queries.

    QueryPic accesses data from either Trove (Australia) or Papers Past (NZ). To explore the results of your query in more detail just click on any point on the graph and you'll see a list of the first 20 matching articles. It's a useful way of investigating trends and checking hunches.

    This is the latest in a line of tools I've been developing for working with the Trove newspaper database. Previously to create these graphs you had to download a script and run it on your machine, but now through the magic of APIs it's all done in your web browser.

    With the public release of an API (Application Programming Interface) to Trove it's possible to develop more robust and flexible tools. Over the next few weeks/months I'll be updating my existing bits and pieces to take advantage of it. Keep an eye on http://discontents.com.au for updates.

    Cheers, Tim
    Tim Sherratt
    @wragge on Twitter
    Words at discontents.com.au
    Experiments at wraggelabs.com and labs.nma.gov.au

  2. #2
    Prolific poster
    Join Date
    Feb 2011
    Posts
    260
    G'day Tim,

    I have had a play with your online QueryPic and I think it is fantastic !!

    I have been very impressed with your graphs shown in the forum previously but I don't know the first thing about running those scripts, so this online interface is great ........ my head is buzzing with all the possibilities. I think many, many people will find this a very useful tool indeed for a whole range of study interests.

    After a bit of experimenting with QueryPic, straight away I am finding answers to questions I had about long term trends in my particular research interests.


    I have a few questions/observations if I may:

    - Is there a way (or a feature to be added in future) to produce graphs on the basis of OR searches, i.e. when there is more than one word/phrase for the thing you want to graph (e.g ship OR boat)?

    - I am not sure I understand how the search for a phrase (without quotes) is conducted. You get very different results for [fire man] than you do for ["fire man"], with the first being much more numerous. Is the option without the quote marks actually an AND search across the articles or do the words need to be adjacent, or how does this work?

    - I notice that comparison of the results between PapersPast and Trove reveals that they deal with the search terms quite differently, so it is difficult to compare the graphed results between New Zealand and Australia in some cases (this is aside from the fact that PapersPast is only using its headlines in the searches). The search results for [fireman] and ["fire man"] are quite different, as you might expect, on PapersPast, but on Trove the ["fire man"] type example always gives you slightly higher (or nearly identical) numbers than the [fireman] type example. The search functionality on Trove was changed a few months ago in how it deals with a ["fire man"] type search, it now automatically searches for "fireman" as well - so I presume this is the cause of the differences between PapersPast and Trove.

    - I also notice that the last few data points on the right are nearly always anomalous for graphs based on Australian papers. I would suggest that this is because nearly all the online newspapers are only available up to 1954 (I think) due to copyright restrictions, and so the last few points would be based on things like the Womens Weekly, which is obviously skewing the last few data points. Perhaps the graphs derived from Australian papers should be limited to 1954 (or whatever the year is).


    I think tools like this are marvellous for discerning long-term trends is in huge datasets like Trove. Although, as always, care does need to be taken in understanding the data and interpreting the results.


    Congratulations on a fantastic tool.


    Spearth

  3. #3
    Trove user wragge's Avatar
    Join Date
    Jun 2010
    Location
    Canberra
    Posts
    63
    Thanks!

    I wanted to keep the interface as simple as possible, so I didn't want to add too many search options. But what I'm intending to do is to add the ability to parse search queries from Trove. What I can then do is create a simple bookmarklet that you can click on once you're happy with your search. QueryPic will then open and graph it. So you'll be able to display complex searches, while keeping the QueryPic interface as simple as possible.

    That said, you might have noticed that I've added an "exact" or "fuzzy" option to try and clarify what was being searched a bit.

    cheers, Tim
    Tim Sherratt
    @wragge on Twitter
    Words at discontents.com.au
    Experiments at wraggelabs.com and labs.nma.gov.au

  4. #4
    Trove addict
    Join Date
    Jul 2010
    Location
    Newcastle, New South Wales
    Posts
    72
    Thanks! This was very interesting (and for me very useful) Tim.
    My research subject involves a very short period during the 1800s so it was interesting to see the frequency from 1803 through.
    Jane

  5. #5
    Trove user mogedon's Avatar
    Join Date
    Jan 2011
    Location
    Sydney
    Posts
    41
    Very, very useful. But I reckon Spearth (and I suspect Tim would argue the same) is right; QueryPic can't be used in and of itself as evidence and interpretation of the results is needed.

    One example I might list is "Australian Federation" and "Federation". The first time round I used the "exact" search function, which ignores Trove's fuzzy search default.

    Screen shot 2012-05-11 at 2.10.19 PM.png

    The peaks during the 1890s and 1900 make perfect sense. That's when the constitutional conventions took place, and 1901 was of course, the first year of Federation. But looking closer it seems as though the use of the word "Federation" and phrase "Australian Federation" are already on the decline before 1900. If this holds true then it might stand as a counterpoint to say, John Hirst's Sentimental Nation (2000), where he argued that Federation was an overemotional mass movement from below. One gets the impression from the graph however, that Australians were losing interest before it was even consummated!

    Doing a "fuzzy" search for the same terms returns a markedly different result:

    Screen shot 2012-05-11 at 2.29.09 PM.png

    In this account, "Federation" usage falls smack bang in 1901, where it should. However, the fact that it barely reduces in incidence for the next fifty years makes me wonder at whether the graph isn't being propped up by writers using the term stylistically to describe a "federation of rabbits" for example, or the creation of a new federation elsewhere around the world. We also have to take into account the quality of the OCR'd text and the words that might be captured in a fuzzy search: "federally" for instance.

    Either way, it does seem odd that exact text searching should return that particular result.

    One other oddity is the spike around 1860. I reckon this can be put down to most of the colonies gaining self-government over the course of the 1850s, but the catalyst appears to have been the publication of a NSW government report on federating the colonies at that very early stage. Interestingly, this early enthusiasm for Federation seems to have been entirely forgotten in the wake of events during the 1890s and 1901.
    Last edited by mogedon; 12-05-2012 at 11:35 AM. Reason: pics + spellcheck

  6. #6
    Trove user wragge's Avatar
    Join Date
    Jun 2010
    Location
    Canberra
    Posts
    63
    Indeed I would, in fact to quote my notes from my Harold White lecture the other night :

    It's a simple tool -- presents number of matching articles over time.

    But it's quite effective, useful and it's also fun.

    Gives you the ability to quickly explore a hunch, to get a sense of context, or to frame a more specific question -- without spending days searching and tabulating.

    There are obvious limitations -- I wouldn't regard one of these graphs as evidence.

    There's a lot to unpack -- workings of the search engine, different usages, quality of the OCR etc.

    It's a starting point, not an end.
    Tim Sherratt
    @wragge on Twitter
    Words at discontents.com.au
    Experiments at wraggelabs.com and labs.nma.gov.au

  7. #7
    Prolific poster
    Join Date
    Feb 2011
    Posts
    260
    G'day Mogedon,

    Quote Originally Posted by mogedon View Post
    Either way, it does seem odd that exact text searching should return that particular result.
    I would guess the difference you see here is due to the quality of the OCR. Around the time of Federation the word "Federation" would have appeared frequently in heading lines and they have a much higher likelihood of being OCRed correctly. I imagine this would influence the outcome of an 'exact' search, whereas a 'fuzzy' search would pick up a lot of the not-quite-right matches as well that are largely going to be in the body of text.

    Quote Originally Posted by mogedon View Post
    We also have to take into account ........ the words that might be captured in a fuzzy search: "federally" for instance.
    I suspect this explains the result in your 'fuzzy' search graph. After Federation there would be the ongoing tension between the Federal Government and the State Governments (as there is still today) and I reckon the constant reference to the "Federal" Government over the decades might explain your result.

    Spearth

  8. #8
    Prolific poster
    Join Date
    Feb 2011
    Posts
    260
    I have played around with QueryPic quite a bit now and I have some further observations that others might find useful in interpreting the graphs.

    The following comments apply to the Australian (Trove) newspapers only.

    It is useful to look at the 'number of articles' graph as well as the 'proportion of total articles' graph. The 'number of articles' graph shows that the total number of articles greatly declines in the first half of the 1800s and I think this has a big effect on the resulting graphs.

    When graphing (as 'proportion of total articles' graphs) words that are significant throughout the 1800s, the graph in most cases becomes very spiky for the first half of the 1800s. I presume this is because there just isn't enough total articles to make the results statistically useful, and so relatively minor changes in the frequency of a word has a big effect on the graph. It appears to me,with the currently available newspapers, that the trends in the graphs are not particularly informative before about the mid-1850s. After that time the number of articles increases dramatically and the 'proportion of total articles' graphs become more sensible.

    I noticed that in a lot of graphs that the 'proportion of total articles' graphs showed slightly higher results in the 1800s than they did in the 1900s - regardless of what was being graphed. So I experimented with graphing some words that will probably always appear regardless of the time in history - I chose "Mr", "news" and "weather" (see the attached 'proportion of total articles' graph).
    QueryPic trends2.png

    This graph seems to back-up this idea that for some reason words that should be roughly constant over the decades are being shown with higher proportions in the 1800s (up to about 1880). The proportions then decline evenly until about 1915 and then become roughly constant again up to 1954. I have no idea why this would be - does anyone have any ideas??

    This graph also demonstrates what I was describing above. Prior to the mid-1850s the graphs become spiky and less useful.

    This graph also nicely shows the anomaly caused by the Womens Weekly after 1954 that I described in an earlier post. I have found that most words of general news interest, e.g. "flood", that you might expect would not be a particular topic of discussion in the Womens Weekly, will show as a strong negative anomaly after 1954. To prove this I tried words like "pavlova", which you would expect is mentioned commonly in the Womens Weekly, and lo and behold you get a very strong positive anomaly after 1954.

    The other thing I have noticed is that the two world wars often have a strong effect on many words in the QueryPic graphs. In the attached graph World War I only causes a slight negative anomaly for "Mr", but is often more obvious for other words, while World War II is obvious in the graphs of "news" and "weather". In most cases the two world wars cause a negative anomaly in the graphs, but as you will see here the word "news" actually causes a positive anomaly at World War II. Presumably the negative anomalies in all sorts of non-war related words is due to the fact that many more of the newspaper articles are focussed on the war and therefore diverting attention away from other news. The negative anomalies at the world wars are often more apparent in the 'number of articles' graph with steep declines in non-war related words up until the end of the war, followed by a sharp return to normal.

    Another important point, as already mentioned, is that you have to be careful about the changing use of a particular word through time. It is worth clicking on a few of the newspaper articles linked to each point in the graph to make sure you are graphing what you think you are graphing.

    I think QueryPic is fantastic, but it does take some thinking about what it is that the graphs are actually telling you. I encourage users to have a good play around ...... keeps me entertained for hours !!!

    :-)
    Attached Images Attached Images
    Last edited by Spearth; 14-05-2012 at 11:35 PM. Reason: Added clearer version of graph (trends2)

  9. #9
    Trove user mogedon's Avatar
    Join Date
    Jan 2011
    Location
    Sydney
    Posts
    41
    I wonder what the effect of Australian foreign language papers will have on Query Pic? At the moment they're limited: Suedaustralische Zeitung (Adelaide, SA : 1850 - 1851), Süd Australische Zeitung (Tanunda and Adelaide, SA : 1860 - 1874), and Il Giornale Italiano (Sydney, NSW : 1932 - 1940). But I suspect later down the track they will need to be accounted for.

  10. #10
    Trove user wragge's Avatar
    Join Date
    Jun 2010
    Location
    Canberra
    Posts
    63
    There's a new version of QueryPic available, see: http://trove.nla.gov.au/forum/showth...paper-searches
    Tim Sherratt
    @wragge on Twitter
    Words at discontents.com.au
    Experiments at wraggelabs.com and labs.nma.gov.au

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •