Page 1 of 2 12 LastLast
Results 1 to 10 of 16

Thread: how do i ask for Fuzzy search instead of Exact search

  1. #1
    Member
    Join Date
    Feb 2011
    Location
    Melbourne
    Posts
    10

    Question how do i ask for Fuzzy search instead of Exact search

    dear people

    how do i get the fuzzy search to happen ?

    i want to find articles where the OCR/text recognition was not quite perfect.

    my searches only return exact matches - eg, a search for "wombat" only shows the articles where "wombat" is spelt correctly, and ignores the articles with "wombut", "wumbat", etc

    : )

    ps. please forgive me if i've missed the obvious. i've read the help pages and scanned the forums, but can't see the solution.

  2. #2
    Staff Moderator mraadgev [NLA]'s Avatar
    Join Date
    Jul 2010
    Posts
    518
    Hi snu3

    Trove, by default does a level of fuzzy searching by default, e.g. a search for Wombat will return Wombat, Wombats, Wombate, and anything else starting with Wombat, although it does not necessarily return the variations on Wombat - e.g. wombut or wumbat. Unfortunately there is no simple way to tell trove to do a fuzzy search where all the possible variations on a word are searched.

    This means that you will need to either do a wildcard search with the start of the word (e.g. womb*), an OR search with the variations that you have found (e.g. wombat OR wumbat OR wombut) or a combination of the two (if there is a common string, e.g. womb* OR wumb*).

    Sorry about this (if I find out otherwise that it can be done I'll let you know...)

    Mark
    Mark
    Trove Support
    National Library of Australia

  3. #3
    Member
    Join Date
    Feb 2011
    Location
    Melbourne
    Posts
    10
    hi mark

    thanks for the quick & helpful reply.

    i'll go and play with the wildcard searches & combinations, to find those missing wombats.

    snu : )

  4. #4
    Prolific poster
    Join Date
    Aug 2011
    Posts
    257
    Sometimes I wish that Trove would not try to second-guess me. When I was searching for "Irmer", I really did not want to wade through "f-armer" "f-irmer" "f-ormer", etc. On the other hand, since it usually appeared as "Inner", I suppose it was a good thing. One just has to cope with it.

  5. #5
    Prolific poster
    Join Date
    Feb 2011
    Posts
    332
    Hi Catessa,

    There actually is a way to prevent the fuzzy search if you want, and to do an exact search. For example:

    1) fulltext:irmer
    2) fulltext:"john irmer"
    3) fulltext:"john irmer"~2

    1) will only find exact matches for "irmer"
    2) will only find exact matches for the phrase "john irmer" [see NOTE below]
    3) will only find exact matches for "john" and "irmer" within two words of each other.

    [NOTE: as of 2014, fulltext:"john irmer" automatically defaults to the same as fulltext:"john irmer"~1 — i.e. there can be up to one word, initial, etc., between "john" and "irmer" in the search results. To find only the exact phrase "john irmer" now requires the following:
    fulltext:"john irmer"~0 ]

    Of course these will not pick up the mistranslations, like "Inner" as you point out. But if you are aware of the common mistranslations then you could try the following:

    fulltext:irmer OR fulltext:inner

    (Though this may be a bad example because there are bound to be a large number of "inner" words, unless you narrow the search down by date, newspaper, etc.)



    However, if I understand Mark's explanation above correctly then the fuzzy search for "Irmer" should not be finding "firmer", "farmer" or "former" anyway.

    I just had quick test of these searches and the search results for 'fulltext:irmer' do actually include results that show "firmer", "farmer" and "former" in the snippets. But on closer inspection you will see that the electronically translated text (mistranslated in this case) does actually include "irmer". The snippet shown in the search results is getting the "firmer", "farmer" or "former" from the preview text (correctly translated) for the first four lines, which are supplied with each article. I notice that "firmer" is very common on market reports and obviously commonly gets mistranslated as "irmer".


    To get around this you can do the following to exclude results containing "firmer", "farmer" and "former":

    fulltext:irmer -firmer -farmer -former

    This gives you 1,338 results, as opposed to 2,564 results for just 'fulltext:irmer'.

    The catch is that with this method you may exclude articles that really do contain the name "Irmer" and also happen to contain the words "firmer", "farmer" or "former" or fuzzy derivatives of these three words. So an improvement that does not exclude fuzzy derivatives of "firmer", "farmer" or "former" would be the following:

    fulltext:irmer -fulltext:firmer -fulltext:farmer -fulltext:former
    (This returns 1,634 search results.)

    This actually does the same as the following—
    fulltext:irmer NOT (fulltext:firmer OR fulltext:farmer OR fulltext:former)


    Since a big part of the unwanted search results relate to mistranslations of the word "firmer" in market reports, you can drastically reduce the number of results by excluding words related to market reports, for example:

    fulltext:irmer NOT (fulltext:firmer OR fulltext:farmer OR fulltext:former OR market OR stock OR shares OR quotations OR trade OR sales OR prices OR commercial)

    This last search only returns 537 results, but again you could possibly be excluding some results that actually contain the name "Irmer". But at least it would be a good starting point to search for the name "Irmer" without most of the unwanted market reports to look through. You could add 'fulltext:' in front of each of the market related words if you wanted, but then while excluding "market" you will still get "markets", etc. You may need to play around to see which search string gives you the most workable number of search results.



    Something else that may interest you, and is more relevant to the original question of the post (though will not solve Snu's wombat problem) —


    If you have searched through all the results in a search for a particular name in its exact form, e.g. fulltext:newman, but then you want to check out the fuzzy search options without having to look through all the exact search options again, then you can do the following:

    newman -fulltext:newman

    So this will do a fuzzy search for "newman", but exclude all exact matches for newman". However, I presume the catch with this is that if a particular article includes both a fuzzy match and an exact match for "newman" then it will be excluded from the results.


    Hope that all helps,

    Spearth
    Last edited by Spearth; 10-08-2014 at 12:53 AM. Reason: 2014 change Noted

  6. #6
    Prolific poster
    Join Date
    Aug 2011
    Posts
    257
    I shall have to print out your answer and see how it works out in practice. (I could not possibly remember them while trying to use them.) I should have tried out the - trick earlier.
    With some of the other examples. the cure could be worse than the disease.

  7. #7
    Prolific poster
    Join Date
    Feb 2011
    Posts
    332
    Hi Catessa,

    It may take a little bit of study to get your head around how these search operations work, but it is well worth the effort — the cure is definitely much better once it is understood. The search trickery can save you hours of searching through irrelevant search results.

    For difficult searches, putting some thought into the 'search string' beforehand can often get you straight to the articles you want with very little sifting through irrelevant articles. It can be hard to anticipate what other irrelevant things a search string might turn up, so I usually do the basic search first and spend a minute flicking through the search results. This quickly shows the common "false positives" and so I then modify my search string to suit and hey presto, the articles you are looking for are often head of the list (or on the first page at least - assuming an ordering by 'Relevance').

    The search operators ('boolean' operators as they are called) are actually quite simple, you just have to think very logically. This is where that stuff that we learnt in maths back in secondary school, and wondered when we would ever need to use, comes back to haunt us !!

    AND - searching for both words, the default when words are listed in the search box
    OR - searching for any of the words
    ( ) - used the same way as you would in maths
    NOT (same as "-") - used to exclude certain words
    ~3 - searching for words near each each other (with up to three words in between in this example)
    fulltext: - exact matches only for words
    publictag: - searching for words in public tags only
    firstpageseq: - searching for a word in a particular sequential page number

    These are explained more fully here:
    http://trove.nla.gov.au/general/newspapers_help

    You have probably noticed that if you use the 'Advanced Search', then when you get your search results the terms you put in the boxes are translated into a 'Boolean' search string at the top of the page. But being clever with these operators can actually allow you to do more effective searches than the 'Advanced Search' allows.


    Spearth
    Last edited by Spearth; 19-12-2011 at 04:59 PM.

  8. #8
    Prolific poster
    Join Date
    Aug 2011
    Posts
    257
    I shall try to work through these suggestions after the Silly Season. At the moment, I am not doing personal research, just ambling about, correcting a dozen or so articles on one topic, then getting bored and following up some other lead. It does not matter much what I work on, as you can never second-guess what somebody might want. Current series on which I am working started out with looking for the date when the rock bar blocking the entrance to Fremantle Harbour was dynamited. That led me to campaign speeches for the first parliament elected under Responsible Government in WA (1890), which led me on to State Education Bursaries, and I have just read that school attendance had been down in (1893?) because of the small-pox epidemic. First I had heard of any smallpox epidemic in WA. My curiosity is tickled. I shall have to hunt that up after finishing my current article, then going back to the harbour bar.

  9. #9
    Member
    Join Date
    Nov 2010
    Location
    Adelaide, South Australia
    Posts
    25
    I am still struggling to understand.

    I do a simple search for Schuetze in South Australia for the full year 1940. We get 122 results, and apart from the desired Schuetze (3 in the first 20 results), we see offered the fuzzy names Schutz, Schutze etc.
    I then search for fulltext:Schuetze and alarmingly there are no results, even though I know there are at least 3 results!
    Interestingly fulltext:Schutz produces what you'd expect.
    Then, if I search for Schuetze NOT (fulltext:Schutz) I get 17 results - including Schuetze, Schutze, Schuetz.
    But if I enter Schuetze NOT (Schutz OR Schutze) I get nothing!
    Similarly if I enter Schuetze NOT (fulltext:Schutz OR fulltext:Schutze OR fulltext:Schuetz) I get nothing.

    Any advice, please?

    David Wilson

  10. #10
    Staff Moderator mraadgev [NLA]'s Avatar
    Join Date
    Jul 2010
    Posts
    518
    Hi David,

    There is a small issue related to the schuetze search that we are currently working to resolve. Unfortunately I can't provide any specific information on when this fix will be applied.

    Regards
    Mark
    Mark
    Trove Support
    National Library of Australia

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •