Digital tools for big research

How Trove's API can uncover rich data from Australian collections

ResBaz Research Bazaar Sydney 2018 welcomed over 400 researchers to Macquarie University for 3 days of workshops. Just like the Turkish bazaar it was named after, participants could wander through different and exciting rooms, in this case to learn about useful digital tools for research. Everything from programming languages to platforms was showcased – all the tools for a modern-day Australian researcher.

Through the Humanities and Social Sciences stream at ResBaz, the Trove team ran a Working with the Trove API workshop. Joining friends from the Australian Data ArchiveAlveo and AURIN, the Trove team demonstrated how to get and use data sets and tools for big research projects. If you missed the workshop, the notes and exercises can be found on our Trove API Workshop [update URL] page.

Ingrid Mason tweet
Participants at the Working with the Trove API workshop, 4 July 2018. Courtesy of Ingrid Mason via Twitter.

Day 3 was Festival Day – a THATCamp (The Humanities and Technology Camp). Dr Steven McEachern – Director of the Australian Data Archive – came along with an interesting research question inspired by the earlier Trove API workshop. He’d seen Certificates of Naturalisation printed in Trove's digitised Government Gazettes. [update URL] These are records of the names, dates and addresses of people who became Australian citizens.

CERTIFICATES OF NATURALISATION, Commonwealth of Australia Gazette
CERTIFICATES OF NATURALISATION, Commonwealth of Australia Gazette (National:1901-1973), 23 Nov 1967, p. 6371. [update URL]

These articles are especially interesting to family history researchers, but Dr McEachern pointed out they are also part of a fabulous demographic data set waiting to be tapped. Could Government Gazettes be used to track more broadly when and where people became Australian citizens?

The answer was 'yes' – with researchers, librarians and data managers all working together. It took only two sessions to go from a simple research question to a completed visualisation.

The steps were:

  1. Narrow down the articles needed from the Government Gazettes. A search for the phrase "Certificate of Naturalisation" in just the Commonwealth of Australia Government Gazette [updte URL] revealed 683 records.
  2. Extract those records from the Trove API, and collate the individual records into a shareable file.
  3. Upload and share the records on CloudStor – AARNet’s cloud platform for storing and sharing research data. Each file was about 15 Mb. Everyone in the workshop could then freely access them, downloading them to their own laptops to start interrogating the data. The data is still there and you can access it too – in XML or JSON format.
  4. Import the file into OpenRefine and then NotePad++. The group plugged one laptop into the big screen and worked for an hour on a series of regular expressions that consecutively cleaned up the data. With some clever insertion of line breaks, removal of commas and normalisation of dates we eventually had a clean CSV file. It consisted of last names, first names, street addresses including suburb, and dates of naturalisation.
  5. Import the CSV into Excel and do some further tidying. This included removing about 500 rows of data we had accidentally captured.
  6. Use Excel’s "pivot table” functionality to create a graph of how many people were naturalised every month.

After a few hours' work, the group had turned 683 articles into more than 55,000 rows, one for each individual person. A graph was produced, showing what Naturalisations looked like by month in the late 1960s. Once it was done, it took only a minute to begin speculating why there were very high spikes in certain months, particularly late January every year. Can you guess why?

Data visualisation of Naturalisations by month over 3 years in the late 1960s.
Data visualisation of Naturalisations by month over 3 years in the late 1960s.

This data set is now ripe for other work – perhaps looking at the suburbs people were living in when they were naturalised and how this changed over time. This graph was a great example of team work: researchers, librarians and data managers all bringing their skills together to solve a research question. And now it begs the further question – what other social science data might be buried in the Australian Government Gazettes, just waiting for a researcher to jump in and start asking the right questions?

If you’re interested in working with the Australian Government Gazettes, you can access them via the Trove API or they will shortly become available as a single data set on CloudStor. For access to the data directly via your CloudStor Jupyter Notebook or to download the files contact the Trove team. [link to contact us]