Website viewer

About the Australian Web Archive

On this page...

What is the Australian Web Archive?

A web archive is collection of snapshots of websites captured while they are accessible on the web, and then preserved in a static copy. 

The collection includes web material that is relevant to the cultural, social, political, research and commercial life and activities of Australia and Australians.

The material contained in the Australian Web Archive includes material collected from:

  • Over 20 years for the PANDORA Web Archive (one of the world’s first web archiving initiatives)
  • government websites formerly accessible through the Australian Government Web Archive
  • websites collected annually through large scale crawl harvests

When do we collect?

The Library collects web material throughout the year through regular scheduled harvesting of selected websites and online publications; and periodic ad hoc harvesting, including:

  • periodic bulk harvests of identified ‘seed’ URLs (particularly in respect to government websites)
  • an annual crawl harvest of the whole .au web domain
  • focused and timely collecting of websites relating to significant events (such as election campaigns)

Material will continue to be added to the Australian Web Archive and online material is collected in accordance with the Library’s function as stated in the National Library Act 1960, the legal deposit provisions of the Copyright Act 1968 and the National Library’s Collection Development Policy.

Why are some websites incomplete, missing or not functioning properly?

Websites are collected using harvest robots scheduled to run at regular intervals. This means we capture cumulative changes rather than changes as they occur on a website. We may also fail to collect content if it is removed between harvest schedules.

The technical limitations of the harvest robots also mean that some online content is not collected, such as: 

  • streaming media (including video content)
  • database content
  • dynamic and interactive elements 
  • some scripting elements required for presentation

There are also constraints on the ability to deliver (or render) content from the archive collection. Whereas the original website was delivered dynamically, the archive is a static copy and functionality such as searching, dynamic menus and other interactive functions will not work.

Websites collected for the archive may link to websites outside the scope of the Library’s collection (e.g. outside the Australian web domain) which are not included in the archive collection.

Access to some content is restricted for legal and commercial reasons.

How do I search and navigate the Australian Web Archive?

Search does not operate like Google, Internet Explorer, Edge, Mozilla Firefox, or any other search engine.

Results are displayed as a list of domains rather than webpages. Click on the links to enter the archive.

How do I view archived websites and webpages?

The web archive contains complete websites, webpages and documents. These are referred to as snapshots.  Hyperlinks within the Australian Web Archive collection will work if there is content available. 

How do I navigate the Australian Web Archive?

Use the snapshot remote to navigate the archive:

Archived websites date selector

 
• the back (<) or forward (>) arrow buttons will take you to the next or previous versions.
• the beginning (|<) or end (>|) buttons will take you to the earliest or most recent snapshots.
• the calendar allows you to move navigate the dates of when the site was captured. A hyperlink will appear and you can click on this to navigate to the version of the website that appeared on that date.
• The timeline enables you to travel through the history of the website.

Where do I find information about the archived webpages?

The sidebar at the left of the snapshot offers the following:

1. Information button – details about the snapshot you are viewing, including the title, description, date and time archived, URL, and links to related information such as a Trove catalogue record 

Achived websites details information

2. Collections button – if the website snapshot appears as part of a collection of sites, it will be listed here. In this example, the Adelaide Comedy Festival site forms part a collection of sites that were saved at a point in time about the 2012 and the 2016 Adelaide Fringe Festivals.

Archived websites collections informations

 

3. Cite button – provides handy ready-made citations for the site you are viewing in a number of standard formats including APA, MLA and Wikipedia

Archived websites citation information

What is special about the Australian Web Archive?

Users can perform full-text searches within the archive, as well as being accessible to users outside the Library’s physical buildings.

The size is also phenomenal, it combines records from the PANDORA Archived websites, AGWA and whole of “.au” web domain collections is approximately 400 terabytes across 8 billion records. This amount of data is equivalent to 12,000 32gb iPhones, which would be approximately the height of a 27-story building when stacked.

How was it built?

It was developed in house by the Library. The development of the archived websites has had a considerable focus on usability during development. Extensive user research was conducted, with the aim of making searching and navigating around the archived websites zone easier for a wide range of users, many of whom will have never visited an archived website before. This has resulted in an innovative and intuitive user interface. 

How long has it taken to build the collection?

Building the Australian Web Archive has taken over twenty years. It’s made up of three components.

1. PANDORA Australia’s Web Archive 1996-2018

The PANDORA archive was established in 1996 and was among the first national web archiving programs in the world. PANDORA was a selective web archiving program which aimed to build a representative collection of online material relating to Australia as well as collecting material associated with events such as election campaigns.

An important consideration in collecting online material for the PANDORA archive was the ability to provide access to the archived content. Prior to amendment of the Copyright Act in February 2016, which extended legal deposit provisions to online materials, the PANDORA archive only included material for which permission to collect and provide ongoing access was obtained.

In 2017, twenty-one years after it was established, the PANDORA archive was amalgamated with the Library’s other web archive collections to form the Trove web archive collection.

2. The Australian Government Web Archive 2014-2018

In May 2014 ‘whole-of-government’ arrangements were approved which provided warrant for the Library to collect Commonwealth Government websites. As a result of this, the Library established a new web archiving program, complimentary to but separate from PANDORA, which utilised a bulk web harvesting infrastructure. A prototype portal to provide access to this material was developed and made public in March 2014 and was known as the Australian Government Web Archive (AGWA). In 2017 this collection of Commonwealth Government websites was also amalgamated into the Trove web archive. Government material can still be searched using the Advanced Search option to limit to the Australian Government Web Archive.

3. Whole Australian Domain Harvests 1996-

In addition to the selective PANDORA archive and bulk harvesting of government websites, starting in 2005 the Library has conducted annual large scale harvests. These harvests have been undertaken in collaboration with the Internet Archive to ensure the Library’s collection of websites relating to Australia is as comprehensive as possible. In order to increase the comprehensive nature of the Library’s web archive, the earliest collection of websites from the .au web domain has also been obtained from the Internet Archive which expands the Library’s collection of websites dating back to 1996. In 2019 this content was made accessible to the public for the first time through Trove.

Does the National Library of Australia review all the content?

No, we may remove or refuse to display content that we reasonably believe violates our policies or the law. But that does not necessarily mean that we review content, so please don’t assume that we do.

If you find material in the National Library’s online collections which you believe infringes your rights, or, for example, contravenes privacy law, is defamatory, or in breach of copyright, you may submit a takedown request.

The web has revolutionised the way people communicate, through email, chat rooms, electronic messaging and instant information access. The web has also provided a free publishing medium, and, as a result there is a range of confronting content online.

Many disturbing websites are not ‘illegal’ which means that it is up to the viewer or a parent to monitor and manage their access.

What can I do if I am concerned about an archived webpage?

If you find material in the National Library’s online collections which you believe infringes your rights, or, for example, contravenes privacy law, is defamatory, or in breach of copyright, you may submit a takedown request.