org.dlese.dpc.oai.harvester

Class Harvester

java.lang.Object

org.dlese.dpc.oai.harvester.Harvester

All Implemented Interfaces:

org.xml.sax.ErrorHandler

public class Harvester
extends java.lang.Object
implements org.xml.sax.ErrorHandler

Harvests metadata from an OAI data provider, saving the results to file or returning the raw XML as an array of Strings. Supports data providers that use resumption tokens for flow control , selective harvesting by date or set , gzip response compression and other protocol features. Supports OAI protocol versions 1.1 and 2.0 .

To perform a harvest, use one of the following methods:

• The static harvest method (for general use): harvest(java.lang.String, java.lang.String, java.lang.String, java.util.Date, java.util.Date, java.lang.String, boolean, org.dlese.dpc.oai.harvester.HarvestMessageHandler, org.dlese.dpc.oai.harvester.OAIChangeListener, boolean, boolean, boolean, int)}
  
  
• The static main method (for command-line use): main(java.lang.String[])
  
  
• The non-static doHarvest method (provides a few additional options): doHarvest(java.lang.String, java.lang.String, java.lang.String, java.util.Date, java.util.Date, java.lang.String, boolean, java.lang.String, java.lang.String, boolean, boolean, boolean).
  
  

Use of this API assumes familiarity with the OAI protocol .

Author:

Steve Sullivan, John Weatherley

See Also:

HarvestMessageHandler, OAIChangeListener

Constructor Summary

Constructors

Constructor and Description
Harvester() Creates a Harvester that uses no HarvestMessageHandler or OAIChangeListener.
Harvester(HarvestMessageHandler msgHandler, OAIChangeListener oaiChangeListener, int timeOutMilliseconds) Creates a Harvester that uses the given HarvestMessageHandler.

Method Summary

Modifier and Type	Method and Description
java.lang.String[][]	doHarvest(java.lang.String baseURL, java.lang.String metadataPrefix, java.lang.String setSpec, java.util.Date from, java.util.Date until, java.lang.String outdir, boolean splitBySet, java.lang.String zipName, java.lang.String zDir, boolean writeHeaders, boolean harvestAll, boolean harvestAllIfNoDeletedRecord) Performs the harvest.
void	error(org.xml.sax.SAXParseException exc) Handles errors.
void	fatalError(org.xml.sax.SAXParseException exc) Handles fatal errors.
long	getEndTime() Gets the endTime when the havest completed either because of an error or at the end of a successful harvest.
java.lang.String	getHarvestedRecordsDir() Gets the harvestedRecordsDir attribute of the Harvester object
long	getHarvestUid() Returns a unique ID for this harvest.
int	getNumRecordsHarvested() Gets the current number of records that have been harvested by this harvester.
int	getNumResumptionTokensIssued() Gets the number of resumption tokens that have currently been issued by the data provider.
long	getStartTime() Gets the startTime when the harvest began, or 0 if it has not begun yet.
static java.lang.String[][]	harvest(java.lang.String baseURL, java.lang.String metadataPrefix, java.lang.String setSpec, java.util.Date from, java.util.Date until, java.lang.String outdir, boolean splitBySet, HarvestMessageHandler msgHandler, OAIChangeListener oaiChangeListener, boolean writeHeaders, boolean harvestAll, boolean harvestAllIfNoDeletedRecord, int timeOutMilliseconds) Harvest the given provider, saving the resulting metadata to file or returning the results as an array of Strings.
boolean	isRunning() Determines whether this Harvester is currently running or not.
void	kill() Gracefully kills the harvest after the current record is finished being harvested.
static void	main(java.lang.String[] args) Command line interface for the harvester.
static void	setDebug(boolean db) Sets the debug attribute object
void	setNumRecordsForNotification(int numRecords) Sets the number of records harvested before statusMessage notifications to the HarvestMessageHandler are made.
void	warning(org.xml.sax.SAXParseException exc) Handles warnings.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Harvester

public Harvester()

Creates a Harvester that uses no HarvestMessageHandler or OAIChangeListener.

Harvester

public Harvester(HarvestMessageHandler msgHandler,
                 OAIChangeListener oaiChangeListener,
                 int timeOutMilliseconds)

Creates a Harvester that uses the given HarvestMessageHandler.

Parameters:

msgHandler - The HarvestMessageHandler that will receive messages as the harvest progresses, or null if none.

oaiChangeListener - The OAIChangeListener that will recieve notifications, or null for none.

timeOutMilliseconds - Number of milliseconds the harvester will wait for a response from the data provider before timing out

Method Detail

main

public static void main(java.lang.String[] args)

Command line interface for the harvester. Harvest status messages are output to standard out.

Arguments (required arguments must be in this order, optional arguments may be in any order):

• outdir (required) - Path to the directory to write the harvested record files, for example "." or "/home/user/harvested_files"
• baseURL (required) - Base URL to harvest from, for example "http://www.dlese.org/oai/provider"
• metadataPrefix (required) - The metadata prefix, for example "oai_dc"
• [ -set:setSpec ] (optional) - The set to harvest, for example -set:myset
• [ -from:fromDate ] (optional) - The harvest from date, for example, -from:2003-12-31T23:59:59Z
• [ -until:untilDate ] (optional) - The harvest until date, for example, -until:2004-12-31T23:59:59Z
• [ -splitBySet:true|False ] (optional) - True to save each record in separate directories split by set inside outdir, false to save all records to the root of outdir (default is false)
• [ -writeHeaders:true|False ] (optional) - True to have OAI headers written to the output, false not to (default is false)

Parameters:

args - The command line arguments

harvest

public static java.lang.String[][] harvest(java.lang.String baseURL,
                                           java.lang.String metadataPrefix,
                                           java.lang.String setSpec,
                                           java.util.Date from,
                                           java.util.Date until,
                                           java.lang.String outdir,
                                           boolean splitBySet,
                                           HarvestMessageHandler msgHandler,
                                           OAIChangeListener oaiChangeListener,
                                           boolean writeHeaders,
                                           boolean harvestAll,
                                           boolean harvestAllIfNoDeletedRecord,
                                           int timeOutMilliseconds)
                                    throws Hexception,
                                           OAIErrorException

Harvest the given provider, saving the resulting metadata to file or returning the results as an array of Strings. A HarvestMessageHandler may be specified to capture harvest progress messages. Use a SimpleHarvestMessageHandler to have harvest messages sent to standard out. A OAIChangeListener may be specified to recieve messages about chages to harvested records.

Parameters:

baseURL - The baseURL of the data provider, for example "http://www.dlese.org/oai/provider"

metadataPrefix - The metadataPrefix, for example "oai_dc"

setSpec - The set to harvest, for example "testset", or null to harvest all sets

from - The from date, for example "2003-12-31T23:59:59Z", or null for none

until - The until date, for example "2003-12-31T23:59:59Z", or null for none

outdir - The path of output dir. If null or "", we return the String[][] array; if specified we return null

msgHandler - A handler for status messages that occur during the harvest, or null to ingnore messages

oaiChangeListener - The OAIChangeListener that will recieve notifications, or null for none

writeHeaders - True to have OAI headers written to the output, false not to

harvestAll - True to delete previous harvested record files and harvest all records again from scratch; false to preserve previous record files and replace or delete only those that have changed

harvestAllIfNoDeletedRecord - True to harvest all record files from scratch if deleted records are not supported

splitBySet - True to save each record in separate directories split by set inside outdir, false to save all records to the root of outdir

timeOutMilliseconds - Number of milliseconds the harvester will wait for a response from the data provider before timing out

Returns:

If outdir is specified returns null; if outdir is null or "", returns one row for each record harvested. Each row has two elements:

• identifier, encoded
• content xml record, or the String deleted if status=deleted.

Throws:

Hexception - If serious error

OAIErrorException - If OAI error

kill

public void kill()

Gracefully kills the harvest after the current record is finished being harvested.

setNumRecordsForNotification

public void setNumRecordsForNotification(int numRecords)

Sets the number of records harvested before statusMessage notifications to the HarvestMessageHandler are made.

Parameters:

numRecords - The new numRecordsForNotification value

getStartTime

public long getStartTime()

Gets the startTime when the harvest began, or 0 if it has not begun yet.

Returns:

The startTime, or 0 if not started yet.

getHarvestedRecordsDir

public java.lang.String getHarvestedRecordsDir()

Gets the harvestedRecordsDir attribute of the Harvester object

Returns:

The harvestedRecordsDir value

getHarvestUid

public long getHarvestUid()

Returns a unique ID for this harvest.

Returns:

The harvestId value

getEndTime

public long getEndTime()

Gets the endTime when the havest completed either because of an error or at the end of a successful harvest. Returns 0 if the harvest is still in progress.

Returns:

The endTime, or 0 if the harvest is still in progress.

getNumRecordsHarvested

public int getNumRecordsHarvested()

Gets the current number of records that have been harvested by this harvester. This number increases as the harvest progresses.

Returns:

The numRecordsHarvested value

getNumResumptionTokensIssued

public int getNumResumptionTokensIssued()

Gets the number of resumption tokens that have currently been issued by the data provider. This number increases as the harvest progresses. This number gives a rough indication of the progression and duration of the harvest.

Returns:

The numResumptionTokensIssued value.

isRunning

public boolean isRunning()

Determines whether this Harvester is currently running or not.

Returns:

True if the harvest is in progress, false otherwise.

doHarvest

public java.lang.String[][] doHarvest(java.lang.String baseURL,
                                      java.lang.String metadataPrefix,
                                      java.lang.String setSpec,
                                      java.util.Date from,
                                      java.util.Date until,
                                      java.lang.String outdir,
                                      boolean splitBySet,
                                      java.lang.String zipName,
                                      java.lang.String zDir,
                                      boolean writeHeaders,
                                      boolean harvestAll,
                                      boolean harvestAllIfNoDeletedRecord)
                               throws Hexception,
                                      OAIErrorException

Performs the harvest. Note that his method is not safe for multiple harvests - a separate Harvester instance should be created for each havest performed.

Parameters:

metadataPrefix - metadataPrefix. e.g., "oai_dc", or null to harvest all formats

setSpec - set. e.g., "testset" or null for none.

from - from date. May be null.

until - until date. May be null.

outdir - path of output dir. If null or "", we return the String[][] array; if specified we return null.

writeHeaders - True to have oai headers written to file, false not to.
The directory structure under outdir is:
outdir/set/subset/subset/metadataPrefix/oaiId_hdr.xml OAI header
outdir/set/subset/subset/metadataPrefix/oaiId_data.xml OAI contents

baseURL - The baseURL of the data provider.

splitBySet - To split set

zipName - Name of the zip file to save to, or null for no zipping

zDir - Directory of the zipfile

harvestAll - True to delete previous harvested records and harvest all records again from scratch

harvestAllIfNoDeletedRecord - True to harvest all records from scratch if deleted records are not supported

Returns:

If outdir is specified returns null; if outdir is null or "", returns one row for each record harvested. Each row has two elements:

• identifier, encoded
• content xml record.

Throws:

Hexception - If serious error.

OAIErrorException - If OAI error was returned by the data provider.

setDebug

public static void setDebug(boolean db)

Sets the debug attribute object

Parameters:

db - The new debug value

fatalError

public void fatalError(org.xml.sax.SAXParseException exc)

Handles fatal errors. Part of ErrorHandler interface.

Specified by:

fatalError in interface org.xml.sax.ErrorHandler

Parameters:

exc - The Exception thrown

error

public void error(org.xml.sax.SAXParseException exc)

Handles errors. Part of ErrorHandler interface.

Specified by:

error in interface org.xml.sax.ErrorHandler

Parameters:

exc - The Exception thrown

warning

public void warning(org.xml.sax.SAXParseException exc)

Handles warnings. Part of ErrorHandler interface.

Specified by:

warning in interface org.xml.sax.ErrorHandler

Parameters:

exc - The Exception thrown