net.nutch.util
Class NutchGenericFileSystem

java.lang.Object
  net.nutch.util.NutchGenericFileSystem

All Implemented Interfaces:

NutchFileSystem

Direct Known Subclasses:

NutchNFSFileSystem, NutchRemoteFileSystem

public abstract class NutchGenericFileSystem

extends Object

implements NutchFileSystem

NutchGenericFileSystem implements the NutchFileSystem interface and adds some generic utility methods for subclasses to use. The standard task any implementor of NutchFileSystem

Author:

Mike Cafarella

Constructor Summary

NutchGenericFileSystem(File dbRoot, ShareSet shareSet, boolean destructivePut)
Create a Nutch Filesystem at the indicated mounted directory.

Method Summary

void	close() Close down the Generic File System
void	completeDir(NutchFile nutchFile) Complete the given directory
protected abstract void	copyFile(File srcFile, String locationMach, String locationStr, String nutchFileName, boolean overwrite) To be implemented by subclasses
void	delete(NutchFile nutchFile) Take the file out of the NutchFileSystem.
protected abstract void	deleteFile(String locationMach, String locationStr, String nutchFileName)
File	get(NutchFile nutchFile) Wait for a NutchFile from somewhere in NutchSpace.
File	get(NutchFile nutchFile, long timeout) Wait for a NutchFile for the specified amount of time.
File	getWorkingFile() Acquire a real File for a name that's not yet under NutchFS control.
void	lock(NutchFile nutchFile, boolean exclusive) Obtain a lock with the given NutchFile.
protected abstract void	lockFile(String locMach, String locStr, String filename, boolean exclusive)
void	put(NutchFile nutchFile, File workingFile, boolean overwrite) Add a single file or a directory of files to the filesystem.
void	release(NutchFile nutchFile) Release the lock for the given NutchFile
protected abstract void	release(String locMach, String locStr, String filename)
protected abstract void	renameFile(File srcFile, String locationMach, String locationStr, String nutchFileName, boolean overwrite)
void	renameTo(NutchFile src, NutchFile dst) Rename the thing.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

NutchGenericFileSystem

public NutchGenericFileSystem(File dbRoot,
                              ShareSet shareSet,
                              boolean destructivePut)
                       throws IOException

Create a Nutch Filesystem at the indicated mounted directory.

Method Detail

getWorkingFile

public File getWorkingFile()
                    throws IOException

Acquire a real File for a name that's not yet under NutchFS control. This may improve performance later on when the File is put() under NutchFS control. It's also handy for finding a file location where there is a lot of extra room.

Specified by:

getWorkingFile in interface NutchFileSystem

Throws:

IOException

get

public File get(NutchFile nutchFile)
         throws IOException

Wait for a NutchFile from somewhere in NutchSpace. Translate it to a regular old filesystem File. The file should already be in place. So we wait until it is.

Specified by:

get in interface NutchFileSystem

Throws:

IOException

get

public File get(NutchFile nutchFile,
                long timeout)
         throws IOException

Wait for a NutchFile for the specified amount of time. Return null if we don't get it before 'timeout' ms have elapsed.

Specified by:

get in interface NutchFileSystem

Throws:

IOException

lock

public void lock(NutchFile nutchFile,
                 boolean exclusive)
          throws IOException

Obtain a lock with the given NutchFile. This might mean obtaining locks across many different machines/filesystems. That's fine, as long as every machine always obtains the locks in a standard ordering.

Specified by:

lock in interface NutchFileSystem

Throws:

IOException

release

public void release(NutchFile nutchFile)
             throws IOException

Release the lock for the given NutchFile

Specified by:

release in interface NutchFileSystem

Throws:

IOException

put

public void put(NutchFile nutchFile,
                File workingFile,
                boolean overwrite)
         throws IOException

Add a single file or a directory of files to the filesystem. If the source File is a directory, we want to reproduce the entire directory structure, rooted at the given NutchFile.

Specified by:

put in interface NutchFileSystem

Throws:

IOException

completeDir

public void completeDir(NutchFile nutchFile)
                 throws IOException

Complete the given directory

Specified by:

completeDir in interface NutchFileSystem

Throws:

IOException

delete

public void delete(NutchFile nutchFile)
            throws IOException

Take the file out of the NutchFileSystem.

Specified by:

delete in interface NutchFileSystem

Throws:

IOException

renameTo

public void renameTo(NutchFile src,
                     NutchFile dst)
              throws IOException

Rename the thing. Usually done at close.

Specified by:

renameTo in interface NutchFileSystem

Throws:

IOException

close

public void close()
           throws IOException

Close down the Generic File System

Specified by:

close in interface NutchFileSystem

Throws:

IOException

copyFile

protected abstract void copyFile(File srcFile,
                                 String locationMach,
                                 String locationStr,
                                 String nutchFileName,
                                 boolean overwrite)
                          throws IOException

To be implemented by subclasses

Throws:

IOException

deleteFile

protected abstract void deleteFile(String locationMach,
                                   String locationStr,
                                   String nutchFileName)
                            throws IOException

Throws:

IOException

renameFile

protected abstract void renameFile(File srcFile,
                                   String locationMach,
                                   String locationStr,
                                   String nutchFileName,
                                   boolean overwrite)
                            throws IOException

Throws:

IOException

lockFile

protected abstract void lockFile(String locMach,
                                 String locStr,
                                 String filename,
                                 boolean exclusive)
                          throws IOException

Throws:

IOException

release

protected abstract void release(String locMach,
                                String locStr,
                                String filename)
                         throws IOException

Throws:

IOException