nl.esciencecenter.xenon.adaptors.filesystems.hdfs

Class HDFSFileSystem

java.lang.Object

nl.esciencecenter.xenon.filesystems.FileSystem

nl.esciencecenter.xenon.adaptors.filesystems.hdfs.HDFSFileSystem

All Implemented Interfaces:

java.lang.AutoCloseable

public class HDFSFileSystem
extends FileSystem

Field Summary

Fields

Modifier and Type	Field and Description
(package private) boolean	closed
private java.lang.String	ERROR_MSG
(package private) org.apache.hadoop.fs.FileSystem	fs

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	HDFSFileSystem(java.lang.String uniqueID, java.lang.String endPoint, Credential credential, org.apache.hadoop.fs.FileSystem fs, int bufferSize, XenonProperties properties)

Method Summary

Modifier and Type	Method and Description
java.io.OutputStream	appendToFile(Path path) Open an existing file and return an OutputStream to append data to this file.
(package private) void	checkClosed()
void	close() Close this FileSystem.
void	createDirectory(Path dir) Creates a new directory, failing if the directory already exists.
void	createFile(Path file) Creates a new empty file, failing if the file already exists.
void	createSymbolicLink(Path link, Path target) Creates a new symbolic link, failing if the link already exists (optional operation).
void	delete(Path path, boolean recursive) Deletes an existing path.
protected void	deleteDirectory(Path path) Delete an empty directory.
protected void	deleteFile(Path file) Delete a file.
boolean	exists(Path path) Tests if a path exists.
(package private) static Path	fromHDFSPath(org.apache.hadoop.fs.Path p)
PathAttributes	getAttributes(Path path) Get the PathAttributes of an existing path.
boolean	isOpen() Return if the connection to the FileSystem is open.
java.lang.Iterable<PathAttributes>	list(Path dir, boolean recursive) List all entries in the directory dir.
protected java.lang.Iterable<PathAttributes>	listDirectory(Path dir) Return the list of entries in a directory.
private java.util.Iterator<PathAttributes>	listIterator(Path dir)
private java.util.Iterator<PathAttributes>	listIteratorNoException(Path dir)
java.io.InputStream	readFromFile(Path path) Open an existing file and return an InputStream to read from this file.
Path	readSymbolicLink(Path link) Reads the target of a symbolic link (optional operation).
void	rename(Path source, Path target) Rename an existing source path to a non-existing target path (optional operation).
void	setPosixFilePermissions(Path path, java.util.Set<PosixFilePermission> permissions) Sets the POSIX permissions of a path (optional operation).
(package private) org.apache.hadoop.fs.Path	toHDFSPath(Path p)
private PathAttributes	toPathAttributes(org.apache.hadoop.fs.FileStatus s)
private java.util.Set<PosixFilePermission>	toPermissionSet(org.apache.hadoop.fs.permission.FsPermission permission)
java.io.OutputStream	writeToFile(Path path) Open a file and return an OutputStream to write to this file.
java.io.OutputStream	writeToFile(Path path, long size) Open a file and return an OutputStream to write to this file.

Methods inherited from class nl.esciencecenter.xenon.filesystems.FileSystem

areSamePaths, assertDirectoryExists, assertFileExists, assertFileIsSymbolicLink, assertIsOpen, assertNotNull, assertParentDirectoryExists, assertPathExists, assertPathIsDirectory, assertPathIsFile, assertPathIsNotDirectory, assertPathNotExists, cancel, copy, copyFile, copySymbolicLink, create, create, create, create, createDirectories, equals, getAdaptorDescription, getAdaptorDescriptions, getAdaptorName, getAdaptorNames, getCredential, getLocation, getPathSeparator, getProperties, getStatus, getWorkingDirectory, hashCode, isDotDot, list, performCopy, setWorkingDirectory, streamCopy, toAbsolutePath, waitUntilDone

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Field Detail

fs

final org.apache.hadoop.fs.FileSystem fs

closed

boolean closed

ERROR_MSG

private final java.lang.String ERROR_MSG

See Also:

Constant Field Values

Constructor Detail

HDFSFileSystem

protected HDFSFileSystem(java.lang.String uniqueID,
                         java.lang.String endPoint,
                         Credential credential,
                         org.apache.hadoop.fs.FileSystem fs,
                         int bufferSize,
                         XenonProperties properties)

Method Detail

checkClosed

void checkClosed()
          throws XenonException

Throws:

XenonException

close

public void close()
           throws XenonException

Description copied from class: FileSystem

Close this FileSystem. If the adaptor does not support closing this is a no-op.

Specified by:

close in interface java.lang.AutoCloseable

Overrides:

close in class FileSystem

Throws:

XenonException - If the FileSystem failed to close or if an I/O error occurred.

isOpen

public boolean isOpen()
               throws XenonException

Description copied from class: FileSystem

Return if the connection to the FileSystem is open. An adaptor which does not support closing is always open.

Specified by:

isOpen in class FileSystem

Returns:

if the connection to the FileSystem is open.

Throws:

XenonException - if the test failed or an I/O error occurred.

rename

public void rename(Path source,
                   Path target)
            throws XenonException

Description copied from class: FileSystem

Rename an existing source path to a non-existing target path (optional operation).

This method only implements a rename operation, not a move operation. Hence, this method will not copy files and should return (almost) instantaneously. The parent of the target path (e.g. target.getParent) must exist. If the target is equal to the source this method has no effect. If the source is a link, the link itself will be renamed, not the path to which it refers. If the source is a directory, it will be renamed to the target. This implies that a moving a directory between physical locations may fail.

Specified by:

rename in class FileSystem

Parameters:

source - the existing source path.

target - the non existing target path.

Throws:

UnsupportedOperationException - If the adapter does not support renaming.

NoSuchPathException - If the source file does not exist or the target parent directory does not exist.

PathAlreadyExistsException - If the target file already exists.

NotConnectedException - If file system is closed.

XenonException - If the move failed.

createDirectory

public void createDirectory(Path dir)
                     throws XenonException

Description copied from class: FileSystem

Creates a new directory, failing if the directory already exists. The parent directory of the file must already exists.

Specified by:

createDirectory in class FileSystem

Parameters:

dir - the directory to create.

Throws:

PathAlreadyExistsException - If the directory already exists.

NoSuchPathException - If the parent directory does not exist.

NotConnectedException - If file system is closed.

XenonException - If an I/O error occurred.

createFile

public void createFile(Path file)
                throws XenonException

Description copied from class: FileSystem

Creates a new empty file, failing if the file already exists. The parent directory of the file must already exists.

Specified by:

createFile in class FileSystem

Parameters:

file - a path referring to the file to create.

Throws:

PathAlreadyExistsException - If the file already exists.

NoSuchPathException - If the parent directory does not exist.

NotConnectedException - If file system is closed.

XenonException - If an I/O error occurred.

createSymbolicLink

public void createSymbolicLink(Path link,
                               Path target)
                        throws XenonException

Description copied from class: FileSystem

Creates a new symbolic link, failing if the link already exists (optional operation). The target is taken as is. It may be absolute, relative path and/or non-normalized path and may or may not exist.

Specified by:

createSymbolicLink in class FileSystem

Parameters:

link - the symbolic link to create.

target - the target the symbolic link should refer to.

Throws:

PathAlreadyExistsException - If the link already exists.

NoSuchPathException - If the target or parent directory of link does not exist

InvalidPathException - If parent of link is not a directory

NotConnectedException - If file system is closed.

XenonException - If an I/O error occurred.

delete

public void delete(Path path,
                   boolean recursive)
            throws XenonException

Description copied from class: FileSystem

Deletes an existing path. If path is a symbolic link the symbolic link is removed and the symbolic link's target is not deleted. If the path is a directory and recursive is set to true, the contents of the directory will also be deleted. If recursive is set to false, a directory will only be removed if it is empty.

Overrides:

delete in class FileSystem

Parameters:

path - the path to delete.

recursive - if the delete must be done recursively

Throws:

DirectoryNotEmptyException - if the directory was not empty (and the delete was not recursive).

NoSuchPathException - if the provided path does not exist.

NotConnectedException - If file system is closed.

XenonException - if an I/O error occurred.

exists

public boolean exists(Path path)
               throws XenonException

Description copied from class: FileSystem

Tests if a path exists.

Specified by:

exists in class FileSystem

Parameters:

path - the path to test.

Returns:

If the path exists.

Throws:

NotConnectedException - If file system is closed.

XenonException - if an I/O error occurred.

list

public java.lang.Iterable<PathAttributes> list(Path dir,
                                               boolean recursive)
                                        throws XenonException

Description copied from class: FileSystem

List all entries in the directory dir. All entries in the directory are returned, but subdirectories will not be traversed by default. Set recursive to true, include the listing of all subdirectories. Symbolic links are not followed.

Overrides:

list in class FileSystem

Parameters:

dir - the target directory.

recursive - should the list recursively traverse the subdirectories ?

Returns:

a List of PathAttributes that iterates over all entries in the directory dir.

Throws:

NoSuchPathException - If a directory does not exists.

InvalidPathException - If dir is not a directory.

NotConnectedException - If file system is closed.

XenonException - if an I/O error occurred.

listIteratorNoException

private java.util.Iterator<PathAttributes> listIteratorNoException(Path dir)

listIterator

private java.util.Iterator<PathAttributes> listIterator(Path dir)
                                                 throws XenonException

Throws:

XenonException

readFromFile

public java.io.InputStream readFromFile(Path path)
                                 throws XenonException

Description copied from class: FileSystem

Open an existing file and return an InputStream to read from this file.

Specified by:

readFromFile in class FileSystem

Parameters:

path - the to read.

Returns:

the InputStream to read from the file.

Throws:

NoSuchPathException - If the file does not exists.

InvalidPathException - If the file is not regular file.

NotConnectedException - If file system is closed.

XenonException - if an I/O error occurred.

writeToFile

public java.io.OutputStream writeToFile(Path path,
                                        long size)
                                 throws XenonException

Description copied from class: FileSystem

Open a file and return an OutputStream to write to this file.

The size of the file (once all data has been written) must be specified using the size parameter. This is required by some implementations (typically blob-stores).

Specified by:

writeToFile in class FileSystem

Parameters:

path - the target file for the OutputStream.

size - the size of the file once fully written.

Returns:

the OutputStream to write to the file.

Throws:

PathAlreadyExistsException - If the target existed.

NoSuchPathException - if a parent directory does not exist.

NotConnectedException - If file system is closed.

XenonException - if an I/O error occurred.

writeToFile

public java.io.OutputStream writeToFile(Path path)
                                 throws XenonException

Description copied from class: FileSystem

Open a file and return an OutputStream to write to this file. (optional operation)

If the file already exists it will be replaced and its data will be lost. The amount of data that will be written to the file is not specified in advance. This operation may not be supported by all implementations.

Specified by:

writeToFile in class FileSystem

Parameters:

path - the target file for the OutputStream.

Returns:

the OutputStream to write to the file.

Throws:

PathAlreadyExistsException - If the target existed.

NoSuchPathException - if a parent directory does not exist.

NotConnectedException - If file system is closed.

XenonException - if an I/O error occurred.

appendToFile

public java.io.OutputStream appendToFile(Path path)
                                  throws XenonException

Description copied from class: FileSystem

Open an existing file and return an OutputStream to append data to this file. (optional operation)

If the file does not exist, an exception will be thrown. This operation may not be supported by all implementations.

Specified by:

appendToFile in class FileSystem

Parameters:

path - the target file for the OutputStream.

Returns:

the OutputStream to write to the file.

Throws:

PathAlreadyExistsException - If the target existed.

NoSuchPathException - if a parent directory does not exist.

InvalidPathException - if not a regular file

NotConnectedException - If file system is closed.

XenonException - if an I/O error occurred.

UnsupportedOperationException - if the adaptor does not support appending

getAttributes

public PathAttributes getAttributes(Path path)
                             throws XenonException

Description copied from class: FileSystem

Get the PathAttributes of an existing path.

Specified by:

getAttributes in class FileSystem

Parameters:

path - the existing path.

Returns:

the FileAttributes of the path.

Throws:

NoSuchPathException - If the file does not exists.

NotConnectedException - If file system is closed.

XenonException - if an I/O error occurred.

deleteFile

protected void deleteFile(Path file)
                   throws XenonException

Description copied from class: FileSystem

Delete a file. Is only called on existing files This operation must be implemented by the various implementations of FileSystem.

Specified by:

deleteFile in class FileSystem

Parameters:

file - the file to remove

Throws:

InvalidPathException - if the provided path is not a file.

NoSuchPathException - if the provided file does not exist.

XenonException - If the file could not be removed.

deleteDirectory

protected void deleteDirectory(Path path)
                        throws XenonException

Description copied from class: FileSystem

Delete an empty directory. Is only called on empty directories This operation can only delete empty directories (analogous to rmdir in Linux). This operation must be implemented by the various implementations of FileSystem.

Specified by:

deleteDirectory in class FileSystem

Parameters:

path - the directory to remove

Throws:

InvalidPathException - if the provided path is not a directory.

NoSuchPathException - if the provided path does not exist.

XenonException - If the directory could not be removed.

listDirectory

protected java.lang.Iterable<PathAttributes> listDirectory(Path dir)
                                                    throws XenonException

Description copied from class: FileSystem

Return the list of entries in a directory. This operation is non-recursive; any subdirectories in dir will be returned as part of the list, but they will not be listed themselves. This operation must be implemented by the various implementations of FileSystem.

Specified by:

listDirectory in class FileSystem

Parameters:

dir - the directory to list

Returns:

a Iterable that iterates over all entries in dir

Throws:

XenonException - If the list could not be retrieved.

toHDFSPath

org.apache.hadoop.fs.Path toHDFSPath(Path p)

fromHDFSPath

static Path fromHDFSPath(org.apache.hadoop.fs.Path p)

toPathAttributes

private PathAttributes toPathAttributes(org.apache.hadoop.fs.FileStatus s)

toPermissionSet

private java.util.Set<PosixFilePermission> toPermissionSet(org.apache.hadoop.fs.permission.FsPermission permission)

readSymbolicLink

public Path readSymbolicLink(Path link)
                      throws XenonException

Description copied from class: FileSystem

Reads the target of a symbolic link (optional operation).

Specified by:

readSymbolicLink in class FileSystem

Parameters:

link - the link to read.

Returns:

a Path representing the target of the link.

Throws:

NoSuchPathException - If the link does not exists.

InvalidPathException - If the source is not a link.

UnsupportedOperationException - If this FileSystem does not support symbolic links.

NotConnectedException - If file system is closed.

XenonException - if an I/O error occurred.

setPosixFilePermissions

public void setPosixFilePermissions(Path path,
                                    java.util.Set<PosixFilePermission> permissions)
                             throws XenonException

Description copied from class: FileSystem

Sets the POSIX permissions of a path (optional operation).

Specified by:

setPosixFilePermissions in class FileSystem

Parameters:

path - the target path.

permissions - the permissions to set.

Throws:

NoSuchPathException - If the target path does not exists.

UnsupportedOperationException - If this FileSystem does not support symbolic links.

NotConnectedException - If file system is closed.

XenonException - if an I/O error occurred.