org.apache.hadoop.io

Class IOUtils

java.lang.Object

org.apache.hadoop.io.IOUtils

@InterfaceAudience.Public
 @InterfaceStability.Evolving
public class IOUtils
extends Object

An utility class for I/O related functionality.

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor and Description
IOUtils()

Method Summary

Modifier and Type	Method and Description
static void	cleanup(org.apache.commons.logging.Log log, Closeable... closeables) Deprecated. use cleanupWithLogger(Logger, java.io.Closeable...) instead
static void	cleanupWithLogger(org.slf4j.Logger logger, Closeable... closeables) Close the Closeable objects and ignore any Throwable or null pointers.
static void	closeSocket(Socket sock) Closes the socket ignoring IOException
static void	closeStream(Closeable stream) Closes the stream ignoring Throwable.
static void	closeStreams(Closeable... streams) Closes the streams ignoring Throwable.
static void	copyBytes(InputStream in, OutputStream out, Configuration conf) Copies from one stream to another.
static void	copyBytes(InputStream in, OutputStream out, Configuration conf, boolean close) Copies from one stream to another.
static void	copyBytes(InputStream in, OutputStream out, int buffSize) Copies from one stream to another.
static void	copyBytes(InputStream in, OutputStream out, int buffSize, boolean close) Copies from one stream to another.
static void	copyBytes(InputStream in, OutputStream out, long count, boolean close) Copies count bytes from one stream to another.
static void	fsync(File fileToSync) Ensure that any writes to the given file is written to the storage device that contains it.
static void	fsync(FileChannel channel, boolean isDir) Ensure that any writes to the given file is written to the storage device that contains it.
static List<String>	listDirectory(File dir, FilenameFilter filter) Return the complete list of files in a directory as strings.
static void	readFully(InputStream in, byte[] buf, int off, int len) Reads len bytes in a loop.
static byte[]	readFullyToByteArray(DataInput in) Reads a DataInput until EOF and returns a byte array.
static void	skipFully(InputStream in, long len) Similar to readFully().
static IOException	wrapException(String path, String methodName, IOException exception) Takes an IOException, file/directory path, and method name and returns an IOException with the input exception as the cause and also include the file,method details.
static int	wrappedReadForCompressedData(InputStream is, byte[] buf, int off, int len) Utility wrapper for reading from InputStream.
static void	writeFully(FileChannel fc, ByteBuffer buf, long offset) Write a ByteBuffer to a FileChannel at a given offset, handling short writes.
static void	writeFully(WritableByteChannel bc, ByteBuffer buf) Write a ByteBuffer to a WritableByteChannel, handling short writes.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

IOUtils

public IOUtils()

Method Detail

copyBytes

public static void copyBytes(InputStream in,
                             OutputStream out,
                             int buffSize,
                             boolean close)
                      throws IOException

Copies from one stream to another.

Parameters:

in - InputStrem to read from

out - OutputStream to write to

buffSize - the size of the buffer

close - whether or not close the InputStream and OutputStream at the end. The streams are closed in the finally clause.

Throws:

IOException - raised on errors performing I/O.

copyBytes

public static void copyBytes(InputStream in,
                             OutputStream out,
                             int buffSize)
                      throws IOException

Copies from one stream to another.

Parameters:

in - InputStrem to read from

out - OutputStream to write to

buffSize - the size of the buffer.

Throws:

IOException - raised on errors performing I/O.

copyBytes

public static void copyBytes(InputStream in,
                             OutputStream out,
                             Configuration conf)
                      throws IOException

Copies from one stream to another. closes the input and output streams at the end.

Parameters:

in - InputStrem to read from

out - OutputStream to write to

conf - the Configuration object.

Throws:

IOException - raised on errors performing I/O.

copyBytes

public static void copyBytes(InputStream in,
                             OutputStream out,
                             Configuration conf,
                             boolean close)
                      throws IOException

Copies from one stream to another.

Parameters:

in - InputStream to read from

out - OutputStream to write to

conf - the Configuration object

close - whether or not close the InputStream and OutputStream at the end. The streams are closed in the finally clause.

Throws:

IOException - raised on errors performing I/O.

copyBytes

public static void copyBytes(InputStream in,
                             OutputStream out,
                             long count,
                             boolean close)
                      throws IOException

Copies count bytes from one stream to another.

Parameters:

in - InputStream to read from

out - OutputStream to write to

count - number of bytes to copy

close - whether to close the streams

Throws:

IOException - if bytes can not be read or written

wrappedReadForCompressedData

public static int wrappedReadForCompressedData(InputStream is,
                                               byte[] buf,
                                               int off,
                                               int len)
                                        throws IOException

Utility wrapper for reading from InputStream. It catches any errors thrown by the underlying stream (either IO or decompression-related), and re-throws as an IOException.

Parameters:

is - - InputStream to be read from

buf - - buffer the data is read into

off - - offset within buf

len - - amount of data to be read

Returns:

number of bytes read

Throws:

IOException - raised on errors performing I/O.

readFully

public static void readFully(InputStream in,
                             byte[] buf,
                             int off,
                             int len)
                      throws IOException

Reads len bytes in a loop.

Parameters:

in - InputStream to read from

buf - The buffer to fill

off - offset from the buffer

len - the length of bytes to read

Throws:

IOException - if it could not read requested number of bytes for any reason (including EOF)

skipFully

public static void skipFully(InputStream in,
                             long len)
                      throws IOException

Similar to readFully(). Skips bytes in a loop.

Parameters:

in - The InputStream to skip bytes from

len - number of bytes to skip.

Throws:

IOException - if it could not skip requested number of bytes for any reason (including EOF)

cleanup

@Deprecated
public static void cleanup(org.apache.commons.logging.Log log,
                                       Closeable... closeables)

Deprecated. use cleanupWithLogger(Logger, java.io.Closeable...) instead

Close the Closeable objects and ignore any Throwable or null pointers. Must only be used for cleanup in exception handlers.

Parameters:

log - the log to record problems to at debug level. Can be null.

closeables - the objects to close

cleanupWithLogger

public static void cleanupWithLogger(org.slf4j.Logger logger,
                                     Closeable... closeables)

Close the Closeable objects and ignore any Throwable or null pointers. Must only be used for cleanup in exception handlers.

Parameters:

logger - the log to record problems to at debug level. Can be null.

closeables - the objects to close

closeStream

public static void closeStream(Closeable stream)

Closes the stream ignoring Throwable. Must only be called in cleaning up from exception handlers.

Parameters:

stream - the Stream to close

closeStreams

public static void closeStreams(Closeable... streams)

Closes the streams ignoring Throwable. Must only be called in cleaning up from exception handlers.

Parameters:

streams - the Streams to close

closeSocket

public static void closeSocket(Socket sock)

Closes the socket ignoring IOException

Parameters:

sock - the Socket to close

writeFully

public static void writeFully(WritableByteChannel bc,
                              ByteBuffer buf)
                       throws IOException

Write a ByteBuffer to a WritableByteChannel, handling short writes.

Parameters:

bc - The WritableByteChannel to write to

buf - The input buffer

Throws:

IOException - On I/O error

writeFully

public static void writeFully(FileChannel fc,
                              ByteBuffer buf,
                              long offset)
                       throws IOException

Write a ByteBuffer to a FileChannel at a given offset, handling short writes.

Parameters:

fc - The FileChannel to write to

buf - The input buffer

offset - The offset in the file to start writing at

Throws:

IOException - On I/O error

listDirectory

public static List<String> listDirectory(File dir,
                                         FilenameFilter filter)
                                  throws IOException

Return the complete list of files in a directory as strings.

This is better than File#listDir because it does not ignore IOExceptions.

Parameters:

dir - The directory to list.

filter - If non-null, the filter to use when listing this directory.

Returns:

The list of files in the directory.

Throws:

IOException - On I/O error

fsync

public static void fsync(File fileToSync)
                  throws IOException

Ensure that any writes to the given file is written to the storage device that contains it. This method opens channel on given File and closes it once the sync is done.
Borrowed from Uwe Schindler in LUCENE-5588

Parameters:

fileToSync - the file to fsync

Throws:

IOException - raised on errors performing I/O.

fsync

public static void fsync(FileChannel channel,
                         boolean isDir)
                  throws IOException

Ensure that any writes to the given file is written to the storage device that contains it. This method opens channel on given File and closes it once the sync is done. Borrowed from Uwe Schindler in LUCENE-5588

Parameters:

channel - Channel to sync

isDir - if true, the given file is a directory (Channel should be opened for read and ignore IOExceptions, because not all file systems and operating systems allow to fsync on a directory)

Throws:

IOException - raised on errors performing I/O.

wrapException

public static IOException wrapException(String path,
                                        String methodName,
                                        IOException exception)

Takes an IOException, file/directory path, and method name and returns an IOException with the input exception as the cause and also include the file,method details. The new exception provides the stack trace of the place where the exception is thrown and some extra diagnostics information. Return instance of same exception if exception class has a public string constructor; Otherwise return an PathIOException. InterruptedIOException and PathIOException are returned unwrapped.

Parameters:

path - file/directory path

methodName - method name

exception - the caught exception.

Returns:

an exception to throw

readFullyToByteArray

public static byte[] readFullyToByteArray(DataInput in)
                                   throws IOException

Reads a DataInput until EOF and returns a byte array. Make sure not to pass in an infinite DataInput or this will never return.

Parameters:

in - A DataInput

Returns:

a byte array containing the data from the DataInput

Throws:

IOException - on I/O error, other than EOF