org.apache.hadoop.hbase.util
Class FSUtils

java.lang.Object
  org.apache.hadoop.hbase.util.FSUtils

public class FSUtils
extends java.lang.Object

Utility methods for interacting with the underlying file system.

Nested Class Summary

static class

FSUtils.DirFilter A PathFilter that returns directories.

Method Summary

org.apache.hadoop.fs.Path	checkdir(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dir) Check if directory exists.
static void	checkFileSystemAvailable(org.apache.hadoop.fs.FileSystem fs) Checks to see if the specified file system is available
static void	checkVersion(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path rootdir, boolean message) Verifies current version of file system
static org.apache.hadoop.fs.Path	create(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path p) Create file.
static boolean	deleteDirectory(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dir) Delete if exists.
static java.lang.String	getPath(org.apache.hadoop.fs.Path p) Return the 'path' component of a Path.
static org.apache.hadoop.fs.Path	getRootDir(org.apache.hadoop.conf.Configuration c)
static java.util.Map<java.lang.String,java.lang.Integer>	getTableFragmentation(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path hbaseRootDir) Runs through the HBase rootdir and checks how many stores for each table have more than one file in them.
static java.util.Map<java.lang.String,java.lang.Integer>	getTableFragmentation(HMaster master) Runs through the HBase rootdir and checks how many stores for each table have more than one file in them.
static int	getTotalTableFragmentation(HMaster master) Returns the total overall fragmentation percentage.
static java.lang.String	getVersion(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path rootdir) Verifies current version of file system
static boolean	isAppendSupported(org.apache.hadoop.conf.Configuration conf) Heuristic to determine whether is safe or not to open a file for append Looks both for dfs.support.append and use reflection to search for SequenceFile.Writer.syncFs() or FSDataOutputStream.hflush()
static boolean	isHDFS(org.apache.hadoop.conf.Configuration conf)
static boolean	isMajorCompacted(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path hbaseRootDir) Runs through the hbase rootdir and checks all stores have only one file in them -- that is, they've been major compacted.
static boolean	isMajorCompactedPre020(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path hbaseRootDir) Runs through the hbase rootdir and checks all stores have only one file in them -- that is, they've been major compacted.
static boolean	isPre020FileLayout(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path hbaseRootDir) Expects to find -ROOT- directory.
static void	recoverFileLease(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path p, org.apache.hadoop.conf.Configuration conf)
static boolean	rootRegionExists(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path rootdir) Checks if root region exists
static void	setVersion(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path rootdir) Sets version of file system
static void	setVersion(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path rootdir, java.lang.String version) Sets version of file system
static org.apache.hadoop.fs.Path	validateRootPath(org.apache.hadoop.fs.Path root) Verifies root directory path is a valid URI with a scheme
static void	waitOnSafeMode(org.apache.hadoop.conf.Configuration conf, long wait) If DFS, check safe mode and if so, wait until we clear it.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

deleteDirectory

public static boolean deleteDirectory(org.apache.hadoop.fs.FileSystem fs,
                                      org.apache.hadoop.fs.Path dir)
                               throws java.io.IOException

Delete if exists.

Parameters:

fs - filesystem object

dir - directory to delete

Returns:

True if deleted dir

Throws:

java.io.IOException - e

checkdir

public org.apache.hadoop.fs.Path checkdir(org.apache.hadoop.fs.FileSystem fs,
                                          org.apache.hadoop.fs.Path dir)
                                   throws java.io.IOException

Check if directory exists. If it does not, create it.

Parameters:

fs - filesystem object

dir - path to check

Returns:

Path

Throws:

java.io.IOException - e

create

public static org.apache.hadoop.fs.Path create(org.apache.hadoop.fs.FileSystem fs,
                                               org.apache.hadoop.fs.Path p)
                                        throws java.io.IOException

Create file.

Parameters:

fs - filesystem object

p - path to create

Returns:

Path

Throws:

java.io.IOException - e

checkFileSystemAvailable

public static void checkFileSystemAvailable(org.apache.hadoop.fs.FileSystem fs)
                                     throws java.io.IOException

Checks to see if the specified file system is available

Parameters:

fs - filesystem

Throws:

java.io.IOException - e

getVersion

public static java.lang.String getVersion(org.apache.hadoop.fs.FileSystem fs,
                                          org.apache.hadoop.fs.Path rootdir)
                                   throws java.io.IOException

Verifies current version of file system

Parameters:

fs - filesystem object

rootdir - root hbase directory

Returns:

null if no version file exists, version string otherwise.

Throws:

java.io.IOException - e

checkVersion

public static void checkVersion(org.apache.hadoop.fs.FileSystem fs,
                                org.apache.hadoop.fs.Path rootdir,
                                boolean message)
                         throws java.io.IOException

Verifies current version of file system

Parameters:

fs - file system

rootdir - root directory of HBase installation

message - if true, issues a message on System.out

Throws:

java.io.IOException - e

setVersion

public static void setVersion(org.apache.hadoop.fs.FileSystem fs,
                              org.apache.hadoop.fs.Path rootdir)
                       throws java.io.IOException

Sets version of file system

Parameters:

fs - filesystem object

rootdir - hbase root

Throws:

java.io.IOException - e

setVersion

public static void setVersion(org.apache.hadoop.fs.FileSystem fs,
                              org.apache.hadoop.fs.Path rootdir,
                              java.lang.String version)
                       throws java.io.IOException

Sets version of file system

Parameters:

fs - filesystem object

rootdir - hbase root directory

version - version to set

Throws:

java.io.IOException - e

validateRootPath

public static org.apache.hadoop.fs.Path validateRootPath(org.apache.hadoop.fs.Path root)
                                                  throws java.io.IOException

Verifies root directory path is a valid URI with a scheme

Parameters:

root - root directory path

Returns:

Passed root argument.

Throws:

java.io.IOException - if not a valid URI with a scheme

waitOnSafeMode

public static void waitOnSafeMode(org.apache.hadoop.conf.Configuration conf,
                                  long wait)
                           throws java.io.IOException

If DFS, check safe mode and if so, wait until we clear it.

Parameters:

conf - configuration

wait - Sleep between retries

Throws:

java.io.IOException - e

getPath

public static java.lang.String getPath(org.apache.hadoop.fs.Path p)

Return the 'path' component of a Path. In Hadoop, Path is an URI. This method returns the 'path' component of a Path's URI: e.g. If a Path is hdfs://example.org:9000/hbase_trunk/TestTable/compaction.dir, this method returns /hbase_trunk/TestTable/compaction.dir. This method is useful if you want to print out a Path without qualifying Filesystem instance.

Parameters:

p - Filesystem Path whose 'path' component we are to return.

Returns:

Path portion of the Filesystem

getRootDir

public static org.apache.hadoop.fs.Path getRootDir(org.apache.hadoop.conf.Configuration c)
                                            throws java.io.IOException

Parameters:

c - configuration

Returns:

Path to hbase root directory: i.e. hbase.rootdir from configuration as a Path.

Throws:

java.io.IOException - e

rootRegionExists

public static boolean rootRegionExists(org.apache.hadoop.fs.FileSystem fs,
                                       org.apache.hadoop.fs.Path rootdir)
                                throws java.io.IOException

Checks if root region exists

Parameters:

fs - file system

rootdir - root directory of HBase installation

Returns:

true if exists

Throws:

java.io.IOException - e

isMajorCompacted

public static boolean isMajorCompacted(org.apache.hadoop.fs.FileSystem fs,
                                       org.apache.hadoop.fs.Path hbaseRootDir)
                                throws java.io.IOException

Runs through the hbase rootdir and checks all stores have only one file in them -- that is, they've been major compacted. Looks at root and meta tables too.

Parameters:

fs - filesystem

hbaseRootDir - hbase root directory

Returns:

True if this hbase install is major compacted.

Throws:

java.io.IOException - e

getTotalTableFragmentation

public static int getTotalTableFragmentation(HMaster master)
                                      throws java.io.IOException

Returns the total overall fragmentation percentage. Includes .META. and -ROOT- as well.

Parameters:

master - The master defining the HBase root and file system.

Returns:

A map for each table and its percentage.

Throws:

java.io.IOException - When scanning the directory fails.

getTableFragmentation

public static java.util.Map<java.lang.String,java.lang.Integer> getTableFragmentation(HMaster master)
                                                                               throws java.io.IOException

Runs through the HBase rootdir and checks how many stores for each table have more than one file in them. Checks -ROOT- and .META. too. The total percentage across all tables is stored under the special key "-TOTAL-".

Parameters:

master - The master defining the HBase root and file system.

Returns:

A map for each table and its percentage.

Throws:

java.io.IOException - When scanning the directory fails.

getTableFragmentation

public static java.util.Map<java.lang.String,java.lang.Integer> getTableFragmentation(org.apache.hadoop.fs.FileSystem fs,
                                                                                      org.apache.hadoop.fs.Path hbaseRootDir)
                                                                               throws java.io.IOException

Runs through the HBase rootdir and checks how many stores for each table have more than one file in them. Checks -ROOT- and .META. too. The total percentage across all tables is stored under the special key "-TOTAL-".

Parameters:

fs - The file system to use.

hbaseRootDir - The root directory to scan.

Returns:

A map for each table and its percentage.

Throws:

java.io.IOException - When scanning the directory fails.

isPre020FileLayout

public static boolean isPre020FileLayout(org.apache.hadoop.fs.FileSystem fs,
                                         org.apache.hadoop.fs.Path hbaseRootDir)
                                  throws java.io.IOException

Expects to find -ROOT- directory.

Parameters:

fs - filesystem

hbaseRootDir - hbase root directory

Returns:

True if this a pre020 layout.

Throws:

java.io.IOException - e

isMajorCompactedPre020

public static boolean isMajorCompactedPre020(org.apache.hadoop.fs.FileSystem fs,
                                             org.apache.hadoop.fs.Path hbaseRootDir)
                                      throws java.io.IOException

Runs through the hbase rootdir and checks all stores have only one file in them -- that is, they've been major compacted. Looks at root and meta tables too. This version differs from isMajorCompacted(FileSystem, Path) in that it expects a pre-0.20.0 hbase layout on the filesystem. Used migrating.

Parameters:

fs - filesystem

hbaseRootDir - hbase root directory

Returns:

True if this hbase install is major compacted.

Throws:

java.io.IOException - e

isAppendSupported

public static boolean isAppendSupported(org.apache.hadoop.conf.Configuration conf)

Heuristic to determine whether is safe or not to open a file for append Looks both for dfs.support.append and use reflection to search for SequenceFile.Writer.syncFs() or FSDataOutputStream.hflush()

Parameters:

conf -

Returns:

True if append support

isHDFS

public static boolean isHDFS(org.apache.hadoop.conf.Configuration conf)
                      throws java.io.IOException

Parameters:

conf -

Returns:

True if this filesystem whose scheme is 'hdfs'.

Throws:

java.io.IOException

recoverFileLease

public static void recoverFileLease(org.apache.hadoop.fs.FileSystem fs,
                                    org.apache.hadoop.fs.Path p,
                                    org.apache.hadoop.conf.Configuration conf)
                             throws java.io.IOException

Throws:

java.io.IOException