User Command Line Interface

Slack Docker Pulls

Alluxio’s command line interface provides user access to various operations, such as:

  • Start or stop processes
  • Filesystem operations
  • Administrative commands

Invoke the executable to view the possible subcommands:

$ ./bin/alluxio
Usage:
  bin/alluxio [command]

Available Commands:
  conf        Get, set, and validate configuration settings, primarily those defined in conf/alluxio-site.properties
  exec        Run the main method of an Alluxio class, or end-to-end tests on an Alluxio cluster.
  fs          Operations to interface with the Alluxio filesystem
  help        Help about any command
  info        Retrieve and/or display info about the running Alluxio cluster
  job         Command line tool for interacting with the job service.
  journal     Journal related operations
  license     Check and manage license status
  mount       Operations to manage mount points
  process     Start/stop cluster processes or remove workers
  quota       Operations to manage quota definitions
  security    Security server related commands

Flags:
      --debug-log               True to enable debug logging
      --skip-license-warnings   Skip the automatic warnings before expiration

Use "bin/alluxio [command] --help" for more information about a command.

To set JVM system properties as part of the command, set the -D flag in the form of -Dproperty=value.

To attach debugging java options specified by $ALLUXIO_USER_ATTACH_OPTS, set the --attach-debug flag

Note that, as a part of Alluxio deployment, the Alluxio shell will also take the configuration in ${ALLUXIO_HOME}/conf/alluxio-site.properties when it is run from Alluxio installation at ${ALLUXIO_HOME}.

conf

Get, set, and validate configuration settings, primarily those defined in conf/alluxio-site.properties

conf get

Usage: bin/alluxio conf get [key] [flags]

The get command prints the configured value for the given key. If the key is invalid, it returns a nonzero exit code. If the key is valid but isn’t set, an empty string is printed. If no key is specified, the full configuration is printed.

Note: This command does not require the Alluxio cluster to be running.

Flags:

  • --master: Show configuration properties used by the master (Default: false)
  • --source: Show source of the configuration property instead of the value (Default: false)
  • --unit: Unit of the value to return, converted to correspond to the given unit. E.g., with “–unit KB”, a configuration value of “4096B” will return 4 Possible options include B, KB, MB, GB, TP, PB, MS, S, M, H, D (Default: “”)

Examples:

# Display all the current node configuration
$ ./bin/alluxio conf get

# Display the value of a property key
$ ./bin/alluxio conf get alluxio.master.hostname

# Display the configuration of the current running Alluxio leading master
$ ./bin/alluxio conf get --master

# Display the source of the configuration
$ ./bin/alluxio conf get --source

# Display the values in a given unit
$ ./bin/alluxio conf get alluxio.user.block.size.bytes.default --unit KB
$ ./bin/alluxio conf get alluxio.master.journal.flush.timeout --unit S

exec

Run the main method of an Alluxio class, or end-to-end tests on an Alluxio cluster.

exec edgeTest

Usage: bin/alluxio exec edgeTest [flags]

Test whether the edge runs successfully.

Flags:

  • --no-cluster: only interact with edge and ufs, not to access Dora cluster. (Default: false)
  • --path: (Required) The path can be:
    1. An Alluxio path like ‘alluxio:///data’ only when not set ‘–no-cluster’ flag
    2. A path without scheme, like ‘/’ or ‘/s3’. This is a syntactic sugar for Alluxio paths.
    3. A UFS path with scheme, like ‘s3://bucket/data’ The path must exists in one of the mount points in Alluxio namespace.

exec ufsIOTest

Usage: bin/alluxio exec ufsIOTest [flags]

A benchmarking tool for the I/O between Alluxio and UFS. This test will measure the I/O throughput between Alluxio workers and the specified UFS path. Each worker will create concurrent clients to first generate test files of the specified size then read those files. The write/read I/O throughput will be measured in the process.

Flags:

  • --io-size: specifies the amount of data each thread writes/reads. (Default: “”)
  • --java-opt: The java options to add to the command line to for the task. This can be repeated. The options must be quoted and prefixed with a space. For example: –java-opt “ -Xmx4g” –java-opt “ -Xms2g”. (Default: [])
  • --path: (Required) specifies the path to write/read temporary data in.
  • --threads: specifies the number of threads to concurrently use on each worker. (Default: 4)

Examples:

# This runs the I/O benchmark to HDFS in your process locally
$ ./bin/alluxio runUfsIOTest --path hdfs://<hdfs-address>

# This invokes the I/O benchmark to HDFS in the Alluxio worker
# 2 threads will be created on each worker
# Each thread is writing then reading 512m of data
$ ./bin/alluxio runUfsIOTest --path hdfs://<hdfs-address> --io-size 512m --threads 2

exec ufsTest

Usage: bin/alluxio exec ufsTest [flags]

Test the integration between Alluxio and the given UFS to validate UFS semantics

Flags:

  • --path: (Required) the full UFS path to run tests against.
  • --test: Test name, this option can be passed multiple times to indicate multipleZ tests (Default: [])

fs

Operations to interface with the Alluxio filesystem For commands that take Alluxio URIs as an argument such as ls or mkdir, the argument should be either

  • A complete Alluxio URI, such as alluxio://<masterHostname>:<masterPort>/<path>
  • A path without its scheme header, such as /path, in order to use the default hostname and port set in alluxio-site.properties

Note: All fs commands require the Alluxio cluster to be running.

Most of the commands which require path components allow wildcard arguments for ease of use. For example, the commandbin/alluxio fs rm '/data/2014*'deletes anything in the data directory with a prefix of 2014.

Some shells will attempt to glob the input paths, causing strange errors. As a workaround, you can disable globbing (depending on the shell type; for example, set -f) or by escaping wildcards For example, the command bin/alluxio fs cat /\\* uses the escape backslash character twice. This is because the shell script will eventually call a java program which should have the final escaped parameters cat /\\*.

fs cat

Usage: bin/alluxio fs cat [path]

The cat command prints the contents of a file in Alluxio to the shell.

Examples:

# Print the contents of /output/part-00000
$ ./bin/alluxio fs cat /output/part-00000

fs check-cached

Usage: bin/alluxio fs check-cached --path|--index-file <path> [--limit <limit-size>] [flags]

Checks if files under a path have been cached in alluxio.

Flags:

  • --index-file: The index file on local that contains a list of file paths to check. Each line should contain a ufs file path. (Default: “”)
  • --limit: Limit number of files to check (Default: 1000)
  • --path: The path to check caching status (Default: “”)
  • --recursive: Whether to check files recursively in the given path (Default: false)

fs checksum

Usage: bin/alluxio fs checksum [path]

The checksum command outputs the md5 value of a file in Alluxio. This can be used to verify the contents of a file stored in Alluxio.

Examples:

# Compare the checksum values
# value from Alluxio filesystem
$ ./bin/alluxio fs checksum /LICENSE
md5sum: bf0513403ff54711966f39b058e059a3
# value from local filesystem
md5 LICENSE
MD5 (LICENSE) = bf0513403ff54711966f39b058e059a3

fs chgrp

Usage: bin/alluxio fs chgrp [group] [path] [flags]

The chgrp command changes the group of the file or directory in Alluxio. Alluxio supports file authorization with POSIX file permissions. The file owner or superuser can execute this command.

Flags:

  • --recursive,-R: change the group recursively for all files and directories under the given path (Default: false)

Examples:

# Change the group of a file
$ ./bin/alluxio fs chgrp alluxio-group-new /input/file1

fs chmod

Usage: bin/alluxio fs chmod [mode] [path] [flags]

The chmod command changes the permission of a file or directory in Alluxio. The permission mode is represented as an octal 3 digit value. Refer to https://en.wikipedia.org/wiki/Chmod#Numerical_permissions for a detailed description of the modes.

Flags:

  • --recursive,-R: change the permission recursively for all files and directories under the given path (Default: false)

Examples:

# Set mode 755 for /input/file
$ ./bin/alluxio fs chmod 755 /input/file1

fs chown

Usage: bin/alluxio fs chown <owner>[:<group>] <path> [flags]

The chown command changes the owner of a file or directory in Alluxio. The ownership of a file can only be altered by a superuser

Flags:

  • --recursive,-R: change the owner recursively for all files and directories under the given path (Default: false)

Examples:

# Change the owner of /input/file1 to alluxio-user
$ ./bin/alluxio fs chown alluxio-user /input/file1

fs compact

Usage: bin/alluxio fs compact [path] [flags]

The compact command compact the position write file.

Flags:

  • --commitTimeout: Default is 1d. Uncommitted files (files with an mtime greater than the manifest’s mtime) are retained during compaction. You can configure this value to delete uncommitted files. For example, we can set it to 8h, meaning files with an mtime greater than 8 hours before the current time will be deleted, while files with an mtime less than 8 hours will be retained. (Default: “”)
  • --deleteOrphanFrameFiles: Default is false. Whether delete orphan frame files. (Default: false)
  • --deleteOrphanFrameListFiles: Default is false. Whether delete orphan frame list files. (Default: false)
  • --deleteSnapshotsReferencedFiles: Default is false. Whether delete the files referenced by the snapshots. If deleted, the files cannot be rolled back to a previous version. (Default: false)
  • --exportPath: Default is null. If the path is specified, during compaction, the frames will be merged into a complete file and output to the given path. (Default: “”)
  • --fileLifecycle: Default is 0. Unreferenced files will be deleted during compaction. You can configure this value to retain some recently created files. For example, we can set it to 8h, meaning files with an mtime (modification time) greater than 8 hours before the current time will be deleted, while files with an mtime less than 8 hours will be retained. (Default: “”)
  • --maxRetainedSnapshots: Default is 4. The maximum number of snapshots to be saved. When the number of snapshots exceeds this limit, the oldest snapshot will be deleted. (Default: “”)
  • --parallelism: Default is 1. The concurrency of compaction tasks. (Default: “”)
  • --recursive: Default is false. Whether compact given path recursively. (Default: false)

Examples:

# Compact file /log.txt
$ ./bin/alluxio fs compact [--deleteOrphanFrameListFiles] [--deleteOrphanFrameFiles] [--fileLifecycle 6h] [--commitTimeout 1h] [--maxRetainedSnapshots 4] [--deleteSnapshotsReferencedFiles] [--exportPath /log.compaction.txt] [--recursive] [--parallelism 8] /log.txt

fs cp

Usage: bin/alluxio fs cp [srcPath] [dstPath] [flags]

Copies a file or directory in the Alluxio filesystem or between local and Alluxio filesystems. The file:// scheme indicates a local filesystem path and the alluxio:// scheme or no scheme indicates an Alluxio filesystem path.

Flags:

  • --buffer-size: Read buffer size when coping to or from local, with defaults of 64MB and 8MB respectively (Default: “”)
  • --preserve,-p: Preserve file permission attributes when copying files; all ownership, permissions, and ACLs will be preserved (Default: false)
  • --recursive,-R: True to copy the directory subtree to the destination directory (Default: false)
  • --thread: Number of threads used to copy files in parallel, defaults to 2 * CPU cores (Default: 0)

Examples:

# Copy within the Alluxio filesystem
$ ./bin/alluxio fs cp /file1 /file2

# Copy a local file to the Alluxio filesystem
$ ./bin/alluxio fs cp file:///file1 /file2

# Copy a file in Alluxio to local
$ ./bin/alluxio fs cp alluxio:///file1 file:///file2

# Recursively copy a directory within the Alluxio filesystem
$ ./bin/alluxio fs cp -R /dir1 /dir2

fs head

Usage: bin/alluxio fs head [path] [flags]

The head command prints the first 1KB of data of a file to the shell. Specifying the -c flag sets the number of bytes to print.

Flags:

  • --bytes,-c: Byte size to print (Default: “”)

Examples:

# Print first 2048 bytes of a file
$ ./bin/alluxio fs head -c 2048 /output/part-00000

fs location

Usage: bin/alluxio fs location [path]

Displays the list of hosts storing the specified file.

fs ls

Usage: bin/alluxio fs ls [path] [flags]

The ls command lists all the immediate children in a directory and displays their file info. Using ls on a file will only display the information for that specific file.

Flags:

  • --cache-filter,-c: Show the cacheability rules for metadata and data of the file based on the cache filter configuration; if cache filter is not enabled, show Disabled. Note the resolved cacheability are based on cache filter observed by the command line. (Default: false)
  • --help: help for this command (Default: false)
  • --human-readable,-h: Print sizes in human readable format (Default: false)
  • --list-dir-as-file,-d: List directories as files (Default: false)
  • --recursive,-R: List subdirectories recursively (Default: false)

Examples:

# List immediate children of /s3/data
$ ./bin/alluxio fs ls /s3/data

fs mkdir

Usage: bin/alluxio fs mkdir [path1 path2 ...]

The mkdir command creates a new directory in the Alluxio filesystem. It is recursive and will create any parent directories that do not exist. Note that the created directory will not be created in the under storage system until a file in the directory is persisted to the underlying storage. Using mkdir on an invalid or existing path will fail.

Examples:

# Creating a folder structure
$ ./bin/alluxio fs mkdir /users
$ ./bin/alluxio fs mkdir /users/Alice
$ ./bin/alluxio fs mkdir /users/Bob

fs mv

Usage: bin/alluxio fs mv [srcPath] [dstPath]

The mv command moves a file or directory to another path in Alluxio. The destination path must not exist or be a directory. If it is a directory, the file or directory will be placed as a child of the directory. The command is purely a metadata operation and does not affect the data blocks of the file.

Examples:

# Moving a file
$ ./bin/alluxio fs mv /data/2014 /data/archives/2014

fs rm

Usage: bin/alluxio fs rm [path] [flags]

The rm command removes a file from Alluxio space and the under storage system. The file will be unavailable immediately after this command returns, but the actual data may be deleted a while later.

Flags:

  • --alluxio-only: True to only remove data and metadata from Alluxio cache (Default: false)
  • --recursive,-R: True to recursively remove files within the specified directory subtree (Default: false)
  • --skip-ufs-check,-U: True to skip checking if corresponding UFS contents are in sync (Default: false)

Examples:

# Remove a file from Alluxio and the under storage system
$ ./bin/alluxio fs rm /tmp/unused-file

# Remove a file from Alluxio filesystem only
$ ./bin/alluxio fs rm --alluxio-only --skip-ufs-check /tmp/unused-file2
# Note it is recommended to use both --alluxio-only and --skip-ufs-check together in this situation

fs stat

Usage: bin/alluxio fs stat [path] [flags]

The stat command dumps the FileInfo representation of a file or a directory to the shell.

Flags:

  • --format,-f: Display info in the given format: “%N”: name of the file “%z”: size of file in bytes “%u”: owner “%g”: group name of owner “%i”: file id of the file “%y”: modification time in UTC in ‘yyyy-MM-dd HH:mm:ss’ format “%Y”: modification time as Unix timestamp in milliseconds “%b”: Number of blocks allocated for file (Default: “”)

Examples:

# Display file's stat
$ ./bin/alluxio fs stat /data/2015/logs-1.txt

# Display directory's stat
$ ./bin/alluxio fs stat /data/2015

# Display the size of file
$ ./bin/alluxio fs stat -f %z /data/2015/logs-1.txt

fs tail

Usage: bin/alluxio fs tail [path] [flags]

The tail command prints the last 1KB of data of a file to the shell. Specifying the –bytes flag sets the number of bytes to print.

Flags:

  • --bytes: Byte size to print (Default: “”)

Examples:

# Print last 2048 bytes of a file
$ ./bin/alluxio fs tail --bytes 2048 /output/part-00000

fs test

Usage: bin/alluxio fs test [path] [flags]

Test a property of a path, returning 0 if the property is true, or 1 otherwise

Flags:

  • --dir,-d: Test if path is a directory (Default: false)
  • --exists,-e: Test if path exists (Default: false)
  • --file,-f: Test if path is a file (Default: false)
  • --not-empty,-s: Test if path is not empty (Default: false)
  • --zero,-z: Test if path is zero length (Default: false)

fs touch

Usage: bin/alluxio fs touch [path]

Create a 0 byte file at the specified path, which will also be created in the under file system

info

Retrieve and/or display info about the running Alluxio cluster

info cluster

Usage: bin/alluxio info cluster

Print the bound cluster information

info nodes

Usage: bin/alluxio info nodes

Show all registered workers’ status

info production

Usage: bin/alluxio info production

Print the production ID

info version

Usage: bin/alluxio info version

Print Alluxio version.

job

Command line tool for interacting with the job service.

job free

Usage: bin/alluxio job free [flags]

The free command triggers a scheduler job to free a directory and release cached pages in all workers.

Flags:

  • --batch-size: [submit] The speed of free files per second per worker. If the value is 0 or empty, it will be set to the value of ‘alluxio.job.batch.size’ (Default: 0)
  • --force: [submit] Trigger free even if some workers are offline (Default: false)
  • --format: [progress] Format of output, either TEXT or JSON (Default: “”)
  • --path: (Required) [All] Source path of free operation
  • --progress: View progress of submitted job (Default: false)
  • --recursive: [submit] recursive free all files (Default: true)
  • --stop: Stop running job (Default: false)
  • --submit: Submit job (Default: false)
  • --verbose: [progress] Verbose output (Default: false)

Examples:

# Submit a free job to free path cache
$ ./bin/alluxio job free --path s3://bucket/path --submit

# View the progress of a submitted job
$ ./bin/alluxio job free --path s3://alluxio/path --progress
Progress for Free path file 's3://alluxio/path':
	Job Id: b21ce9fb-f332-4d39-8bb4-554f9a4fa601
	Job Submitted: Fri Feb 02 21:28:56 CST 2024
	Job path: s3://alluxio/path
	Job State: SUCCEEDED, finished at Fri Feb 02 21:29:01 CST 2024
	Free Info :	totalFile:4 totalByte:3072.00KB
	Free Files Failed: 0
	Free Bytes Failed: 0B
	Free Files Succeeded: 4
	Free Bytes Succeeded: 3072.00KB

# Stop a submitted job with best effort. Workers will stop freeing cache under the specified path as soon as possible.
$ ./bin/alluxio job free --path s3://alluxio/path --stop

job list

Usage: bin/alluxio job list [flags]

According to job type and job state list jobs.

Flags:

  • --job-state: job state. RUNNING VERIFYING STOPPED SUCCEEDED FAILED ALL (Default: “”)
  • --job-type: job type. LOAD FREE COPY MOVE ALL (Default: “”)

Examples:

$./bin/alluxio job list --job-type all --job-state all
List job type:ALL state:ALL
{"mJobId":"1c723247-8381-4d69-bd29-b0f4fb34fc8d","mFreePath":"s3://alluxio/test-dir/","mJobType":"free","mJobState":"SUCCEEDED","mFreeBytes":81920000,"mFreeFileNums":20000,"mFreeFailFileNums":0,"mFreeFailBytes":0,"mFailInfos":{},"mTasks":{"WorkerInfo{id=0, identity=worker-1, address=WorkerNetAddress{host=hostname, containerHost=, rpcPort=29999, dataPort=29997, webPort=30000, domainSocketPath=, secureRpcPort=0, httpServerPort=0}, lastContactSec=0, state=LIVE, capacityBytes=0, usedBytes=0, startTimeMs=0, capacityBytesOnTiers=null, usedBytesOnTiers=null, version=, revision=}":"SUCCESS"},"mVerbose":true,"mStartTime":1722490595894,"mEndTime":1722490606649}
{"mPath":"s3://alluxio/test-dir/","mIndexFile":"","mJobType":"load","mVerbose":true,"mJobState":"SUCCEEDED","mVerificationEnabled":false,"mSkippedByteCount":0,"mLoadedByteCount":81920000,"mScannedInodesCount":20000,"mLoadedNonEmptyFileCopiesCount":20000,"mTotalByteCount":81920000,"mThroughput":1462857,"mFailureFilesPercentage":0.0,"mFailureSubTasksPercentage":0.0,"mRetrySubTasksPercentage":0.0,"mFailedFileCount":0,"mRecentFailedSubtasksWithReasons":[],"mRecentRetryingSubtasksWithReasons":[],"mSkipIfExists":false,"mMetadataOnly":false,"mRunningStage":"","mRetryDeadLetterQueueSize":0,"mTimeElapsed":52693,"mSegmentEnabled":false}

job load

Usage: bin/alluxio job load [flags]

The load command moves data from the under storage system into Alluxio storage. For example, load can be used to prefetch data for analytics jobs. If load is run on a directory, files in the directory will be recursively loaded.

Flags:

  • --bandwidth: [submit] Single worker read bandwidth limit (Default: “”)
  • --batch-size: [submit] # of batch size to load at every worker. If the value is 0 or empty, it will be set to the value of alluxio.job.batch.size (Default: 0)
  • --file-filter-regx: [submit] Skip files that match the regx pattern (Default: “”)
  • --force: [submit] Trigger metadata sync even if some workers are offline (Default: false)
  • --format: [progress] Format of output, either TEXT or JSON (Default: “”)
  • --index-file: [all] Source path of the index file for load operation (Default: “”)
  • --metadata-only: [submit] Only load file metadata (Default: false)
  • --partial-listing: [submit] Use partial directory listing, initializing load before reading the entire directory but cannot report on certain progress details (Default: false)
  • --path: [all] Source path of load operation (Default: “”)
  • --progress: View progress of submitted job (Default: false)
  • --replicas: [submit] # of replicas to load (Default: 1)
  • --skip-if-exists: [submit] Skip existing fullly cached files (Default: false)
  • --skip-quota-check: [submit] skip quota check and force submit load hob (Default: false)
  • --stop: Stop running job (Default: false)
  • --submit: Submit job (Default: false)
  • --verbose: [progress] Verbose output (Default: false)
  • --verify: [submit] Run verification when load finishes and load new files if any (Default: false)

Examples:

# Submit a load job
$ ./bin/alluxio job load --path <full-ufs-path> --submit
OR
$ ./bin/alluxio job load --index-file <full-ufs-path> --submit

If --index-file is used, load command will read the index file and load the file paths in it.
Both file and directory are supported. In the index file, each line represents an UFS path.

An example index file looks like the following:

hdfs://cluster/root/file
s3://bucket/directory/
#file:///ignored

The hdfs path will be loaded as a single file,
the s3 path will be loaded recursively as a directory,the s3 path will be loaded recursively as a directory,
because it ends with the separator / ,
while the local file will be ignored as it starts with # and considered as a line of comment.

# View the progress of a submitted job
$ ./bin/alluxio job load --path <full-ufs-path> --progress
OR
$ ./bin/alluxio job load --index-file <full-ufs-path> --progress
# Example output
Progress for loading path '/path':
        Settings:       bandwidth: unlimited    verify: false
        Job State: SUCCEEDED
        Files Processed: 1000
        Bytes Loaded: 125.00MB
        Throughput: 2509.80KB/s
        Block load failure rate: 0.00%
        Files Failed: 0

# Stop a submitted job
$ ./bin/alluxio job load --path <full-ufs-path> --stop
OR
$ ./bin/alluxio job load --index-file <full-ufs-path> --stop

journal

Journal related operations

journal checkpoint

Usage: bin/alluxio journal checkpoint

The checkpoint command creates a checkpoint the leading Alluxio master’s journal. This command is mainly used for debugging and to avoid master journal logs from growing unbounded. Checkpointing requires a pause in master metadata changes, so use this command sparingly to avoid interfering with other users of the system.

journal format

Usage: bin/alluxio journal format

The format command formats the local Alluxio master’s journal.

Warning: Formatting should only be called while the cluster is not running.

journal read

Usage: bin/alluxio journal read [flags]

The read command parses the current journal and outputs a human readable version to the local folder. This command may take a while depending on the size of the journal.

Note: This command requies that the Alluxio cluster is NOT running.

Flags:

  • --end: end log sequence number (exclusive) (Default: -1)
  • --input-dir: input directory on-disk to read the journal content from (Default: “”)
  • --master: name of the master class (Default: “”)
  • --output-dir: output directory to write journal content to (Default: “”)
  • --start: start log sequence number (inclusive) (Default: 0)

Examples:

$ ./bin/alluxio journal read
# output
Dumping journal of type EMBEDDED to /Users/alluxio/journal_dump-1602698211916
2020-10-14 10:56:51,960 INFO  RaftStorageDirectory - Lock on /Users/alluxio/alluxio/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/in_use.lock acquired by nodename 78602@alluxio-user
2020-10-14 10:56:52,254 INFO  RaftJournalDumper - Read 223 entries from log /Users/alluxio/alluxio/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_0-222.

license

Check and manage license status

license check-expiration

Usage: bin/alluxio license check-expiration

Validates the license and print warnings if any constraint is about to be exceeded

license show

Usage: bin/alluxio license show [flags]

Show the details of the license based on the prod jar and configuration

Flags:

  • --output,-o: Output format, could be json/yaml (Default: “”)

license status

Usage: bin/alluxio license status [flags]

List the current status of the cluster that license may use

Flags:

  • --raw: Output raw JSON data instead of human-readable format for bytes, datetime, and duration. (Default: false)

license update

Usage: bin/alluxio license update [flags]

Use the license from the current site properties file to update

Flags:

  • --process: The process you want to update with the new license (Default: “”)

mount

The mount command manges the mapping from under storage path to an Alluxio path, where files and folders created in Alluxio space under the path will be backed by a corresponding file or folder in the under storage path.

mount add

Usage: bin/alluxio mount add [flags]

The add command can be used to make data in another storage system available in Alluxio.

Note that the –readonly flag mounts are useful to prevent accidental write operations. If multiple Alluxio satellite clusters mount a remote storage cluster which serves as the central source of truth, the –readonly option could help prevent any write operations on the satellite cluster from wiping out the remote storage.

To connect to the UFS for a mount point, Alluxio looks for the corresponding connector under ${ALLUXIO_HOME}/lib/ and will use the first one that supports the path. The connector jars look like lib/alluxio-underfs-hdfs-2.7.1.jar. The logic to decide whether a connector supports a path depends on the UnderFileSystemFactory implementation. When there are multiple connectors for the same UFS, like lib/alluxio-underfs-hdfs-2.7.1.jar, lib/alluxio-underfs-hdfs-2.7.1-patch1.jar, lib/alluxio-underfs-hdfs-2.7.1-patch2.jar, the option “alluxio.underfs.strict.version.match.enabled” can be used to make sure the correct one is picked up. For example, if the HDFS is running with 2.7.1-patch1, you can use “alluxio.underfs.version” and “alluxio.underfs.strict.version.match.enabled=true” to ensure “lib/alluxio-underfs-hdfs-2.7.1-patch1.jar” is used to connect to the target HDFS at hdfs://ns1/

Flags:

  • --master: Set true to create mount point tracked by Alluxio master (Default: false)
  • --option: Configuration options, in the form of =, associated with the mount point, such as credentials (Default: [])
  • --path: (Required) Alluxio path to mount onto
  • --shared: Sets the permission bits of the mount point to be accessible for all Alluxio users (Default: false)
  • --ufs-uri: (Required) UFS URI to mount

Examples:

# Add a mount point at /hdfs to access hdfs://host1:9000/data/
# Note that the mount point must be under the root directory
$ ./bin/alluxio mount add --path /hdfs --ufs-uri hdfs://host1:9000/data/

# Add a read only mount that is accessible to all Alluxio users
$ ./bin/alluxio mount add --path /hdfs2 --ufs-uri hdfs://host2:9000/data/ --shared --readonly

# Add a S3 bucket mount with S3 credentials required to access the bucket
$ ./bin/alluxio mount add --path /s3 --ufs-uri s3://data-bucket/ --option s3a.accessKeyId=<accessKey> --option s3a.secretKey=<secretKey> --option alluxio.underfs.s3.region=us-east-1

# Add a HDFS mount using a specific UFS jar in lib/
$ ./bin/alluxio mount add --path /ns1 --ufs-uri hdfs://ns1/ --option alluxio.underfs.version=2.7.1-patch1 --option alluxio.underfs.strict.version.match.enabled=true

mount list

Usage: bin/alluxio mount list [flags]

List all known mount points set on the Alluxio filesystem

Flags:

  • --master: Set true to list mount points under the master based registration (Default: false)

Examples:

# List mount points
$ ./bin/alluxio mount list

mount remove

Usage: bin/alluxio mount remove [flags]

Removes the mount point at the specified path

Flags:

  • --master: Set true to delete mount point under the master based registration (Default: false)
  • --path: (Required) Alluxio path to unmount

Examples:

# Remove mount point at /mnt/hdfs
$ ./bin/alluxio mount remove --path /mnt/hdfs

process

Start/stop cluster processes or remove workers

process remove-worker

Usage: bin/alluxio process remove-worker [flags]

Remove given worker from the cluster, so that clients and other workers will not consider the removed worker for services. The worker must have been stopped before it can be safely removed from the cluster.

Flags:

  • --name,-n: (Required) Worker id

process start

Usage: bin/alluxio process start [flags]

Starts a single process locally or a group of similar processes across the cluster. For starting a group, it is assumed the local host has passwordless SSH access to other nodes in the cluster. The command will parse the hostnames to run on by reading the conf/coordinator and conf/workers files, depending on the process type.

Flags:

  • --async,-a: Asynchronously start processes without monitoring for start completion (Default: false)
  • --console-log,-c: Log output to stdout in addition to log file (Default: false)
  • --direct,-d: (For use in docker) Directly run the start command, skipping all other steps and avoid using nohup to launch (Default: false)
  • --skip-kill-prev,-N: Avoid killing previous running processes when starting (Default: false)

process stop

Usage: bin/alluxio process stop [flags]

Stops a single process locally or a group of similar processes across the cluster. For stopping a group, it is assumed the local host has passwordless SSH access to other nodes in the cluster. The command will parse the hostnames to run on by reading the conf/coordinator and conf/workers files, depending on the process type.

Flags:

  • --soft,-s: Soft kill only, don’t forcibly kill the process (Default: false)

quota

The quota command manges all quota definitions in Alluxio system. Alluxio supports setting quota on directories in Alluxio path.

quota add

Usage: bin/alluxio quota add [flags]

Use the add command to set a new quota definition on a directory in Alluxio path. The quota definition will be applied to the total cache in all Alluxio workers.

Flags:

  • --directory: (Required) Alluxio path to set quota on
  • --quota-size: (Required) Size of the quota definition, eg. 10GB, 1TB

Examples:

# Add a quota of 10GB to the directory /s3/ in Alluxio namespace
# The cache in all Alluxio workers combined will be limited to 10GB
$ bin/alluxio quota add --directory /s3/ --quota-size 10GB

quota list

Usage: bin/alluxio quota list [flags]

Use the list command to list the current usage of all quotas defined in Alluxio. You can also use the –interval option to refresh the quota usage information every specified interval.

Flags:

  • --interval: Interval to refresh the quota usage information. (Default: “”)

Examples:

# Lists current quota usage
$ bin/alluxio quota list
Alluxio path: /local, Quota capacity bytes: 11534336, Used bytes: 10486645, State: Available
Alluxio path: /s3/data, Quota capacity bytes: 10737418240, Used bytes: Calculating, State: Available

# Keeps running until interrupted by user by pressing ctrl+C
$ bin/alluxio quota list --interval 5s
Polling quota usage summary from the master every 5s
Alluxio path: /local, Quota capacity bytes: 11534336, Used bytes: 10486645, State: Available
Alluxio path: /s3/data, Quota capacity bytes: 10737418240, Used bytes: Calculating, State: Available
Alluxio path: /local, Quota capacity bytes: 11534336, Used bytes: 10486645, State: Available
Alluxio path: /s3/data, Quota capacity bytes: 10737418240, Used bytes: Calculating, State: Available
Alluxio path: /local, Quota capacity bytes: 11534336, Used bytes: 10486645, State: Available
Alluxio path: /s3/data, Quota capacity bytes: 10737418240, Used bytes: 10486645, State: Available

quota remove

Usage: bin/alluxio quota remove [flags]

Use the remove command to remove an existing quota definition on an Alluxio path.

Flags:

  • --directory: (Required) Alluxio directory to remove quota on

Examples:

$ bin/alluxio quota remove --directory /s3/
Successfully removed quota definition for path /s3/data.

quota update

Usage: bin/alluxio quota update [flags]

Use the update command to update an existing quota definition.

Flags:

  • --directory: (Required) Alluxio path to set quota on
  • --force: Whether to force the update of the quota, even if it violates the existing space usage (Default: false)
  • --quota-size: (Required) Size of the quota definition, eg. 10GB, 1TB

Examples:

$ bin/alluxio quota update --directory /local/data/ --quota-size 100GB

security

Security server related commands

security token

Usage: bin/alluxio security token

Get one token from the security service