Performance Optimization
This document introduces features to improve Alluxio performance for specific scenarios.
Client Async Prefetch
If the current file is being read sequentially, the Alluxio client will prefetch a range of data after the current read position and start to cache this data on the client. When the current read reaches the cached data, the Alluxio client will return the cached data instead of sending an RPC to the worker.
The prefetch window is self-adjusting. If the reads always starts at the end of last read position, the prefetch window will increase. If the reads are not continuous, the prefetch window will decrease. In the case that the reads are completely random reads, the prefetch window will eventually be reduced to 0.
Async prefetch caches data in client direct memory. Performance can be improved by increasing the direct memory assigned to the jvm process.
The client async prefetch is always enabled. The following parameters allow the user to tune the feature.
| Configuration item | Recommended value | Description |
|---|---|---|
| alluxio.user.position.reader.streaming.async.prefetch.thread | 64 | The overall async prefetch concurrency |
| alluxio.user.position.reader.streaming.async.prefetch.part.length | 4MB | The size of the prefetch unit |
| alluxio.user.position.reader.streaming.async.prefetch.max.part.number | 8 | The maximum number of units a single opened file can have. For example, if the prefetch unit size is 4MB and the max number of unit is 8, Alluxio will fetch at most 32MB data ahead for an opened file. |
| alluxio.user.position.reader.streaming.async.prefetch.file.length.threshold | 4MB | If the file size is less than the specified threshold, Alluxio will max out the prefetch window immediately instead of starting with a small window. This configuration is used to improve small file read performance. |
Enable the slow async prefetch pool
Users may have different situations that require different async prefetch parameters, such as for cold reads vs cache filter reads. Cold reads usually require more concurrency to maximize the network bandwidth and achieve the best performance. Alluxio has a secondary async prefetch pool dedicated for alternative configurations, labeled as the slow thread pool. To enable and configure this secondary pool, set the following configuration:
| Configuration item | Recommended value | Description |
|---|---|---|
| alluxio.user.position.reader.streaming.async.prefetch.use.slow.thread.pool | true | Set to true to enable the slow pool |
| alluxio.user.position.reader.streaming.async.prefetch.use.slow.thread.pool.for.cold.read | true | If set to true, the slow pool will be used for cold read as well. Otherwise, the slow pool will only be used cache filter read. |
| alluxio.user.position.reader.streaming.slow.async.prefetch.thread | 256 | The overall async prefetch concurrency for the slow pool |
| alluxio.user.position.reader.streaming.slow.async.prefetch.part.length | 1MB | The size of the prefetch unit used by the slow pool |
| alluxio.user.position.reader.streaming.slow.async.prefetch.max.part.number | 64 | The maximum number of units a single opened file can have for the slow pool |
Client Large File Preload
Large file preload is an optimization for the cold read of large files. If the feature is enabled, Alluxio will load the whole file concurrently into Alluxio workers as the file is read initially by the client. When running the FIO benchmark for a single 100GB file stored on S3, Alluxio’s cold read performance with this feature achieves a comparable read performance as a fully cached hot read.
Deduplication is handled on both the client and worker side to avoid excessive RPC calls and redundant traffic to the UFS. Note that since Alluxio always fully loads the file, this feature can cause read amplification if the application does not need to read the whole file.
Enabling the Feature
| Configuration item | Recommended value | Description |
|---|---|---|
| alluxio.user.position.reader.preload.data.enabled | true | Set to true to enable large file preloading |
| alluxio.user.position.reader.preload.data.file.size.threshold.min | 1GB | The minimum file size to trigger the async preload |
| alluxio.user.position.reader.preload.data.file.size.threshold.max | 200GB | The maximum file size to trigger the async preload. This is useful to avoid loading extremely large files that would completely fill up the page store capacity and trigger cache eviction. |
| alluxio.worker.preload.data.thread.pool.size | 64 | The number of concurrent jobs on the worker to load the data of the file into UFS in parallel. Each job loads a page into Alluxio. For example, if the page size is 4MB and this config is set to 64, the worker will concurrently load 256M per iteration. |