Fix signature issue for some underfs modules #18494
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
### What changes are proposed in this pull request? We will make copy existing directory fail the validation, which will skip copy the directory. But in the meanwhile, we shall pass the validation for move, since we want to delete any existing moved directory. ### Why are the changes needed? bugs from e2e tests. ### Does this PR introduce any user facing changes? na pr-link: Alluxio#18085 change-id: cid-7619ee0b6902dd95ae520a70631ddffc31b7670c
### What changes are proposed in this pull request? rearrange nav menu to move: - fuse sdk docs under reference/client apis/posix api - presto sdk under compute/presto - glossary & troubleshooting under reference ### Why are the changes needed? better doc flow ### Does this PR introduce any user facing changes? n/a pr-link: Alluxio#18084 change-id: cid-be60f2eb69f210c0d2044b1e2ecf9e2fa28f062a
New cli for collectInfo does not seem to work, fix the bug to keep the arguments in line with the Java code. pr-link: Alluxio#18080 change-id: cid-2a677899ad0964d79349e287b170afa0715c6cf7
### What changes are proposed in this pull request? Improve pom module name pr-link: Alluxio#18032 change-id: cid-c1ba348d25318ff0581e2bcbc5748bba1602631e
### What changes are proposed in this pull request? Worker checks UFS rename operation result and throws expcetion if failed ### Why are the changes needed? The rename operation result is omitted before. If operation fails, Alluxio client does not know the result. ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#18089 change-id: cid-d6f93fe33adb42d2f9f5500a170cb2e15ab27aba
### What changes are proposed in this pull request? Remove tiered identity ### Why are the changes needed? This feature is no longer supported ### Does this PR introduce any user facing changes? N/A Entry points have been removed previously. This PR removes the src code pr-link: Alluxio#18033 change-id: cid-92a6d61ad51f12b708721e26ceab063359b5f2e5
### What changes are proposed in this pull request? Fix alluxio-fuse bug with `/tmp` symlink on macOS On macOS, the directory `/tmp/` is a symbolic link to `/private/tmp/`. This caused the `fuse_mounted` function to fail when attempting to mount a UFS to `/tmp/mnt`, as the `mount` command outputs the path as `/private/tmp/mnt`. This commit corrects this issue by resolving the symlink before checking the mount status. ### Why are the changes needed? Fix this bug which fails to detect if alluxio-fuse successfully mounts a remote UFS ### Does this PR introduce any user facing changes? NA pr-link: Alluxio#18093 change-id: cid-60a42e81b3c5f6ffe3f75b733817f4a58ec047cc
### What changes are proposed in this pull request? should use interface factory for membership mgr module for scheduler at time of master startup, otherwise setting STATIC membership mgr will not start master. ### Why are the changes needed? if alluxio.worker.membership.manager.type is set to STATIC, master won't be able to start as it needs etcd related info. ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#18096 change-id: cid-c052114c1aef013049dcf9fccaded4663bf2dd8a
This PR fixed the shaded client version problem in dora example pom.
Current version in dora/example/pom.xml written as `<version>2.9.0</version>`. As a result, all updates on the shaded client after 2.9.0 won't work (e.g. removal of swift ufs). Now it is written as `<version>${project.version}</version>`
pr-link: Alluxio#18100
change-id: cid-0cb94bb5a3360b3cd347c2c09c7c741517ce8647
Run `bash -c java -version` might be not work on remote machines, reporting the following bug. ``` java.lang.RuntimeException: Failed to successfully complete the job: Task execution failed: Error: error defining alluxio environment --- at /cli/src/alluxio.org/cli/launch/launch.go:49 (Run.func1) --- Caused by: error checking java version compatibility --- at /cli/src/alluxio.org/cli/env/env.go:132 (InitAlluxioEnv) --- Caused by: error finding java version from `/usr/lib/jvm/java/bin/java -version` --- at /cli/src/alluxio.org/cli/env/env.go:246 (checkJavaVersion) --- Caused by: exec: "bash": executable file not found in $PATH ``` Run `java -version` directly can solve this problem. pr-link: Alluxio#18105 change-id: cid-4bd1baf636c25fe4ef6614a4f58d3d27d9485ddc
### What changes are proposed in this pull request? The reverse resolution should follow the same pattern as defined in `DoraCacheFileSystem.convertAlluxioPathToUfsPath()` and be made public. This further allows inheritance and overriding. pr-link: Alluxio#18106 change-id: cid-072732ed2bc3bd3889e173c4cda4000e8860d35b
remove the indentation in sitemap and manage indentation in css pr-link: Alluxio#18087 change-id: cid-40d1e854933ab1502e8976c9335d3e8b6a1b42ec
Handle the reserved character ':' for HTTP server. pr-link: Alluxio#18104 change-id: cid-3134592ee4256cf3ee1396117ae02bbaf55d9c36
### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18112 change-id: cid-7f60491f8a152bf9d1ba4319d2558b1e0ac7ac4c
### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18113 change-id: cid-586e7941f2f4b737f0b0ebe7c15cb13d182e6d0c
### What changes are proposed in this pull request?
Return FileNotFound when a file is being create/write. No need to refresh its metadata.
Metadata will be refreshed when file is closed.
### Why are the changes needed?
If a file is being written to, and another getStatus() comes for such file,
this file probably has not been created in UFS. At that time,
the getStatus() thinks that this file does not exists. So the worker tries to remove its metadata and its data from cache.
ERROR logs were generated in such case.
This patch is to handle this correctly and remove such ERROR logs:
```
2023-09-05 17:16:32,802 ERROR LocalCacheManager - Failed to delete page PageId{FileId=f4b0d1f405d7419069129a1262eae9329e9c19ed2584fc3fe27656a698344047, PageIndex=0} (isTemporary: false) from pageStore.
alluxio.exception.PageNotFoundException: /Volumes/ramdisk/LOCAL/1048576/824/f4b0d1f405d7419069129a1262eae9329e9c19ed2584fc3fe27656a698344047/0
at alluxio.client.file.cache.store.LocalPageStore.delete(LocalPageStore.java:135)
at alluxio.client.file.cache.LocalCacheManager.deletePage(LocalCacheManager.java:889)
at alluxio.client.file.cache.LocalCacheManager.delete(LocalCacheManager.java:694)
at alluxio.client.file.cache.LocalCacheManager.delete(LocalCacheManager.java:706)
at alluxio.client.file.cache.LocalCacheManager.lambda$deleteFile$5(LocalCacheManager.java:839)
at java.lang.Iterable.forEach(Iterable.java:75)
at alluxio.client.file.cache.LocalCacheManager.deleteFile(LocalCacheManager.java:839)
at alluxio.client.file.cache.NoExceptionCacheManager.deleteFile(NoExceptionCacheManager.java:209)
at alluxio.worker.dora.DoraMetaManager.invalidateCachedFile(DoraMetaManager.java:337)
at alluxio.worker.dora.DoraMetaManager.removeFromMetaStore(DoraMetaManager.java:178)
at alluxio.worker.dora.DoraMetaManager.loadFromUfs(DoraMetaManager.java:131)
at alluxio.worker.dora.PagedDoraWorker.getGrpcFileInfo(PagedDoraWorker.java:392)
at alluxio.worker.grpc.DoraWorkerClientServiceHandler.getStatus(DoraWorkerClientServiceHandler.java:201)
at alluxio.grpc.BlockWorkerGrpc$MethodHandlers.invoke(BlockWorkerGrpc.java:1589)
at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:355)
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:867)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
```
### Does this PR introduce any user facing changes?
No
pr-link: Alluxio#18108
change-id: cid-47d15fcde921a539a6e3c64e7ac25fc400f26034
### What changes are proposed in this pull request? Add isReadOnly() for FuseFileStream and implement this interface in various Streams. ### Why are the changes needed? This interface is used to determine if the Stream is only. This is needed to determine if a real data flush is needed. ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#18114 change-id: cid-6a60e9e917bb0f8a439d76b51beaaba2b723dde4
Script `launch-process` was renamed to `launch-process-bash`. Update the name of the script in `entrypoint.sh` pr-link: Alluxio#18115 change-id: cid-7053532e9fac93ef3f987e1525ae7260ac3a8dc0
### What changes are proposed in this pull request? Add a config to turn on/off the sdk cache fallback ### Why are the changes needed? the read fallback should be turn off by default to avoid retry storm to UFS ### Does this PR introduce any user facing changes? Addition of property keys pr-link: Alluxio#18099 change-id: cid-2c893a617ac5e90086db98610434db6d763c1fe9
One more place to count the cache hit requests pr-link: Alluxio#17878 change-id: cid-0cb55b7ca627832ce2b53e80ea58cfd6a2730bfa
### What changes are proposed in this pull request? Fix the array out of bound exception in presto ### Why are the changes needed? Because we saw earlier that the LocalCacheManager swallows all exceptions, if there is an error accessing one of the cached files, the exception will be ignored. However, the offset may have changed during the failed read attempt in LocalCacheManager. When LocalCacheManager returns -1 on error and tries to reread from the lower layer storage, the offset could be out of bounds. Regarding why this issue only appears in versions 2.9.3 and later, it's because the offset became a member variable of our target buffer starting from 2.9.3. In earlier versions it was always a local variable, so there was no compounding of offset errors. ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18098 change-id: cid-c7b949d9146847b14ba672df56e21bc4bc5ad705
Count the hit and miss of ListStatus and GetFileInfo in worker pr-link: Alluxio#17848 change-id: cid-97c8efb81088d7e9b636ef927fcbc62a166fe085
### What changes are proposed in this pull request? Change metric type of LocalCacheState from counter to gauge. ### Why are the changes needed? The type of LocalCacheState metric is not reasonable since the value of this metric is a enum type not an increasable value ### Does this PR introduce any user facing changes? No pr-link: Alluxio#18070 change-id: cid-61037d3580e2f25f782e1c876ced0ecad5b312a4
### What changes are proposed in this pull request? Change name of NoopMembershipManager to MasterMembershipManager. ### Why are the changes needed? N/A ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#18097 change-id: cid-9f39e3b1a9b8cdad51e49bbb5d12e86d21b07e23
Reverts Alluxio#18070 revert per request on backward compatibility pr-link: Alluxio#18119 change-id: cid-aeecbff3bc9d6bff41eec25405e87a2f2b79079f
### What changes are proposed in this pull request? Refactor the Netty read handler of worker. ### Why are the changes needed? The previous implementation creates a state machine per read request, instead of per channel. This implies that if two read requests are sent over the same channel, the worker would possibly use one channel to send data of different files or regions. This can lead to data corruption. This PR proposes to use a state machine per channel, and handles channel events throughout the whole lifecycle of the channel. Things like a faulty client sending a second request over the same channel before the first request is completed, is handled gracefully with a client error. The state transitions look like the following:  ### Does this PR introduce any user facing changes? No. ### Tests Tests have been done with basic Alluxio CLI tools, as well as automated PrestoDB and TPC-DS tests. pr-link: Alluxio#17479 change-id: cid-bb8b2c70f0bf0bd84e73d9e858bf6e80706427aa
Refactor Netty read handler to allow subclassing. pr-link: Alluxio#18120 change-id: cid-d0d4125459644d8e92c04dd9decdb622f14cbdb6
### What changes are proposed in this pull request? Implement per-thread cache context ### Why are the changes needed? Enable the fine-grained cache admission ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18029 change-id: cid-839bc71b2df158a4aeaedf22c5c7fb40dfd769e8
Fuse is supported on K8s. Remove the outdated limitation. pr-link: Alluxio#18122 change-id: cid-99cacf64405867a720a5304ab5604b778c0b2127
### What changes are proposed in this pull request? Change `runClass` in Benchmark to `exec class` ### Why are the changes needed? Since new alluxio cli change from `runClass` to `exec class`, related benchmark code should be change as well to keep it work ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18079 change-id: cid-2e8b23a7fc272ed0266c81f5e816442fdb9cd25b
### What changes are proposed in this pull request? Now WORKER_FUSE_ENABLED is invalid. ### Why are the changes needed? improve code quality. ### Does this PR introduce any user facing changes? no. pr-link: Alluxio#18349 change-id: cid-8064a8ef19179c31cdce01a833847ff530c1f776
### What changes are proposed in this pull request? Add call points of the new metrics ### Why are the changes needed? The new metrics are already defined in the previous pr, and need to be called in this pr. ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18351 change-id: cid-1b35a87f41a1f836005c368881378329f4c77b25
Fixed a single spelling error. ### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18310 change-id: cid-77c21de69cf13a6c94be418adf2be95e0f347d42
### What changes are proposed in this pull request? clean up unmount of fuse ### Why are the changes needed? alluxio-fuse may cause AlluxioFuse continues to hang, and also the current flow to unmount is scattered and all over the place. ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18241 change-id: cid-0246bd831466396697efe1977547d365ad63ba3d
### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18412 change-id: cid-fa4b35605d8f97d79c159140613aacf27d3c1836
Fix the bug that `HdfsUfsStatusIterator` returns wrong path (the getName() method of `UfsStatus` should return a relative path, but not just the file name). pr-link: Alluxio#18424 change-id: cid-8b15fbd345f26562a06c5d9f52bdbc90a53b2039
Fix the bug that HDFS iterator doesn't work. pr-link: Alluxio#18429 change-id: cid-dbb5f1a246bce706f7b3b5d179ad158218dccf83
### What changes are proposed in this pull request? Shade all the class under javax ### Why are the changes needed? We need this change to fix the maven build error in presto ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18450 change-id: cid-91aa8f5a00867b0f44165eb486fa168cc710504c
### What changes are proposed in this pull request? Add a `WorkerClusterView` to allow more efficiently indexing and filtering workers by different criteria. ### Why are the changes needed? The current return type `List<WorkerInfo>` of the methods `getAllWorkers` etc. of `MembershipManager` does not allow efficiently indexing and filtering workers, e.g. getting a worker by it ID, finding all lost workers. This is partially motivated by the review comments in Alluxio#18434 (comment) ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18441 change-id: cid-e28beb96752cc45cdad973b98f2d689ce1581b0f
### What changes are proposed in this pull request? Fix the shading issue for javax security ### Why are the changes needed? you might see class not found in javax's security module ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18455 change-id: cid-d04ba461930b7abba2613810e8d7d2ae77d7e3be
### What changes are proposed in this pull request? Make the buffer size in s3 proxy configurable, which is currently 8KB and unchangeable. pr-link: Alluxio#18459 change-id: cid-7ebb81819b6f13222496a341cbef743cf50bda7b
### What changes are proposed in this pull request? 1. Move `WorkerState` enum from master package to the wire package. 2. Add a new worker state `UNRECOGNIZED` and use it as the default state. 3. Make worker state in `WorkerInfo` an enum. 4. Add copy constructors to `WorkerInfo` and `WorkerNetAddress`. ### Why are the changes needed? 1. Make sure the state of worker can be enumerated. 2. Allow safely copying mutable `WorkerInfo` and `WorkerNetAddress` objects. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18460 change-id: cid-8daf9c1e3ebe8e862a9b0dabb669c80918f5b8b0
### What changes are proposed in this pull request? Put back the restriction of reuse worker identity for non-k8s env deployment. ### Why are the changes needed? Worker identity gets reused by wrong deployment behaviors such as copy conf/ over for new worker setup, as opposed to k8s deployment is thru operator / automation, bare metal deployment has no way of prevention, thus putting back the restriction for non-k8s deployment. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18454 change-id: cid-295ea352895b16c2a5f0a23fa790c9f42a5e3881
### What changes are proposed in this pull request? Replace `List<BlockWorkerInfo>` with `WorkerClusterView` in APIs. Important APIs that are changed: 1. `FileSystemContext.getCachedWorkers` now returns `WorkerClusterView` 2. `WorkerLocationPolicy.getPreferredWorkers` (as well as all its implementations) now accepts a `WorkerClusterView` as the first argument (but still returns `List<BlockWorkerInfo>` as the returned list must be ordered) APIs that are using `List<BlockWorkerInfo>` (or `List<WorkerInfo>`) but *not* migrated to `WorkerClusterView`: 1. `alluxio.master.scheduler.WorkerProvider.getWorkerInfos` returns `List<WorkerInfo>`. 2. Job service related APIs, e.g. `alluxio.job.plan.PlanDefinition.selectExecutors` Notable behavior change: Now `EtcdMembershipManager` assigns the correct state (`LIVE` or `LOST`) for all workers in its `WorkerInfo` struct. Before this change, this information is not available and the state defaults to `UNRECOGNIZED`. ### Why are the changes needed? Allow more efficient indexing and filtering workers by worker ID. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18451 change-id: cid-5052d2faa506f4de6e4b0df7062c5def3e09df1c
### What changes are proposed in this pull request? Fix outdated worker address info returned by consistent hash policy Summary of changes: 1. `ConsistentHashProvider` only concerns about `WorkerIdentity` when building the hash ring. Therefore, the APIs have been limited to accept and return `WorkerIdentity`s. 2. `ConsistentHashProvider.refresh` now accepts a set of worker identities instead of a list, as the order does not matter. 3. Added a test to cover the bug fix. ### Why are the changes needed? Fix a bug where the consistent hash provider caches the `BlockWorkerInfo` of all workers, and when a worker changes its network addresses but its ID stays the same, the hash provider won't update the worker's info. A client will continue to use the outdated network address. The fix is to make the hash provider to only consider `WorkerIdentity`s, and let the client to figure out the worker's address with the ID provided by the consistent hash provider. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18434 change-id: cid-93f1601d846385f314c79a556c8705d3983a1199
This PR improves RESTful load api, use JSON format as the response content.
## Examples:
### Submit Job Example:
```
// 20231211174310
// http://localhost:28080/v1/load?path=hdfs://node01:8020/testRoot/testDirectory2&opType=submit&verbose=true
{
"success": true,
"jobId": "0dbc0f47-580b-420f-b50e-d08a170746c8",
"path": "hdfs://node01:8020/testRoot/testDirectory2",
"message": "Load 'hdfs://node01:8020/testRoot/testDirectory2' is successfully submitted. JobId: 0dbc0f47-580b-420f-b50e-d08a170746c8\n"
}
```
### Get Job Progress Example:
```
// 20231211180110
// http://localhost:28080/v1/load?path=hdfs://node01:8020/testRoot/testDirectory2&opType=progress&verbose=true
{
"jobState": "RUNNING",
"path": "hdfs://node01:8020/testRoot/testDirectory2",
"message": "Progress for loading path 'hdfs://node01:8020/testRoot/testDirectory2':\n\tSettings:\tbandwidth: unlimited\tverify: false\tmetadata-only: false\n\tTime Elapsed: 00:00:03\n\tJob State: RUNNING\n\tStage: RETRYING\n\tInodes Scanned: 4\n\tInodes Processed: 4\n\tBytes Loaded: 0B out of 0B\n\tThroughput: 0B/s\n\tFile Failure rate: 0.00%\n\tSubtask Failure rate: 0.00%\n\tFiles Failed: 0\n\tRecent failed subtasks: \n\tRecent retrying subtasks: \n\tSubtask Retry rate: 0.00%\n\tSubtasks on Retry Dead Letter Queue: 0\n",
"respProperties": {
"Files Failed": "0",
"Recent failed subtasks": "",
"Subtask Retry rate": "0.00%",
"Throughput": "0B/s",
"File Failure rate": "0.00%",
"Subtasks on Retry Dead Letter Queue": "0",
"Time Elapsed": "00",
"Bytes Loaded": "0B out of 0B",
"Stage": "RETRYING",
"Inodes Scanned": "4",
"Inodes Processed": "4",
"Recent retrying subtasks": "",
"Subtask Failure rate": "0.00%",
"Settings": "bandwidth",
"Job State": "RUNNING",
"Progress for loading path 'hdfs": "//node01"
}
}
```
```
// 20231211174358
// http://localhost:28080/v1/load?path=hdfs://node01:8020/testRoot/testDirectory2&opType=progress&verbose=true
{
"jobState": "SUCCEEDED",
"path": "hdfs://node01:8020/testRoot/testDirectory2",
"message": "Progress for loading path 'hdfs://node01:8020/testRoot/testDirectory2':\n\tSettings:\tbandwidth: unlimited\tverify: false\tmetadata-only: false\n\tTime Elapsed: 00:00:16\n\tJob State: SUCCEEDED\n\tInodes Scanned: 4\n\tInodes Processed: 4\n\tBytes Loaded: 0B out of 0B\n\tThroughput: 0B/s\n\tFile Failure rate: 0.00%\n\tSubtask Failure rate: 0.00%\n\tFiles Failed: 0\n\tRecent failed subtasks: \n\tRecent retrying subtasks: \n\tSubtask Retry rate: 0.00%\n\tSubtasks on Retry Dead Letter Queue: 0\n",
"respProperties": {
"Files Failed": "0",
"Recent failed subtasks": "",
"Subtask Retry rate": "0.00%",
"Throughput": "0B/s",
"File Failure rate": "0.00%",
"Subtasks on Retry Dead Letter Queue": "0",
"Time Elapsed": "00",
"Bytes Loaded": "0B out of 0B",
"Inodes Scanned": "4",
"Inodes Processed": "4",
"Recent retrying subtasks": "",
"Subtask Failure rate": "0.00%",
"Settings": "bandwidth",
"Job State": "SUCCEEDED",
"Progress for loading path 'hdfs": "//node01"
}
}
```
### Stop Job Example:
```
// 20231211180219
// http://localhost:28080/v1/load?path=hdfs://node01:8020/testRoot/testDirectory2&opType=stop&verbose=true
{
"success": true,
"path": "hdfs://node01:8020/testRoot/testDirectory2",
"message": "Load 'hdfs://node01:8020/testRoot/testDirectory2' is successfully stopped.\n"
}
```
```
// 20231211180153
// http://localhost:28080/v1/load?path=hdfs://node01:8020/testRoot/testDirectory2&opType=stop&verbose=true
{
"success": false,
"path": "hdfs://node01:8020/testRoot/testDirectory2",
"message": "Cannot find load job for path hdfs://node01:8020/testRoot/testDirectory2, it might have already been stopped or finished\n"
}
```
pr-link: Alluxio#18464
change-id: cid-1fff9a23457064ab71534909449c60a6b0123f22
### What changes are proposed in this pull request? Remove additional white space in alluxio-fuse script ### Why are the changes needed? alluxio-fuse unmount <mnt_point> is unable to find the pid of AlluxioFuse process because the grep content isn't correct. ### Does this PR introduce any user facing changes? No pr-link: Alluxio#18465 change-id: cid-3e70b0c8edbaa0ba50d744fd6155b0d494a243f9
Support get page with RESTful API by specifying offset and length.
PAGE_URL_FORMAT = (
"[http://{worker_host}:{http_port}/v1/file/{path_id}/page/{page_index}?offset=100&length=1024](http://{worker_host}:%7Bhttp_port%7D/v1/file/%7Bpath_id%7D/page/%7Bpage_index%7D?offset=100&length=1024)"
)
pr-link: Alluxio#18474
change-id: cid-ba5b0c5050843ccc5642950beadc8a0b049948be
Support write page with RESTful API ### Usage ``` HTTP Method: POST Request URL: http://localhost:28080/v1/file/<fileId>/page/<pageIndex> HTTP Body: <page bytes> ``` pr-link: Alluxio#18481 change-id: cid-cab175a007bfcaf294e89adbe47531419036a245
``` $ df -h /mnt/fuse/ Filesystem Size Used Avail Use% Mounted on alluxio-fuse 910T 0 910T 0% /mnt/fuse ``` ### What changes are proposed in this pull request? Add fake numbers for statfs ### Why are the changes needed? Some application checks the available space in a file system before continuing to do file operations. ### Does this PR introduce any user facing changes? A fake number (1 Petabytes) is provided to statfs. This number does not reflect real available storage space. pr-link: Alluxio#18482 change-id: cid-9f60d185393b616be02bf8f473b2026f2047f28c
### What changes are proposed in this pull request? Add Ketama Hashing, Jump Consistent Hashing, Maglev Hashing, and Multi Probe Hashing. ### Why are the changes needed? Now alluxio's user worker selection policy is Consistent Hash Policy. It bings too much time cost, and it is not enough uniform, and not strictly consistent. Ketama: https://github.com/RJ/ketama Jump Consistent Hashing: https://arxiv.org/pdf/1406.2294.pdf Maglev Hashing: https://static.googleusercontent.com/media/research.google.com/zh-CN//pubs/archive/44824.pdf Multi Probe Hasing: https://arxiv.org/pdf/1505.00062.pdf We strongly recommend using Maglev Hashing for User Worker Selection Policy. Under most situation, it has the minimum time cost, and it is the most uniform and balanced hashing policy. ### Does this PR introduce any user facing changes? `alluxio.user.worker.selection.policy` has the following values: `CONSISTENT`, `JUMP`, `KETAMA`, `MAGLEV`, `MULTI_PROBE`, `LOCAL`, `REMOTE_ONLY`, corresponding to consistent hash policy, maglev hash policy, ketama hash policy, maglev hash policy, multi-probe respectively hash policy, local worker policy, remote only policy. The current default value is `CONSISTENT`. We recommend using Maglev Hash, which has the best hash consistency and is the least time-consuming. That is to say, set the value of `alluxio.user.worker.selection.policy` to `MAGLEV`. We will also consider setting this as the default value in the future. **Ketama Hasing** `alluxio.user.ketama.hash.replicas`: This is the value of replicas in the ketama hashing algorithm. When workers changes, it will guarantee the hash table is changed only in a minimal. The value of replicas should be X times the physical nodes in the cluster, where X is a balance between efficiency and cost. **Jump Consistent Hashing** None. **Maglev Hashing** `alluxio.user.maglev.hash.lookup.size`: This is the size of the lookup table in the maglev hashing algorithm. It must be a prime number. In the maglev hashing, it will generate a lookup table for workers. The bigger the size of the lookup table, the smaller the variance of this hashing algorithm will be. But bigger look up table will consume more time and memory. **Multi Probe Hashing** `alluxio.user.multi.probe.hash.probe.num`: This is the number of probes in the multi-probe hashing algorithm. In the multi-probe hashing algorithm, the bigger the number of probes, the smaller the variance of this hashing algorithm will be. But more probes will consume more time and memory. pr-link: Alluxio#17817 change-id: cid-bad21c6e5ad83eb3da15a8960ba372b14c67b081
…b.login.autorenewal' in HDFS docs ### What changes are proposed in this pull request? Update the correct kerberos configuration 'alluxio.hadoop.kerberos.keytab.login.autorenewal' in HDFS docs to avoid user confusion. ### Why are the changes needed? Fix Alluxio#18486 pr-link: Alluxio#18487 change-id: cid-b8f08e2f67e5f10aa1426de7629b8e268e339433
### What changes are proposed in this pull request? Create metadata directory in initiateMultipartUpload method. ### Why are the changes needed? Each request calls the initialization method of the handler and sends an `exists` request to the master, which is unnecessary. pr-link: Alluxio#18462 change-id: cid-3efd076d7eb33cc063609fa1e1003e3aff480be6
`format` commands don't exist in cli anymore, so as job services. Delete some entrypoint code. Solves Alluxio#18466 pr-link: Alluxio#18490 change-id: cid-0ee0b45a012a29df4a793107d7f8cce4ca98fc99
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Fix signature issues for some underfs modules
Why are the changes needed?
I encountered the following exceptions while executing "alluxio-fuse mount" with the latest version 308-SNAPSHOT
and checking one of the jar with
jarsigneralso has the same issueAfter some tests, it seems that if the child module also defines a filter of "maven-shade-plugin", then the filter defined in its parent module does not take effect. So I add the needed "exclude" items to each child module as well, and it resolved the issue I encountered.
Does this PR introduce any user facing changes?
NO