Tags: Zygo/bees
Tags
bees v0.11-rc4
Bug fixes and cleanup. Fix some unnecessary CPU usage and subvol state
tracking.
Highlights:
* Reduce log level of some common debug messages.
* Fix builds for `openat2` on old linux-libc-headers.
* Fix multiple `beesd` issues.
* Better support for `$BEESHOME` on non-btrfs filesystems.
* Fix `--strip-paths` when starting `bees` from the command line.
* Turn on workaround for `LOGICAL_INO`-vs-clone kernel bug unconditionally.
Performance improvements:
* Disabled a CPU-intensive debug tracing log.
* Stop enumerating subvols in extent scan mode.
* Skip short extents and metadata block groups in extent scan.
Shortlog:
Kai Krakow (2):
context: demote "abandoned toxic match" to debug log level
scripts/beesd: Unshare namespace without systemd
Zygo Blaxell (27):
log: demote a lot of BEESLOGWARN to higher verbosity levels
extent scan: make sure we run every extent crawler once per transaction
openat2: log a warning when we fall back to openat
openat2: supply the missing definitions for building with old headers and new kernel
scripts/beesd: no need for `$BEESHOME` to be a subvol
scripts/beesd: harden the mount options
openat2: #include <linux/types.h> so we can know `__u64`
progress: adjust minimum thresholds for ETA to 10 seconds and 1 GiB of data
readahead: clean up the code, update docs
seeker: turn off debug logging
fs: add do_ioctl_nothrow and fsid methods to btrfs fs info
hash: handle $BEESHOME on non-btrfs
main: the base directory for `--strip-paths` should be root_fd, not cwd
fs: add a ntoa function for chunk types
btrfs-tree: introduce BtrfsDataExtentTreeFetcher to read data extents without metadata
btrfs-tree: drop BtrfsFsTreeFetcher and clean up class comments
fs: add a runtime debug stream for btrfs tree searches
btrfs-tree: connect methods to the debug stream interface
btrfs-tree: add a method to get root backref items to BtrfsRootFetcher
btrfs-tree: harden `rlower_bound` against exceptional objects
roots: rework open_root_nocache to use btrfs-tree
extent scan: refactor BeesCrawl, BeesScanMode*
docs: update event counters after extent scan refactoring and crawl skipping
crawl: deprecate use of BeesCrawl to search the extent tree
extent scan: implement an experimental ordered scan mode
roots: drop unnecessary mutex unlock in stop_request
main: unconditionally enable workaround for the logical_ino-vs-clone kernel bug
bees v0.11-rc3
Doc updates and scheduler improvements. Add support for `openat2`.
Highlights:
* Use `openat2` when available to further reduce symlink attack surface.
* Documentation updates: extent scan now exists, new btrfs features, `btrfs send` interoperability.
* Improve progress tracking during exit/restart cycles.
* Keep idle Tasks off of the normal-priority queue.
Shortlog:
Zygo Blaxell (17):
openat2: create a weak syscall wrapper for it
process: replace crucible::gettid() with a weak symbol
roots: use openat2 instead of openat when available
task: get rid of the `insert_task` method
progress: put a timestamp in the bottom row
docs: update feature interaction lists
docs: update README.md
docs: update kernel bugs page for January 2025
docs: update the bug reporting and status instructions
docs: config.md updates
docs: expand "Threads and load management" to suggest not running bees so much
docs: deprecate the `--workaround-btrfs-send` option
task: add an `insert` method for priority-queueing Tasks by age
task: flatten queues of dependent Tasks
task: do not allow queue loops in priority queueing mode
Revert "roots: use a non-idle task for next_transid"
task: fixes for priority and idle Tasks
bees v0.11-rc2
Addressing some issues raised by rc1.
Highlights:
* Progress report redesign after Github discussion.
* Fix handling of ENOENT errors when a subvol is deleted.
* Added SIGUSR1 (pause) and resume (SIGUSR1) controls to gracefully pause and close file descriptors.
* Detect `btrfs send` and wait for it to finish.
* New `--throttle-factor` option to reduce deferred btrfs work at source.
* Option `--help` now returns exit status 0.
* Fix handling of mixed-bg filesystems.
Shortlog:
Zygo Blaxell (37):
context: add a PROGRESS: header in $BEESSTATUS
context: spell "progress" correctly
roots: avoid copying a BtrfsIoctlSearchKey
bytevector: don't deadlock on operator<<
hexdump: be a little more lock-friendly
fs: drop extra declaration of hexdump
readahead: use the right parameter order when checking the range
fs: set the correct nr_items to 0 in the ENOENT search case
task: start up more worker threads when unpausing
main: add support for pause (SIGUSR1) and resume (SIGUSR2)
task: ignore paused status while calculating dynamic thread count
progress: relabel the inaccurate ETA column
progress: move the "finished" tag to a column where it won't obscure data
bytevector: rename the argument to operator[] to be more descriptive
time: add more methods to support dynamic rate throttling
roots: move the transid cache update into transid_max_nocache()
roots: drop method `transid_re`
context: wait for btrfs send to finish, then try dedupe again
extent scan: no need for "No ref for extent" debug message
bees: bump the file limits again, 512k files and 64k dirs
throttle: add delays to match deferred request rate to btrfs completion rate
options: clean up the parser, prepare for new options with no short form
throttle: add --throttle-factor option to control throttling factor
throttle: track time values more closely
docs: add `--throttle-factor` option
throttle: set default to 0.0
docs: default throttle is zero
throttle: don't hold the multilock during throttle
options: return EXIT_SUCCESS after displaying help message
options: improve message when too many--or too few--path arguments given
extent scan: pick the right block groups for mixed-bg filesystems
extent scan: don't put non-data block groups in the data extent map
roots: don't access a shared variable after releasing a lock
context: report time in scan_one_extent
roots: use a non-idle task for next_transid
progress: rework the progress table after github discussion
progress: rename "ctime" -> "tm_left"
bees v0.11-rc1
Significant performance improvements from a new type of extent scanner, and
removing expensive workarounds for kernel bugs that were fixed years ago.
Highlights:
* Extent scan mode to handle snapshots and replace slow subvol scan modes.
* Extent scan mode is the default, replacing subvol scan modes.
* Extent size sorting to skip over the small extents to dedupe the large extents first.
* Progress reporting and ETA estimation.
* Duplicate read elimination.
* Minimized temporary space usage.
* Less fragmentation and nuisance dedupes.
* Removed most expensive and unnecessary toxic extent workarounds.
* Improved IO scheduling.
* Improved dedupe task scheduling.
* Reduced kernel memory usage.
Zygo Blaxell (50):
lib: fix btrfs_data_container pointer casts for 32-bit userspace on 64-bit kernels
usage: the default scan mode is 1 (independent)
context: reduce the size of LOGICAL_INO buffers
trace: add file and line numbers all the way up the stack
crawl: rename next_transid() to avoid confusion with BeesScanMode::next_transid()
docs: add the 6.10..6.12 delayed refs bug
usage: the default scan mode is 3 (recent)
Revert "context: add experimental code for avoiding tiny extents" because this problem is better solved elsewhere.
fs: get rid of 16 MiB limit on dedupe requests
context: when a task fails to acquire an extent lock, don't go ahead and scan the extent anyway
docs: add allocator regression in 6.0+ kernels
hash: use kernel readahead instead of bees_readahead to prefetch hash table
readahead: inject some sanity at the foundation of an insane architecture
readahead: inject more sanity at the foundation of an insane architecture
docs: event counter updates after readahead sanity improvements
bees: reduce extent ref limit to 9999
docs: resolve_overflow limit is only 655050 when BTRFS_MAX_EXTENT_REF_COUNT is
bees: increase file cache size limits
task: fix try_lock argument description
btrfs-tree: add root refs and extent flags fields
btrfs-tree: accessors for TreeFetcher classes' type and tree values
fs: handle ENOENT within lib
multilock: allow turning it off
counters: fix counter names for scan_eof, scan_no_fd, scanf_deferred_inode
types: add shrink_begin and shrink_end methods for BeesFileRange and BeesRangePair
scan_one_extent: eliminate nuisance dedupes, drop caches after reading data
docs: event counter updates after fixing counter names and scan_one_extent improvements
scan_one_extent: reduce the number of LOGICAL_INO calls before finding a duplicate block range
scan_one_extent: in skip/scan lines, log whether extent is compressed
BeesRangePair: drop the _really_ expensive toxic extent workaround
scan_one_extent: remove the unreadahead after benchmark results
docs: remove "matched_" prefix event counters
table: add a simple text table renderer
fs: add some performance metrics for TREE_SEARCH_V2 calls
task: add an idle queue
extent scan: introduce SCAN_MODE_EXTENT
extent scan: put all the refs in a single Task, sort them, use idle task
extent scan: refactor BeesScanMode so derived classes decide their own scan scheduling
docs: add event counters for extent scan
progress: estimate actual data sizes for progress report
extent scan: fix crawl_map creation
progress: put the progress table in the stats and status files
progress: squeeze the progress table into 80 columns or less
docs: add scan mode 4, "extent"
docs: old missing features are not missing any more
main: set default scan mode to mode 4 (EXTENT)
docs: default scan mode is 4, "extent"
extent scan: don't serialize dedupe and LOGICAL_INO when using extent scan mode
bees: post-kernel-5.7 toxic extent handling
docs: post-5.7 toxic extent handling
Bees v0.10
Mostly maintenance.
Highlights:
* Update kernel bugs list to 6.4.1
* Update docs
* Build fixes for GCC 13 and clang 16
Shortlog:
Zygo Blaxell (17):
docs: update kernel bugs and workarounds list for 6.2.0
docs: update the feature interactions page
docs: simplify the exit-with-SIGTERM description
docs: various gotcha updates
docs: minor changes to how-it-works based on past user questions
docs: update front page
docs: update GCC versions list and clarify markdown statement
docs: add "missing" features that have been in development for some time already
roots: make sure transid_max's computed value isn't max
docs: fill in missing LTS backports for "1119a72e223f btrfs: tree-checker: do not error out if extent ref hash doesn't match"
docs: working around `btrfs send` issues isn't really a feature
btrfs-tree: fix build on clang++16
test: GCC 13 fix for limits.cc
context: downgrade toxic extent workaround message
docs: add IGNORE_OFFSET regression in 6.2..6.3 to kernel bugs list
context: log when LOGICAL_INO returns 0 refs
docs: add vmalloc bug to kernel bugs list
Bees v0.9.3
Two bug fixes related to memory utilization:
* Fix the bees checkpoint progress tracker so that it deallocates
completed work tracking items immediately. If the first work item was
stalled for a while, bees would continue allocating memory to track
new work items, but did not delete the completed items as long as
the first item was incomplete. bees can go through a _large_ number
of work items in a few minutes. In some cases this was resulting in
double-digit numbers of GiB allocated for ProgressTracker objects,
triggering the OOM-killer.
* Fix repeated allocate/free cycles of 16MiB `LOGICAL_INO` extent
reference buffers. This was causing crippling performance losses when
bees runs with mimalloc or jemalloc, and moderate losses with tcmalloc
too. Apparently glibc malloc is quite happy to not deallocate memory,
so it was not affected.
After these fixes, bees memory usage consists of:
* The hash table (user-configurable size)
* About 512 KiB of overhead per 1 GiB of hash table
* 16 MiB per thread for `LOGICAL_INO` extent reference list buffers
* About 0.8-4 MiB of miscellaneous data per thread (read from the filesystem)
These can be reduced further, but they aren't bad enough to warrant a
bug fix release now.
Shortlog:
Zygo Blaxell (3):
ProgressTracker: reduce memory usage with long-running work items
fs: allow BtrfsIoctlLogicalInoArgs to be reused, remove virtual methods
context: create a Pool of BtrfsIoctlLogicalInoArgs objects
Bees v0.9
This release rounds up a number of bug fixes and workarounds.
Highlights:
* Work around a kernel bug which can be triggered by running the
LOGICAL_INO ioctl and dedupe on the same extent at the same time.
* Prevent worker threads from being blocked by extent and inode
locks. Defer the blocked item and find something else for that
worker thread to do.
* Fix the labelling of threads so they aren't all "task_consumer".
* Speed up SIGTERM process termination to have a better chance of
flushing the hash table and crawl state to $BEESHOME before the
process is killed by a service timeout.
* Reduce the hash table writeback rate to 128K/s.
* Reduce the interval between crawl restarts to one transid.
* Add 'recent' scan mode, which dedupes new data in fully scanned
subvols instead of waiting for every old subvol to be scanned.
* Better behavior when there are write errors in $BEESHOME.
* Drop the unused and obsolete `fiemap` and `fiewalk` binaries.
Adam Faiz (1):
docs: fix reference direction
Hilton Chain (1):
beesd: Honor DESTDIR on installation.
Vladimir Panteleev (2):
scripts: Update beescrawl.dat file name after UUID removal
scripts: Remove beescrawl.dat with -f
Zygo Blaxell (80):
cache: add a method to get estimated cache size
ntoa: fix type of mask
fs: export btrfs_compress_type_ntoa
namedptr: add some doxygen, fix the #endif comment
fd: add some doxygen
bees: drop m_parent_ctx
roots: correctly track crawl dirty state
bees: drop bees_sync, we will not need it
multilocker: serialize conflicting parallel operations
bees: use MultiLocker to serialize dedupe and logical_ino
readahead: use emulation
task: get rid of the separate Barrier and BarrierLock
task: get rid of separate Exclusion and ExclusionState
task: don't hold the mutex while disposing of pending Tasks
task: use const for current_consumer
task: simplify clear_queue
task: add a pause() method as an alternative to cancel()
task: delete the queue after deleting all of its children
task: add more Doxygen comments for PairLock
task: increase saved thread name length to 64
task: rescue post-exec queue on Task destruction
task: export load tracking statistics
task: use exponential backoff algorithm to set thread count
context: dump current load tracking stats
context: speed up orderly process termination
context: drop long-dead ExtentWalker code
bees: drop the balance/logical workaround that has been disabled for two years
bytevector: add ostream output with hexdump
bytevector: add some fugly mutexes
bytevector: don't need _all_ of those mutexes
bytevector: do not deadlock in self-assignment
bytevector: validate length in get<T>()
BeesFileRange: coalesce is not used, subtract was never implemented
seeker: backward searching template function
btrfs-tree: introduce lightweight classes for btrfs tree search operations
roots: rework btrfs send workaround using btrfs-tree
roots: use symbolic names for SCAN_MODEs
roots: use scan mode 'independent' by default
roots: organize scan workers by inode instead of extent
context: don't let multiple worker Tasks get stuck on a single extent or inode
context: process PREALLOC extents synchronously in extent's Task worker
roots: improve thread status tracking messages
roots: emit "crawl finished" at the correct time
roots: add 'recent' crawl mode for a mix of new and old data
roots: remove duplicate default scan mode setting
roots: reimplement scan modes using virtual base and methods
docs: update documentation for new 'recent' scan mode
roots: disable recent sorting by max_transid
docs: remove the line discussing 'max_transid' in recent scan mode
roots: run insert_new_crawl from within a Task
context: keep the resolve cache smaller
roots: replace BEES_TRANSID_FACTOR with BEES_TRANSID_POLL_INTERVAL
context: don't forget to retry locked extents
context: don't count MultiLock waiting time in dedup_ms
Merge github PR #148
docs: remove duplicate (and wrong) default scan mode
task: use pthread_setname_np correctly
trace: use pthread_setname wrapper
roots: fix extent lock failure handling
docs: add crawl_again, drop crawl_restart
readahead: report the original size in BEESTOOLONG
btrfs-tree: add chunk items: length and type
btrfs-tree: translate item types for error messages
btrfs-tree: fix whitespace and const
fs: get rid of base class btrfs_ioctl_same_extent_info
main: catch exceptions and exit gracefully
docs: fix broken link in options.md
hash: don't spin when writes fail
context: remove the one call to operator vector<> method in BtrfsIoctlLogicalInoArgs
fiemap, fiewalk: drop dead example/test code
fs: remove duplicate BTRFS_COMPRESS_ definitions
fs: get rid of base class btrfs_ioctl_logical_ino_args
fd: pwrite returns ssize_t not int
fd: FS_IOC_SETFLAGS takes an int* argument not a long*
lib: simplify dependency generation
lib: drop version.cc entirely
src: simplify Makefile
src: bees-version.cc cleanups
test: simplify Makefile
hash: flush the table more slowly
Bees v0.8
This release catches up to a year of compiler and kernel header
development. Some CPU performance hotspots have been cooled.
When the btrfs send workaround is enabled, dedupe in read-only subvols
is now paused, and can be resumed by making the subvol read-write or
disabling the workaround. Previously, the workaround would skip to
the end of read-only subvols, permanently excluding their contents
from dedupe.
Highlights:
* Improved compatibility with new compilers and headers
* Specialize some generic classes for speed
* Better handling of read-only subvols with send workaround
* Fetch fewer objects at a time from btrfs to avoid stale data
* Minor improvements to concurrency and error handling
Shortlog:
Ayla Ounce (1):
Fix beesd script arg parsing to respect PREFIX
Javi Vilarroig (1):
Minimal changes in beesd script to make it functional in my system
Khalil Santana (1):
Get rid of errors by using grep -E
KhalilSantana (1):
Fixes a bad grep pattern caused by dffd6e0
Zygo Blaxell (62):
fs: fix FIEMAP_MAX_OFFSET type silliness in fiemap.h
beesd: add missing RuntimeDirectory
roots: ignore subvol when it is read-only and send workaround is enabled
roots: use const more
endian: fix uint16_t specialization of le_to_cpu
roots: reduce number of objects per TREE_SEARCH_V2, drop BEES_MAX_CRAWL_ITEMS and BEES_MAX_CRAWL_BYTES
error: introduce THROW_CHECK4, the long-awaited sequel to THROW_CHECK3
lib: introduce ByteVector as a replacement for vector<uint8_t> and Spanner
fs: drop virtual do_ioctl methods for btrfs_ioctl_search_key
extentwalker: use default sizing of TREE_SEARCH_V2 buffers
fs: convert vector<uint8_t> and Spanner to ByteVector and rewrite TREE_SEARCH_V2 wrapper
fd: start deprecating vector<uint8_t> for p{read,write}_or_die
bees: deprecate vector<uint8_t> and replace with ByteVector
fd: finish deprecating vector<uint8_t> in IO wrapper functions
spanner: drop Spanner, replaced by ByteVector
string: drop vector_copy_struct, obsoleted by ByteVector
roots: use default nr_items
fs: add an item type parameter to next_min
roots: use the new type argument to next_min
fs: dump the TREE_SEARCH_V2 parameters on exception
context: add experimental code for avoiding tiny extents
context: fix the status message that will never be seen
context: add a comment explaining why we are not adding bees_unreadahead
task: optimize for common case of single following Task
docs: document resolve_overflow
fd: better error messages for pread/pwrite
context: stop using deprecated memset_zero template
lib: deprecate memset_zero template, use C99 compound literals instead
readahead: update comments to reflect bakeoff results
namedptr: concurrency and const cleanup
hash: move the random generator out of bees-hash.cc
bees: clean up #include list
task: delete the move constructor for TaskState
task: concurrency cleanups
docs: remove some stray whitespace
lib: add Uname, a constructor for utsname
hash: add utsname fields to log output
hash: drop bees_unreadahead
resolve: reword the too-many-duplicates exception message
gitignore: clang creates a lot of *.tmp files
docs: add missing 'adjust_offset_hit' counter
context: get rid of resolve (LOGICAL_INO) serializer
bees: style cleanups: const, size_t, symbolic names
fs: yet another const
progress: lock down some const methods
context: use consistent status for dedupe in log and thread note
hash: initialize m_dirty in BeesHashTable
bytevector: introduce BEES_VALGRIND to help work around valgrind
types: member m_fd in BeesFileRange must be protected against data races
docs: update kernel bugs list for 2022-07-29
README: update copyright year 2022
docs: update kernel bugs list for 5.18 ptvf fix
fs: get rid of silly base class that causes build failures now
bytevector: fix length check
extentwalker: drop explicit default constructors
bees: fix deprecated-copy warnings for clang-14
fs: get rid of base class btrfs_data_container
fs: get rid of base class fiemap
roots: make sure we can never get a uint_max transid
roots: sprinkle on some more const
fs: update btrfs compatibility header: add csum types, BTRFS_FS_INFO_FLAG_GENERATION and _METADATA_UUID
fs: make dedupe work again after a really unfortunate build fix
gin66 (1):
Remove duplicated //etc for make install
suorcd (1):
docs: spell "snapshot" correctly
PreviousNext