Error on create/check, is my repository corrupted? #7538

BBaoVanC · 2023-04-23T01:29:44Z

BBaoVanC
Apr 23, 2023

I'm using borgmatic, and I got a notification from BorgBase that I haven't had any backups for a day. I looked at my logs and I see that it's failing with an error. Here's what I get when trying with -v 2:

ae6e261dfd3d:/# borgmatic --stats --files -v 2
Ensuring legacy configuration is upgraded
/etc/borgmatic.d/config.yaml: No commands to run for pre-everything hook
borg --version --debug --show-rc
ssh://[REDACTED]: Running actions for repository
/etc/borgmatic.d/config.yaml: No commands to run for pre-actions hook
/etc/borgmatic.d/config.yaml: No commands to run for pre-backup hook
ssh://[REDACTED]: Creating archive
ssh://[REDACTED]: Calling postgresql_databases hook function remove_database_dumps
ssh://[REDACTED]: Removing PostgreSQL database dumps
ssh://[REDACTED]: Calling mysql_databases hook function remove_database_dumps
ssh://[REDACTED]: Removing MySQL database dumps
ssh://[REDACTED]: Calling mongodb_databases hook function remove_database_dumps
ssh://[REDACTED]: Removing MongoDB database dumps
ssh://[REDACTED]: Calling sqlite_databases hook function remove_database_dumps
ssh://[REDACTED]: Removing SQLite database dumps
ssh://[REDACTED]: Calling mysql_databases hook function dump_databases
ssh://[REDACTED]: Dumping MySQL databases
ssh://[REDACTED]: Querying for "all" MySQL databases to dump
mysql --host mysql --protocol tcp --user root --skip-column-names --batch --execute show schemas
ssh://[REDACTED]: Dumping MySQL database "all" to /root/.borgmatic/mysql_databases/mysql/all
mysqldump --add-drop-database --host mysql --protocol tcp --user root --databases creative_coreprotect luckperms --result-file /root/.borgmatic/mysql_databases/mysql/all
ssh://[REDACTED]: Collecting special file paths
borg create --exclude-from /tmp/tmpngsnyhen --compression auto,zstd --one-file-system --read-special --lock-wait 60 --list --filter AMEx- ssh://[REDACTED]::borgmatic-{now:%Y-%m-%dT%H:%M:%S.%f} /mnt/source /root/.borgmatic --dry-run --list
borg create --exclude-from /tmp/tmpngsnyhen --compression auto,zstd --one-file-system --read-special --lock-wait 60 --list --filter AMEx- ssh://[REDACTED]::borgmatic-{now:%Y-%m-%dT%H:%M:%S.%f} /mnt/source /root/.borgmatic --stats --debug --show-rc
using builtin fallback logging configuration
33 self tests completed in 0.11 seconds
SSH command line: ['ssh', '[REDACTED]@[REDACTED].repo.borgbase.com', 'borg', 'serve', '--debug']
Remote: using builtin fallback logging configuration
Remote: borg selftest disabled via BORG_SELFTEST env variable
Remote: using builtin fallback logging configuration
Remote: Initialized logging system for JSON-based protocol
Remote: Resolving repository path b'/./repo'
Remote: Resolved repository path to '/srv/repos/[REDACTED]/repo'
Remote: Verified integrity of /srv/repos/[REDACTED]/repo/index.126865
Remote: Traceback (most recent call last):
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1453, in get_fd
    ts, fd = self.fds[segment]
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/lrucache.py", line 21, in __getitem__
    value = self._cache[key]  # raise KeyError if not found
KeyError: 126862
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/remote.py", line 240, in serve
    res = f(**args)
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1204, in get
    return self.io.read(segment, offset, id)
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1569, in read
    fd = self.get_fd(segment)
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1455, in get_fd
    fd = open_fd()
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1436, in open_fd
    fd = open(self.segment_filename(segment), 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/srv/repos/[REDACTED]/repo/data/126/126862'
RemoteRepository: 198 B bytes sent, 4.34 kB bytes received, 4 messages sent
Traceback (most recent call last):
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1453, in get_fd
    ts, fd = self.fds[segment]
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/lrucache.py", line 21, in __getitem__
    value = self._cache[key]  # raise KeyError if not found
KeyError: 126862
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/remote.py", line 240, in serve
    res = f(**args)
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1204, in get
    return self.io.read(segment, offset, id)
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1569, in read
    fd = self.get_fd(segment)
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1455, in get_fd
    fd = open_fd()
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1436, in open_fd
    fd = open(self.segment_filename(segment), 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/srv/repos/[REDACTED]/repo/data/126/126862'
Borg server: Platform: Linux box-us7.borgbase.com 4.18.0-425.19.2.el8_7.x86_64 #1 SMP Fri Mar 17 01:52:38 EDT 2023 x86_64
Borg server: Linux: Unknown Linux
Borg server: Borg: 1.2.4  Python: CPython 3.9.13 msgpack: 1.0.3 fuse: None [pyfuse3,llfuse]
Borg server: PID: 1381721  CWD: /srv/repos/[REDACTED]
Borg server: sys.argv: ['/opt/venvs/borg-LATEST/bin/borg', 'serve', '--restrict-to-repository', '/srv/repos/[REDACTED]/repo', '--storage-quota', '100000M']
Borg server: SSH_ORIGINAL_COMMAND: 'borg serve --debug'
Platform: Linux ae6e261dfd3d 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64
Linux: Unknown Linux
Borg: 1.2.4  Python: CPython 3.11.2 msgpack: 1.0.5 fuse: llfuse 1.4.2 [pyfuse3,llfuse]
PID: 51  CWD: /
sys.argv: ['/usr/local/bin/borg', 'create', '--exclude-from', '/tmp/tmpngsnyhen', '--compression', 'auto,zstd', '--one-file-system', '--read-special', '--lock-wait', '60', '--list', '--filter', 'AMEx-', 'ssh://[REDACTED]::borgmatic-{now:%Y-%m-%dT%H:%M:%S.%f}', '/mnt/source', '/root/.borgmatic', '--stats', '--debug', '--show-rc']
SSH_ORIGINAL_COMMAND: None
terminating with error status, rc 2
ssh://[REDACTED]: Error running actions for repository
Command 'borg create --exclude-from /tmp/tmpngsnyhen --compression auto,zstd --one-file-system --read-special --lock-wait 60 --list --filter AMEx- ssh://[REDACTED]::borgmatic-{now:%Y-%m-%dT%H:%M:%S.%f} /mnt/source /root/.borgmatic --stats --debug --show-rc' returned non-zero exit status 2.
/etc/borgmatic.d/config.yaml: No commands to run for on-error hook
/etc/borgmatic.d/config.yaml: An error occurred
/etc/borgmatic.d/config.yaml: No commands to run for post-everything hook

summary:
/etc/borgmatic.d/config.yaml: An error occurred
ssh://[REDACTED]: Error running actions for repository
...
Traceback (most recent call last):
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/remote.py", line 240, in serve
    res = f(**args)
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1204, in get
    return self.io.read(segment, offset, id)
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1569, in read
    fd = self.get_fd(segment)
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1455, in get_fd
    fd = open_fd()
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1436, in open_fd
    fd = open(self.segment_filename(segment), 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/srv/repos/[REDACTED]/repo/data/126/126862'
Borg server: Platform: Linux box-us7.borgbase.com 4.18.0-425.19.2.el8_7.x86_64 #1 SMP Fri Mar 17 01:52:38 EDT 2023 x86_64
Borg server: Linux: Unknown Linux
Borg server: Borg: 1.2.4  Python: CPython 3.9.13 msgpack: 1.0.3 fuse: None [pyfuse3,llfuse]
Borg server: PID: 1381721  CWD: /srv/repos/[REDACTED]
Borg server: sys.argv: ['/opt/venvs/borg-LATEST/bin/borg', 'serve', '--restrict-to-repository', '/srv/repos/[REDACTED]/repo', '--storage-quota', '100000M']
Borg server: SSH_ORIGINAL_COMMAND: 'borg serve --debug'
Platform: Linux ae6e261dfd3d 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64
Linux: Unknown Linux
Borg: 1.2.4  Python: CPython 3.11.2 msgpack: 1.0.5 fuse: llfuse 1.4.2 [pyfuse3,llfuse]
PID: 51  CWD: /
sys.argv: ['/usr/local/bin/borg', 'create', '--exclude-from', '/tmp/tmpngsnyhen', '--compression', 'auto,zstd', '--one-file-system', '--read-special', '--lock-wait', '60', '--list', '--filter', 'AMEx-', 'ssh://[REDACTED]::borgmatic-{now:%Y-%m-%dT%H:%M:%S.%f}', '/mnt/source', '/root/.borgmatic', '--stats', '--debug', '--show-rc']
SSH_ORIGINAL_COMMAND: None
terminating with error status, rc 2
Command 'borg create --exclude-from /tmp/tmpngsnyhen --compression auto,zstd --one-file-system --read-special --lock-wait 60 --list --filter AMEx- ssh://[REDACTED]::borgmatic-{now:%Y-%m-%dT%H:%M:%S.%f} /mnt/source /root/.borgmatic --stats --debug --show-rc' returned non-zero exit status 2.

Need some help? https://torsion.org/borgmatic/#issues

I decided to try borgmatic check, but it's also giving an error:

ae6e261dfd3d:/# borgmatic check -v 2
Ensuring legacy configuration is upgraded
borg --version --debug --show-rc
ssh://[REDACTED]@[REDACTED].repo.borgbase.com/./repo: Running actions for repository
/etc/borgmatic.d/config.yaml: No commands to run for pre-actions hook
/etc/borgmatic.d/config.yaml: No commands to run for pre-check hook
ssh://[REDACTED]@[REDACTED].repo.borgbase.com/./repo: Running consistency checks
borg info --lock-wait 60 --json ssh://[REDACTED]@[REDACTED].repo.borgbase.com/./repo
Traceback (most recent call last):

  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1453, in get_fd
    ts, fd = self.fds[segment]

  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/lrucache.py", line 21, in __getitem__
    value = self._cache[key]  # raise KeyError if not found

KeyError: 126862


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/remote.py", line 240, in serve
    res = f(**args)

  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1204, in get
    return self.io.read(segment, offset, id)

  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1569, in read
    fd = self.get_fd(segment)

  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1455, in get_fd
    fd = open_fd()

  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1436, in open_fd
    fd = open(self.segment_filename(segment), 'rb')

FileNotFoundError: [Errno 2] No such file or directory: '/srv/repos/[REDACTED]/repo/data/126/126862'

Borg server: Platform: Linux box-us7.borgbase.com 4.18.0-425.19.2.el8_7.x86_64 #1 SMP Fri Mar 17 01:52:38 EDT 2023 x86_64
Borg server: Linux: Unknown Linux  
Borg server: Borg: 1.2.4  Python: CPython 3.9.13 msgpack: 1.0.3 fuse: None [pyfuse3,llfuse]
Borg server: PID: 1381836  CWD: /srv/repos/[REDACTED]
Borg server: sys.argv: ['/opt/venvs/borg-LATEST/bin/borg', 'serve', '--restrict-to-repository', '/srv/repos/[REDACTED]/repo', '--storage-quota', '100000M']
Borg server: SSH_ORIGINAL_COMMAND: 'borg serve'
Platform: Linux ae6e261dfd3d 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64
Linux: Unknown Linux  
Borg: 1.2.4  Python: CPython 3.11.2 msgpack: 1.0.5 fuse: llfuse 1.4.2 [pyfuse3,llfuse]
PID: 55  CWD: /
sys.argv: ['/usr/local/bin/borg', 'info', '--lock-wait', '60', '--json', 'ssh://[REDACTED]@[REDACTED].repo.borgbase.com/./repo']
SSH_ORIGINAL_COMMAND: None

ssh://[REDACTED]@[REDACTED].repo.borgbase.com/./repo: Error running actions for repository
Command '('borg', 'info', '--lock-wait', '60', '--json', 'ssh://[REDACTED]@[REDACTED].repo.borgbase.com/./repo')' returned non-zero exit status 2.
/etc/borgmatic.d/config.yaml: No commands to run for on-error hook
/etc/borgmatic.d/config.yaml: An error occurred

summary:
/etc/borgmatic.d/config.yaml: An error occurred
ssh://[REDACTED]@[REDACTED].repo.borgbase.com/./repo: Error running actions for repository
Command '('borg', 'info', '--lock-wait', '60', '--json', 'ssh://[REDACTED]@[REDACTED].repo.borgbase.com/./repo')' returned non-zero exit status 2.

Need some help? https://torsion.org/borgmatic/#issues

What should I do? I have already tried renaming the borg cache directory, which did not help.

ThomasWaldmann · 2023-04-23T18:17:22Z

ThomasWaldmann
Apr 23, 2023
Maintainer

FileNotFoundError: [Errno 2] No such file or directory: '/srv/repos/[REDACTED]/repo/data/126/126862'

Looks like you maybe have lost a file with some data.

You can try borg check --repair -v --progress ...

11 replies

ThomasWaldmann Apr 24, 2023
Maintainer

The repository part of the check didn't complain, so it seems that all the segment files still there were OK (crc32 check ok).

But, guess that one segment file was really lost somehow and contained all these archive metadata blocks it could not find any more while doing the (more high-level) archives part of the check.

So, the primary question is how that one segment file went missing, but guess that might be difficult or impossible to find out.

ThomasWaldmann Apr 24, 2023
Maintainer

@m3nu can you check if disk / fs is ok?

m3nu Apr 24, 2023
Maintainer

Yes, there was a pretty bad power equipment failure in one of the older US data centers we still have a few servers in and the file system of one machine needed repair despite running with a backup battery. Everyone affected was notified on the same day or via the usual monitoring alerts. Basically some data files touched just before the outage ended up in lost+found. Hence the error above.

As always, for BorgBase-specific issues, just email [email protected].

BBaoVanC Apr 24, 2023
Author

Yes, there was a pretty bad power equipment failure in one of the older US data centers we still have a few servers in and the file system of one machine needed repair despite running with a backup battery. Everyone affected was notified on the same day or via the usual monitoring alerts. Basically some data files touched just before the outage ended up in lost+found. Hence the error above.

As always, for BorgBase-specific issues, just email [email protected].

Hmm I did not get any notification apart from the automated notice from there not being backups for over 1 day (since they failed)...

I didn't think far enough ahead to make a copy of the repository in some way before running the repair... is it out of any chance of repair?

m3nu Apr 24, 2023
Maintainer

Hard to tell without even knowing the repo ID or the server it actually used. Really best to just email to the address above.

witten · 2023-04-24T20:16:10Z

witten
Apr 24, 2023

You know what, I bet it got stuck waiting for me to type YES on the repair (but I can't through borgmatic)

borgmatic switches to an interactive console (no logging capture) when using check --repair for exactly that reason—so you can type "YES" or "NO" if that prompt comes up.

The borgmatic traceback (I'm assuming resulting from ctrl-C?) just indicates it was waiting for output from another process (in this case Borg).

1 reply

BBaoVanC Apr 24, 2023
Author

Hmm, it didn't even give me the prompt message saying to type YES when running through borgmatic (note I was using borgmatic borg check ... instead of borgmatic check ...)

witten · 2023-04-24T21:41:34Z

witten
Apr 24, 2023

Ah, yes, borgmatic borg isn't smart enough to know whether to switch between interactive output or not, because it doesn't parse Borg's arguments (for the most part) and passes them directly to Borg. Instead, use borgmatic check. But if you ever want to drop down to run Borg directly, for instance for debugging purposes as suggested above, I'd recommend running borgmatic with --verbosity 2 and observing the exact Borg command it's invoking. Then you can always copy-and-paste it to run Borg manually. Hope that helps.

1 reply

BBaoVanC Apr 24, 2023
Author

I'd recommend running borgmatic with --verbosity 2 and observing the exact Borg command it's invoking.

Yep that's what I ended up doing earlier, to do the repair :)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error on create/check, is my repository corrupted? #7538

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 13 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Error on create/check, is my repository corrupted? #7538

BBaoVanC Apr 23, 2023

Replies: 3 comments · 13 replies

ThomasWaldmann Apr 23, 2023 Maintainer

ThomasWaldmann Apr 24, 2023 Maintainer

ThomasWaldmann Apr 24, 2023 Maintainer

m3nu Apr 24, 2023 Maintainer

BBaoVanC Apr 24, 2023 Author

m3nu Apr 24, 2023 Maintainer

witten Apr 24, 2023

BBaoVanC Apr 24, 2023 Author

witten Apr 24, 2023

BBaoVanC Apr 24, 2023 Author

BBaoVanC
Apr 23, 2023

Replies: 3 comments 13 replies

ThomasWaldmann
Apr 23, 2023
Maintainer

ThomasWaldmann Apr 24, 2023
Maintainer

ThomasWaldmann Apr 24, 2023
Maintainer

m3nu Apr 24, 2023
Maintainer

BBaoVanC Apr 24, 2023
Author

m3nu Apr 24, 2023
Maintainer

witten
Apr 24, 2023

BBaoVanC Apr 24, 2023
Author

witten
Apr 24, 2023

BBaoVanC Apr 24, 2023
Author