Exception: AttributeError: module 'hashlib' has no attribute 'file_digest' #33
Comments
|
👋 @RobertBerger nice to see you here 😄
I just checked and file_digest is indeed a quite new function that has been added in Python 3.11: https://docs.python.org/3/library/hashlib.html#hashlib.file_digest I must admit that I didn't thoroughly validate backwards compatibility with older Python versions. I will check for alternatives that work with older Python versions (earliest next week though). |
|
Ran into the same issue. That's how I "fixed" it temporarily on my site: diff --git a/classes/osselot.bbclass b/classes/osselot.bbclass
index 382aaf6..ba5bcfc 100644
--- a/classes/osselot.bbclass
+++ b/classes/osselot.bbclass
@@ -115,6 +115,14 @@ python do_osselot_create_s_checksums() {
import hashlib
import os
+ def md5sum(f):
+ h = hashlib.md5()
+ b = bytearray(128*1024)
+ mv = memoryview(b)
+ while n := f.readinto(mv):
+ h.update(mv[:n])
+ return h.hexdigest()
+
workdir = d.getVar("WORKDIR")
s = d.getVar("S")
pn = d.getVar("PN")
@@ -149,7 +157,8 @@ python do_osselot_create_s_checksums() {
bb.warn(f"Could not open file {item.path}")
else:
with fb:
- digest = hashlib.file_digest(fb, osselot_hash_algorithm).hexdigest()
+# digest = hashlib.file_digest(fb, osselot_hash_algorithm).hexdigest()
+ digest = md5sum(fb)Obviously this sets the hash algorithm to md5 and cannot be configured anymore. Unfortunately I don't have a generic solution so far. |
|
@PatrickVog thanks for your input! Another short-term alternative to changing source code could be to use pyenv to change the Python version on a per-project basis. I know this is not ideal and I hope we can find a way to support a wider range of Python versions.
Side-note on the "only MD5" thing: Actually, I was already pondering whether I want to drop support for changing the hash algorithm, since we currently do some preconfigured hash equivalence on recipes which will not apply anymore, if another algorithm is used (see: #29, alternative would be to do double the source hashing work for a package, if hash equivalence in another algorithm is present). I don't see the harm in forcing the use of MD5, since we only check for changed source files and don't do anything security relevant. The only thing I am concerned about is whether md5sum will continue to be supported in future SPDX SBOMs. Alternatively we could switch to something like SHA256, however I am not quite sure how much of a difference in avg. calculation time we are talking here. |
|
Just a stupid idea ;) |
Quite like the idea, however that wouldn't fix the issue for kirkstone branch since python version there is 3.10 |
Unfortunately it doesn't seem to be possible. 😔 Which is also why I limited myself to functions that are part of the Python base installation, otherwise users would have to install additional packages on their host system, a no-go in my book. I'll try to dig up the mailinglist thread on this again and post it here. |
|
Found it: https://docs.yoctoproject.org/pipermail/yocto/2017-November/039012.html
So the only way to make this work with python-native would be to move the logic into a standalone script and call in using the nativepython executable in a bitbake task. We could evaluate that (not sure what that means in regards of working with variables from the d datastore), but the issue that python versions shipped with kirkstone and dunfell would also be too old. So we would have to backport a never version of python in this layer. I am not quite sure whether that is a good idea... |
|
Maybe the best course of action is to check how many functions currently used require python > 3.5.0 (the minimal requirement for dunfell) and see how easily these could be replaced. |
|
Well I guess it's either "use something defined" which is shipped with each Yocto version, or use a certain subset, which works on all the supported host Linux versions. Probably this is an interesting question to the Yocto mailing list and/or chat. |
|
Maybe this explains a bit, from the chat: RobertBerger: What's the best practice with respect to (host/native) Python version compatibility when you write a bbclass with Python tasks? RobertBerger: To be more precise: #33 rburton: RobertBerger: if its supported by bitbake for that release then it should work in your class rburton: obviously whoever wrote the class could just say "this needs 3.11" but that would be annoying RobertBerger Well in this case it's about https://docs.python.org/3/library/hashlib.html#hashlib.file_digest which is supported from Python 3.11 RobertBerger: And changing the Python Version just for ONE layer, which does not work with the default Python version on Ubuntu 22 makes me a bit nervous RP: RobertBerger: that is a bit too recent unfortunately :/ RobertBerger: There are already many moving targets, not sure I also want to use an "untested" Python version RobertBerger: And apparently there are people who don't use the latest poky master, so that's even worse I guess. JPEW: RobertBerger: That function is convenient, but it's not hard to roll it yourself, so I suggest doing that for compatiblity RobertBerger: The BitBake readme says 3.8 or newer https://git.yoctoproject.org/poky/tree/bitbake/README#n16 RobertBerger: Is there somewhere not newer then x.x? JPEW: As a user, you can run bitbake with python 3.8 or newer; as a developer, you can't use any python features that are newer than 3.8 RobertBerger: OK that makes sense RobertBerger: Thanks! |
|
Totally agree, I think supporting Python 3.8 is reasonable. I am going to close this issue in favor of #34 which generalizes the problem, if that is OK with you @RobertBerger? On why I didn't check required Python version in the first place: I only discovered the fact that OS python is used quite late during the initial development. Thus, I assumed it would "just work" for everyone with the python-native recipe defined in the poky tree. 😞 |
|
@RobertBerger @PatrickVog FYI, this has been fixed with #35 🙂 |
BB_VERSION = "2.7.2"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "universal"
TARGET_SYS = "arm-resy-linux-gnueabi"
MACHINE = "multi-v7-ml"
DISTRO = "resy"
DISTRO_VERSION = "4.3.66"
TUNE_FEATURES = "arm armv7a vfp thumb neon callconvention-hard"
TARGET_FPU = "hard"
meta
meta-poky
meta-yocto-bsp = "master:3aff9b01e559529bf1ce2aa9b318a0157092add9"
meta-multi-v7-ml-bsp-master = "master:e1831dcf9a0d9ce998e3a7f0d2f42e3caa3a26a4"
meta-u-boot-wic-bsp-master = "master:850440d2a48150520d7d9b989f93e11d8d4fc562"
meta-resy-master = "master:ceacfc5dd8cd85679def4d942f40e370cb532a02"
meta-oe
meta-networking
meta-filesystems
meta-python = "master:dea8afa45ef5a3226c88b062c13284d68f380e18"
meta-my-yocto-layer
meta-my-yocto-images-layer
workspace = ":"
meta-osselot = "master:8608fffeeb1a2a41dc5283eea0601098a76e00a4"
Checking sstate mirror object availability: 100% |#############################################################################################################################| Time: 0:00:02
Sstate summary: Wanted 222 Local 0 Mirrors 0 Missed 222 Current 176 (0% match, 44% complete)
Removing 3 stale sstate objects for arch x86_64: 100% |########################################################################################################################| Time: 0:00:00
NOTE: Executing Tasks
ERROR: binutils-cross-arm-2.41-r0 do_osselot_create_s_checksums: Error executing a python function in exec_func_python() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function:
0001:
*** 0002:do_osselot_create_s_checksums(d)
0003:
File: '/workdir/sources/meta-osselot/classes/osselot.bbclass', lineno: 152, function: do_osselot_create_s_checksums
0148: except:
0149: bb.warn(f"Could not open file {item.path}")
0150: else:
0151: with fb:
*** 0152: digest = hashlib.file_digest(fb, osselot_hash_algorithm).hexdigest()
0153: checksums[os.path.relpath(item.path, s)] = {}
0154: checksums[os.path.relpath(item.path, s)][osselot_hash_algorithm] = digest
0155: write_json(osselot_s_checksums_file, checksums)
0156:}
Exception: AttributeError: module 'hashlib' has no attribute 'file_digest'
ERROR: Logfile of failure stored in: /workdir/build/multi-v7-ml-debug-training-master/tmp/work/x86_64-linux/binutils-cross-arm/2.41/temp/log.do_osselot_create_s_checksums.3833
ERROR: Task (/workdir/sources/poky-training-master/meta/recipes-devtools/binutils/binutils-cross_2.41.bb:do_osselot_create_s_checksums) failed with exit code '1'
NOTE: Tasks Summary: Attempted 237 tasks of which 229 didn't need to be rerun and 1 failed.
NOTE: The errors for this build are stored in /workdir/build/multi-v7-ml-debug-training-master/tmp/log/error-report/error_report_20240208151920.txt
You can send the errors to a reports server by running:
send-error-report /workdir/build/multi-v7-ml-debug-training-master/tmp/log/error-report/error_report_20240208151920.txt [-s server]
NOTE: The contents of these logs will be posted in public if you use the above command with the default server. Please ensure you remove any identifying or proprietary information when prompted before sending.
NOTE: Writing buildhistory
NOTE: Writing buildhistory took: 2 seconds
Summary: 1 task failed:
/workdir/sources/poky-training-master/meta/recipes-devtools/binutils/binutils-cross_2.41.bb:do_osselot_create_s_checksums
Summary: There was 1 ERROR message, returning a non-zero exit code.
The text was updated successfully, but these errors were encountered: