Confusing error when creating a dataset on top of existing

[ChristopherRabotin]

Hi there,

First, thanks for all of your work on this library. There's obviously been a lot of thought and time put into it.

Secondly, I am trying to store some Monte Carlo data in hdf5 using your work: I'm trying to serialize 1-dimension arrays of f64 data. I'm running on OpenSuSE Tumbleweed, hdf5-devel version 1.10.7 and blosc-devel version 1.21.1. When I go about actually serializing the data either an attribute or a dataset, the unwrap fails with nothing more than unknown library error.

Even with RUST_BACKTRACE=full and running a debug build, I get no pointers as to where in the hdf5 library the error happens (full backtrace at the bottom of this issue). The exact line of the error happens either on the attribute or dataset writing, whichever happens first, e.g.

let _ds = builder
            .with_data(&arr1(&data))
            // finalize and write the dataset
            .create(&*format!("Dispersion {}", param))
            .unwrap();

Since this is my first time using this library, it's quite likely that I'm using it incorrectly. At the top of my serialization function, I call blosc_set_nthreads(4); and then, every builder initialization is like so: let builder = run_group.new_dataset_builder().blosc_zstd(9, true);

Solution

I've found that by removing all references to blosc, the serialization is successful, and all of the data is present.

Backtrace

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: unknown library error', src/serialize.rs:140:14
stack backtrace:
   0:     0x55e123f3780c - std::backtrace_rs::backtrace::libunwind::trace::h09f7e4e089375279
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x55e123f3780c - std::backtrace_rs::backtrace::trace_unsynchronized::h1ec96f1c7087094e
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x55e123f3780c - std::sys_common::backtrace::_print_fmt::h317b71fc9a5cf964
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/sys_common/backtrace.rs:67:5
   3:     0x55e123f3780c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::he3555b48e7dfe7f0
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/sys_common/backtrace.rs:46:22
   4:     0x55e123f5a93c - core::fmt::write::h513b07ca38f4fb1b
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/fmt/mod.rs:1149:17
   5:     0x55e123f33ca5 - std::io::Write::write_fmt::haf8c932b52111354
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/io/mod.rs:1697:15
   6:     0x55e123f393f0 - std::sys_common::backtrace::_print::h195c38364780a303
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/sys_common/backtrace.rs:49:5
   7:     0x55e123f393f0 - std::sys_common::backtrace::print::hc09dfdea923b6730
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/sys_common/backtrace.rs:36:9
   8:     0x55e123f393f0 - std::panicking::default_hook::{{closure}}::hb2e38ec0d91046a3
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:211:50
   9:     0x55e123f38fa5 - std::panicking::default_hook::h60284635b0ad54a8
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:228:9
  10:     0x55e123f39aa4 - std::panicking::rust_panic_with_hook::ha677a669fb275654
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:606:17
  11:     0x55e123f39580 - std::panicking::begin_panic_handler::{{closure}}::h976246fb95d93c31
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:502:13
  12:     0x55e123f37cb4 - std::sys_common::backtrace::__rust_end_short_backtrace::h38077ee5b7b9f99a
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/sys_common/backtrace.rs:139:18
  13:     0x55e123f394e9 - rust_begin_unwind
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:498:5
  14:     0x55e1235a26f1 - core::panicking::panic_fmt::h35f3a62252ba0fd2
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/panicking.rs:107:14
  15:     0x55e1235a27e3 - core::result::unwrap_failed::hb53671404b9e33c2
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/result.rs:1613:5
  16:     0x55e1235f3019 - core::result::Result<T,E>::unwrap::h872e53f299f901b9
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/result.rs:1295:23
  17:     0x55e1235e9218 - nyxpeg::serialize::serialize::h88b1d40028d049f1
                               at /home/chris/redacted/src/serialize.rs:136:19
  18:     0x55e1235cd5b2 - montecarlo::main::hfe86ac0737a16e8e
                               at /home/chris/redacted/src/bin/montecarlo/main.rs:132:5
  19:     0x55e1235d0612 - core::ops::function::FnOnce::call_once::h3f8cde54c75e2f42
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/ops/function.rs:227:5
  20:     0x55e1235ba04a - std::sys_common::backtrace::__rust_begin_short_backtrace::h2bd796ed255e3d99
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/sys_common/backtrace.rs:123:18
  21:     0x55e1235e119c - std::rt::lang_start::{{closure}}::hc8d32e34b8ecbcff
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/rt.rs:145:18
  22:     0x55e123f3749b - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h7e688d7cdfeb7e00
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/ops/function.rs:259:13
  23:     0x55e123f3749b - std::panicking::try::do_call::h4be824d2350b44c9
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:406:40
  24:     0x55e123f3749b - std::panicking::try::h0a6fc7affbe5088d
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:370:19
  25:     0x55e123f3749b - std::panic::catch_unwind::h22c320f732ec805e
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panic.rs:133:14
  26:     0x55e123f3749b - std::rt::lang_start_internal::{{closure}}::hd38309c108fe679d
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/rt.rs:128:48
  27:     0x55e123f3749b - std::panicking::try::do_call::h8fcaf501f097a28e
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:406:40
  28:     0x55e123f3749b - std::panicking::try::h20e906825f98acc1
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:370:19
  29:     0x55e123f3749b - std::panic::catch_unwind::h8c5234dc632124ef
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panic.rs:133:14
  30:     0x55e123f3749b - std::rt::lang_start_internal::hc4dd8cd3ec4518c2
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/rt.rs:128:20
  31:     0x55e1235e1160 - std::rt::lang_start::h118c7256a7dd8f86
                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/rt.rs:144:17
  32:     0x55e1235cd6fc - main
  33:     0x7f8178f295c0 - __libc_start_call_main
  34:     0x7f8178f2967c - __libc_start_main_alias_2
  35:     0x55e1235a2eb5 - _start
                               at /home/abuild/rpmbuild/BUILD/glibc-2.34/csu/../sysdeps/x86_64/start.S:116
  36:                0x0 - <unknown>

[mulimoen]

Does the simple example of this crate compile and run for you? I.e. cargo r --example simple --features blosc

[ChristopherRabotin]

Yes, it does.

So after returning to the code about thirty minutes ago, it's become clear that the issue was when I was trying to add a dataset of name X in group Y when a dataset of the same name and in the same group already existed. I called an unwrap() on that call and that triggered the error above. By checking the existence, that issue was resolved.

So I guess that my only recommendation, if at all possible, would be to specify what function call caused the error and maybe add a tip to why the call failed.

Specifically, the following call failed:

            let builder = dispersions_group.new_dataset_builder();
            let _ds = builder
                .with_data(&arr1(&data))
                // finalize and write the dataset
                .create(&*format!("Dispersion {}", param))
                .unwrap();

It would have been helpful if that runtime failure would report `error on creation of dataset "Disperion blah" (tip: does this dataset already exist). Now, I have no idea whether that is possible to given that you're plugging into the C bindings.

[mulimoen]

with_data is a wrapper around many calls, it would indeed be nice if error at the specific layer returned such a hint. We have discussed introducing some helper methods for dataset creating, e.g. #151, which could mimimise the confusion a bit.