Arrow GLib

Arrow GLib is a wrapper library for Arrow C++. Arrow GLib provides C API.

Arrow GLib supports GObject Introspection. It means that you can create language bindings at runtime or compile time.

For example, you can use Apache Arrow from Ruby by Arrow GLib and gobject-introspection gem with the following code:

# Generate bindings at runtime
require "gi"
Arrow = GI.load("Arrow")

# Now, you can access arrow::BooleanArray in Arrow C++ by
# Arrow::BooleanArray
p Arrow::BooleanArray

In Ruby case, you should use red-arrow gem. It's based on gobject-introspection gem. It adds many convenient features to raw gobject-introspection gem based bindings.

Install

You can use packages or build by yourself to install Arrow GLib. It's recommended that you use packages.

We use Meson and Ninja as build tools. If you find problems when installing please see common build problems.

Packages

See install document for details.

How to build by users

Arrow GLib users should use released source archive to build Arrow GLib (replace the version number in the following commands with the one you use):

$ wget 'https://www.apache.org/dyn/closer.lua?action=download&filename=arrow/arrow-12.0.0/apache-arrow-12.0.0.tar.gz' \
    --output-document apache-arrow-12.0.0.tar.gz
$ tar xf apache-arrow-12.0.0.tar.gz
$ cd apache-arrow-12.0.0

You need to build and install Arrow C++ before you build and install Arrow GLib. See Arrow C++ document about how to install Arrow C++.

If you use macOS with Homebrew, you must install required packages.

macOS:

$ brew bundle --file=c_glib/Brewfile
$ meson setup c_glib.build c_glib --buildtype=release
$ meson compile -C c_glib.build
$ sudo meson install -C c_glib.build

Others:

$ meson setup c_glib.build c_glib --buildtype=release
$ meson compile -C c_glib.build
$ sudo meson install -C c_glib.build

How to build by developers

You need to install Arrow C++ before you install Arrow GLib. See Arrow C++ document about how to install Arrow C++.

You need GTK-Doc and GObject Introspection to build Arrow GLib. You can install them by the followings:

On Debian GNU/Linux or Ubuntu:

$ sudo apt install -y -V gtk-doc-tools libgirepository1.0-dev meson ninja-build

On CentOS 7:

$ sudo yum install -y gtk-doc gobject-introspection-devel ninja-build
$ sudo pip3 install meson

On CentOS 8 or later:

$ sudo dnf install -y --enablerepo=powertools gtk-doc gobject-introspection-devel ninja-build
$ sudo pip3 install meson

On macOS with Homebrew:

$ brew bundle --file=c_glib/Brewfile

You can build and install Arrow GLib by the followings:

macOS:

$ XML_CATALOG_FILES=$(brew --prefix)/etc/xml/catalog
$ meson setup c_glib.build c_glib -Dgtk_doc=true
$ meson compile -C c_glib.build
$ sudo meson install -C c_glib.build

Warning

When building Arrow GLib, it typically uses the Arrow C++ installed via Homebrew. However, this can lead to build failures if there are mismatches between the changes in Arrow's GLib and C++ libraries. To resolve this, you may need to reference the Arrow C++ library built locally. In such cases, use the --cmake-prefix-path option with the meson setup command to explicitly specify the library path.

$ meson setup c_glib.build c_glib --cmake-prefix-path=${arrow_cpp_install_prefix} -Dgtk_doc=true

Others:

$ meson c_glib.build c_glib -Dgtk_doc=true
$ meson compile -C c_glib.build
$ sudo meson install -C c_glib.build

Usage

You can use Arrow GLib with C or other languages. If you use Arrow GLib with C, you use C API. If you use Arrow GLib with other languages, you use GObject Introspection based bindings.

C

You can find API reference in the /usr/local/share/gtk-doc/html/arrow-glib/ directory. If you specify --prefix to meson, the directory will be different.

You can find example codes in the example/ directory.

Language bindings

You can use Arrow GLib with non-C languages with GObject Introspection based bindings. Here are languages that support GObject Introspection:

• Ruby: red-arrow gem should be used.

  • Examples: https://github.com/red-data-tools/red-arrow/tree/master/example

• Python: PyGObject should be used. (Note that you should prefer PyArrow over Arrow GLib.)

• Lua: LGI should be used.

  • Examples: example/lua/ directory.

• Go: Go-gir-generator should be used. (Note that you should use Apache Arrow for Go than Arrow GLib.)

See also Projects/GObjectIntrospection/Users - GNOME Wiki! for other languages.

How to run test

Arrow GLib has unit tests. You can confirm that you install Arrow GLib correctly by running unit tests.

You need to install the followings to run unit tests:

• Ruby
• gobject-introspection gem
• test-unit gem

You can install them by the followings:

On Debian GNU/Linux or Ubuntu:

$ sudo apt install -y -V ruby-dev
$ sudo gem install bundler
$ (cd c_glib && bundle install)

On CentOS 7 or later:

$ sudo yum install -y git
$ git clone https://github.com/sstephenson/rbenv.git ~/.rbenv
$ git clone https://github.com/sstephenson/ruby-build.git ~/.rbenv/plugins/ruby-build
$ echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bash_profile
$ echo 'eval "$(rbenv init -)"' >> ~/.bash_profile
$ exec ${SHELL} --login
$ sudo yum install -y gcc make patch openssl-devel readline-devel zlib-devel
$ latest_ruby_version=$(rbenv install --list 2>&1 | grep '^[0-9]' | tail -n1)
$ rbenv install ${latest_ruby_version}
$ rbenv global ${latest_ruby_version}
$ gem install bundler
$ (cd c_glib && bundle install)

On macOS with Homebrew:

$ (cd c_glib && bundle install)

Now, you can run unit tests by the followings:

$ cd c_glib.build
$ BUNDLE_GEMFILE=../c_glib/Gemfile bundle exec ../c_glib/test/run-test.sh

Note

If debugging is necessary, you can proceed using the DEBUGGER option as follows:

$ DEBUGGER=lldb BUNDLE_GEMFILE=../c_glib/Gemfile bundle exec ../c_glib/test/run-test.sh

Common build problems

build failed - /usr/bin/ld: cannot find -larrow

Arrow C++ must be installed to build Arrow GLib. Run make install on Arrow C++ build directory. In addition, on linux, you may need to run sudo ldconfig.

build failed - unable to load http://docbook.sourceforge.net/release/xsl/current/html/chunk.xsl

You need to set the following environment variable on macOS:

$ export XML_CATALOG_FILES="$(brew --prefix)/etc/xml/catalog"

build failed - Symbol not found, referenced from libsource-highlight.4.dylib

You may get the following error on macOS:

dyld: Symbol not found: __ZN5boost16re_detail_10650112perl_matcherIPKcNSt3__19allocatorINS_9sub_matchIS3_EEEENS_12regex_traitsIcNS_16cpp_regex_traitsIcEEEEE14construct_initERKNS_11basic_regexIcSC_EENS_15regex_constants12_match_flagsE
  Referenced from: /usr/local/Cellar/source-highlight/3.1.8_7/lib/libsource-highlight.4.dylib
  Expected in: flat namespace
 in /usr/local/Cellar/source-highlight/3.1.8_7/lib/libsource-highlight.4.dylib

To fix this error, you need to upgrade source-highlight:

$ brew upgrade source-highlight

test failed - Failed to load shared library '...' referenced by the typelib: dlopen(...): dependent dylib '@rpath/...' not found for '...'. relative file paths not allowed '@rpath/...'

You may get the following error on macOS by running test:

(NULL)-WARNING **: Failed to load shared library '/usr/local/lib/libparquet-glib.400.dylib' referenced by the typelib: dlopen(/usr/local/lib/libparquet-glib.400.dylib, 0x0009): dependent dylib '@rpath/libparquet.400.dylib' not found for '/usr/local/lib/libparquet-glib.400.dylib'. relative file paths not allowed '@rpath/libparquet.400.dylib'
        from /Library/Ruby/Gems/2.6.0/gems/gobject-introspection-3.4.3/lib/gobject-introspection/loader.rb:215:in `load_object_info'
        from /Library/Ruby/Gems/2.6.0/gems/gobject-introspection-3.4.3/lib/gobject-introspection/loader.rb:68:in `load_info'
        from /Library/Ruby/Gems/2.6.0/gems/gobject-introspection-3.4.3/lib/gobject-introspection/loader.rb:43:in `block in load'
        from /Library/Ruby/Gems/2.6.0/gems/gobject-introspection-3.4.3/lib/gobject-introspection/repository.rb:34:in `block (2 levels) in each'
        from /Library/Ruby/Gems/2.6.0/gems/gobject-introspection-3.4.3/lib/gobject-introspection/repository.rb:33:in `times'
        from /Library/Ruby/Gems/2.6.0/gems/gobject-introspection-3.4.3/lib/gobject-introspection/repository.rb:33:in `block in each'
        from /Library/Ruby/Gems/2.6.0/gems/gobject-introspection-3.4.3/lib/gobject-introspection/repository.rb:32:in `each'
        from /Library/Ruby/Gems/2.6.0/gems/gobject-introspection-3.4.3/lib/gobject-introspection/repository.rb:32:in `each'
        from /Library/Ruby/Gems/2.6.0/gems/gobject-introspection-3.4.3/lib/gobject-introspection/loader.rb:42:in `load'
        from /Library/Ruby/Gems/2.6.0/gems/gobject-introspection-3.4.3/lib/gobject-introspection.rb:44:in `load'
        from /Users/karlkatzen/Documents/code/arrow-dev/arrow/c_glib/test/run-test.rb:60:in `<main>'
Traceback (most recent call last):
        17: from /Users/karlkatzen/Documents/code/arrow-dev/arrow/c_glib/test/run-test.rb:80:in `<main>'
        16: from /Library/Ruby/Gems/2.6.0/gems/test-unit-3.4.0/lib/test/unit/autorunner.rb:66:in `run'
        15: from /Library/Ruby/Gems/2.6.0/gems/test-unit-3.4.0/lib/test/unit/autorunner.rb:434:in `run'
        14: from /Library/Ruby/Gems/2.6.0/gems/test-unit-3.4.0/lib/test/unit/autorunner.rb:106:in `block in <class:AutoRunner>'
        13: from /Library/Ruby/Gems/2.6.0/gems/test-unit-3.4.0/lib/test/unit/collector/load.rb:38:in `collect'
        12: from /Library/Ruby/Gems/2.6.0/gems/test-unit-3.4.0/lib/test/unit/collector/load.rb:136:in `add_load_path'
        11: from /Library/Ruby/Gems/2.6.0/gems/test-unit-3.4.0/lib/test/unit/collector/load.rb:43:in `block in collect'
        10: from /Library/Ruby/Gems/2.6.0/gems/test-unit-3.4.0/lib/test/unit/collector/load.rb:43:in `each'
         9: from /Library/Ruby/Gems/2.6.0/gems/test-unit-3.4.0/lib/test/unit/collector/load.rb:46:in `block (2 levels) in collect'
         8: from /Library/Ruby/Gems/2.6.0/gems/test-unit-3.4.0/lib/test/unit/collector/load.rb:85:in `collect_recursive'
         7: from /Library/Ruby/Gems/2.6.0/gems/test-unit-3.4.0/lib/test/unit/collector/load.rb:85:in `each'
         6: from /Library/Ruby/Gems/2.6.0/gems/test-unit-3.4.0/lib/test/unit/collector/load.rb:87:in `block in collect_recursive'
         5: from /Library/Ruby/Gems/2.6.0/gems/test-unit-3.4.0/lib/test/unit/collector/load.rb:112:in `collect_file'
         4: from /Library/Ruby/Gems/2.6.0/gems/test-unit-3.4.0/lib/test/unit/collector/load.rb:136:in `add_load_path'
         3: from /Library/Ruby/Gems/2.6.0/gems/test-unit-3.4.0/lib/test/unit/collector/load.rb:114:in `block in collect_file'
         2: from /Library/Ruby/Gems/2.6.0/gems/test-unit-3.4.0/lib/test/unit/collector/load.rb:114:in `require'
         1: from /Users/karlkatzen/Documents/code/arrow-dev/arrow/c_glib/test/test-extension-data-type.rb:18:in `<top (required)>'
/Users/karlkatzen/Documents/code/arrow-dev/arrow/c_glib/test/test-extension-data-type.rb:19:in `<class:TestExtensionDataType>': uninitialized constant Arrow::ExtensionArray (NameError)

You can't use @rpath in Arrow C++. To fix this error, you need to build Arrow C++ with -DARROW_INSTALL_NAME_RPATH=OFF:

$ cmake -S cpp -B cpp.build -DARROW_INSTALL_NAME_RPATH=OFF ...
$ cmake --build cpp.build
$ sudo cmake --build cpp.build --target install