Skip to content

Commit

Permalink
KUDU-2411: Update license information for binary test jars
Browse files Browse the repository at this point in the history
This patch updates the license files to account for transitive
dependencies we ship with the binary jar files and updates the scripting
infrastructure to validate the licensing information at binary jar build
time.

This patch also introduces binary-jar-specific license and notice files
to account for licensing information only relevant to the binary
artifacts.

Change-Id: If864384e60013d592c5c7315e2d9a99c83f31449
Reviewed-on: http://gerrit.cloudera.org:8080/12431
Reviewed-by: Grant Henke <[email protected]>
Tested-by: Kudu Jenkins
  • Loading branch information
mpercy committed Feb 11, 2019
1 parent 4f3baf7 commit 420eba8
Show file tree
Hide file tree
Showing 9 changed files with 660 additions and 12 deletions.
7 changes: 4 additions & 3 deletions LICENSE.txt
Original file line number Diff line number Diff line change
Expand Up @@ -479,6 +479,7 @@ src/kudu/util/x509_check_host.*: OpenSSL software license:

OpenSSL License
---------------

====================================================================
Copyright (c) 1998-2016 The OpenSSL Project. All rights reserved.

Expand Down Expand Up @@ -532,8 +533,8 @@ src/kudu/util/x509_check_host.*: OpenSSL software license:
Hudson ([email protected]).


Original SSLeay License
-----------------------
Original SSLeay License
-----------------------

Copyright (C) 1995-1998 Eric Young ([email protected])
All rights reserved.
Expand Down Expand Up @@ -570,7 +571,7 @@ src/kudu/util/x509_check_host.*: OpenSSL software license:
Eric Young ([email protected])"
The word 'cryptographic' can be left out if the rouines from the library
being used are not cryptographic related :-).
4. If you include any Windows specific code (or a derivative thereof) from
4. If you include any Windows specific code (or a derivative thereof) from
the apps directory (application code) you must include an acknowledgement:
"This product includes software written by Tim Hudson ([email protected])"

Expand Down
482 changes: 482 additions & 0 deletions build-support/mini-cluster/LICENSE-BINARY-JAR-LINUX.txt

Large diffs are not rendered by default.

12 changes: 12 additions & 0 deletions build-support/mini-cluster/LICENSE-BINARY-JAR-OSX.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
=======================================================================
License information for macOS binary test jar distribution dependencies
=======================================================================

In addition to dependencies described elsewhere, the below dependencies are
distributed with the macOS version of the Kudu binary test jar.

--------------------------------------------------------------------------------
OpenSSL: OpenSSL software license
libraries: libssl, libcrypto
Source: Homebrew repository

13 changes: 13 additions & 0 deletions build-support/mini-cluster/NOTICE-BINARY-JAR-LINUX.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
===========================================================
Notices for Linux binary test jar distribution dependencies
===========================================================

OpenSSL:

This product includes software developed by the OpenSSL Project
for use in the OpenSSL Toolkit (http://www.openssl.org/)

Cyrus SASL:

This product includes software developed by Computing Services
at Carnegie Mellon University (http://www.cmu.edu/computing/).
9 changes: 9 additions & 0 deletions build-support/mini-cluster/NOTICE-BINARY-JAR-OSX.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
===========================================================
Notices for macOS binary test jar distribution dependencies
===========================================================

OpenSSL:

This product includes software developed by the OpenSSL Project
for use in the OpenSSL Toolkit (http://www.openssl.org/)

31 changes: 25 additions & 6 deletions build-support/mini-cluster/build_mini_cluster_binaries.sh
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ set -e

SOURCE_ROOT=$(cd $(dirname $0)/../..; pwd)
BUILD_ROOT=$SOURCE_ROOT/build/mini-cluster
MINI_CLUSTER_SRCDIR=$SOURCE_ROOT/build-support/mini-cluster
TARGETS="kudu kudu-tserver kudu-master"

cd $SOURCE_ROOT
Expand Down Expand Up @@ -113,7 +114,7 @@ NUM_PROCS=$(getconf _NPROCESSORS_ONLN)
make -j$NUM_PROCS $TARGETS

# Relocate the binaries.
$SOURCE_ROOT/build-support/mini-cluster/relocate_binaries_for_mini_cluster.py $BUILD_ROOT $TARGETS
$MINI_CLUSTER_SRCDIR/relocate_binaries_for_mini_cluster.py $BUILD_ROOT $TARGETS

ARTIFACT_NAME=$(ls -d kudu-binary* | sed 's#/##' | head -1)

Expand Down Expand Up @@ -152,9 +153,24 @@ artifact.version=$ARTIFACT_VERSION
EOF

# Include the basic legal files.
for file in LICENSE.txt NOTICE.txt; do
cp -p $SOURCE_ROOT/$file $ARTIFACT_NAME/
done
# Create a platform-specific NOTICE file.
JAR_NOTICE=$MINI_CLUSTER_SRCDIR/NOTICE-BINARY-JAR-LINUX.txt
if [ $MACOS ]; then
JAR_NOTICE=$MINI_CLUSTER_SRCDIR/NOTICE-BINARY-JAR-OSX.txt
fi
cat $SOURCE_ROOT/NOTICE.txt \
$JAR_NOTICE \
> $ARTIFACT_NAME/NOTICE.txt

# Create a platform-specific LICENSE file.
JAR_LICENSE=$MINI_CLUSTER_SRCDIR/LICENSE-BINARY-JAR-LINUX.txt
if [ $MACOS ]; then
JAR_LICENSE=$MINI_CLUSTER_SRCDIR/LICENSE-BINARY-JAR-OSX.txt
fi
cat $SOURCE_ROOT/LICENSE.txt \
$SOURCE_ROOT/thirdparty/LICENSE.txt \
$JAR_LICENSE \
> $ARTIFACT_NAME/LICENSE.txt

# Include the web UI template files.
cp -Rp $SOURCE_ROOT/www $ARTIFACT_NAME/
Expand All @@ -165,11 +181,14 @@ This archive contains Kudu binaries for use in a "mini cluster" environment for
TESTING ONLY.
The binaries in this archive should never be deployed to run an actual Kudu
service, whether in production or development, because all security
dependencies are copied from the build system and will not be updated if the
service, whether in production or development, because many security-related
dependencies are copied from the build system and will not be patched when the
operating system on the runtime host is patched.
EOF

echo "Running license check on artifact..."
$SOURCE_ROOT/build-support/mini-cluster/check-license.pl $ARTIFACT_NAME

echo Creating archive...
ARTIFACT_FILE=$ARTIFACT_NAME.jar
jar cf $ARTIFACT_FILE $PROP_DIR/ $ARTIFACT_NAME/
Expand Down
84 changes: 84 additions & 0 deletions build-support/mini-cluster/check-license.pl
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
#!/usr/bin/perl
###############################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
###############################################################################
# This script will check a Kudu binary jar distribution to ensure that all
# included shared objects are mentioned in LICENSE.txt;
# The JAR must first be unpacked and this script pointed to the directory
# within the JAR containing the LICENSE.txt file
###############################################################################
use strict;
use warnings;
use File::Basename qw(dirname);

if (scalar @ARGV != 1) {
print STDERR "Usage: $0 binary-jar-unpacked-prefix-dir\n";
print STDERR " Where binary-jar-unpacked-prefix-dir is the directory within the jar\n";
print STDERR " containing the LICENSE.txt file.\n";
exit 1;
}
my $jar_prefix = $ARGV[0];

# Read the CMake config files and parse out the libraries that are part of the
# Kudu project.
my $script_dir = dirname $0;
my $src_root = "$script_dir/../..";
chomp(my @project_deps = `find $src_root/src -name CMakeLists.txt | xargs egrep 'add_library|ADD_EXPORTABLE_LIBRARY'`);
for (@project_deps) {
s/^.*?://; # Strip off leading filename from grep.
s/^[^(]+\(//; # Strip off CMake function / macro name
s/ .*//; # Retain only the first argument to each add_library() call which is the library name.
s/^/lib/; # Prepend "lib" to each library name to match the shared object name.
}

# Read the LICENSE.txt file from the binary test jar and parse out the library
# dependencies.
my $jar_lic_file = "$jar_prefix/LICENSE.txt";
open(FILE, "< $jar_lic_file") or die "Cannot open $jar_lic_file: $!";
chomp(my @contents = grep { /^libraries:/ } <FILE>);
close FILE;
my @external_deps;
foreach my $line (@contents) {
$line =~ s/^libraries: //;
my @deps = split(/,\s*/, $line);
push @external_deps, @deps;
}

# Create a regular expression to determine if there are any libraries shipped
# in the jar file that are not accounted for by either the CMake project files
# or the LICENSE.txt file.
my @pats = map { "\\b$_\\b" } @project_deps, @external_deps;
my $pat_str = join("|", @pats);
my $pat_known_deps = qr($pat_str);

# List the libraries in the binary test jar and print any that don't correspond
# to known deps.
my $seen_unknown_deps = 0;
chomp(my @jar_libs = `cd $jar_prefix && find lib/ -type f`);
foreach my $lib (@jar_libs) {
if ($lib !~ $pat_known_deps) {
print STDERR "unknown license: $lib\n";
$seen_unknown_deps++;
}
}
if (!$seen_unknown_deps) {
print "OK\n";
exit 0;
}
print "Found $seen_unknown_deps unknown dependencies\n";
exit 1;
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,14 @@
librt|
libdl|
libresolv|
libgcc.*
libgcc.*|
libcrypt|
libm|
libkeyutils|
libcom_err|
libdb-[\d.]+|
libselinux|
libtinfo|
)\.so""", re.VERBOSE)

# We don't want to ship libSystem because it includes kernel and thread
Expand All @@ -73,6 +80,7 @@
DiskArbitration$|
IOKit$|
Foundation$|
Kerberos$|
Security$|
SystemConfiguration$|
libCRFSuite|
Expand Down
24 changes: 22 additions & 2 deletions thirdparty/LICENSE.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,13 @@
==========================================================
License information for Kudu third-party dependencies
==========================================================

This directory contains scripts which download and install several third-party
dependencies of Kudu. Most of these dependencies are statically linked into
Kudu binaries, though a few are used only at build-time.
dependencies of Kudu. In production builds of Kudu, most of these dependencies
are statically linked into the Kudu binaries, while in the Kudu binary test
jar, some of these libraries are distributed as dynamically-linked shared
objects. A few of these dependencies are used only at build time and are noted
as such.

--------------------------------------------------------------------------------
thirdparty/lz4-svn/: BSD 2-clause license
Expand Down Expand Up @@ -35,6 +42,7 @@ Source: https://github.com/Cyan4973/lz4

--------------------------------------------------------------------------------
thirdparty/gflags-*/: BSD 3-clause dependency
libraries: libgflags
source: https://github.com/gflags/gflags

Copyright (c) 2006, Google Inc.
Expand Down Expand Up @@ -68,6 +76,7 @@ source: https://github.com/gflags/gflags

--------------------------------------------------------------------------------
thirdparty/glog-*/: BSD 3-clause license
libraries: libglog
Source: https://github.com/google/glog

Copyright (c) 2008, Google Inc.
Expand Down Expand Up @@ -137,6 +146,7 @@ Source: https://github.com/google/glog

--------------------------------------------------------------------------------
thirdparty/gperftools-*/: BSD 3-clause
libraries: libprofiler, libtcmalloc
Source: https://github.com/gperftools/gperftools

Copyright (c) 2005, Google Inc.
Expand Down Expand Up @@ -170,6 +180,7 @@ Source: https://github.com/gperftools/gperftools

--------------------------------------------------------------------------------
thirdparty/libev-*/: BSD 2-clause
libraries: libev
Source: http://software.schmorp.de/pkg/libev.html

All files in libev are
Expand Down Expand Up @@ -245,6 +256,7 @@ Source: https://github.com/henryr/cpp-mustache

--------------------------------------------------------------------------------
thirdparty/protobuf-*/: BSD 3-clause
libraries: libprotobuf
Source: https://github.com/google/protobuf

Copyright 2008, Google Inc.
Expand Down Expand Up @@ -307,6 +319,7 @@ Source: https://github.com/miloyip/rapidjson

--------------------------------------------------------------------------------
thirdparty/snappy-*/: BSD 3-clause
libraries: libsnappy
Source: https://github.com/google/snappy

Copyright 2011, Google Inc.
Expand Down Expand Up @@ -398,6 +411,7 @@ Source: https://github.com/catapult-project/catapult

--------------------------------------------------------------------------------
thirdparty/zlib-*/: Zlib license
libraries: libz
Source: http://www.zlib.net/

(C) 1995-2013 Jean-loup Gailly and Mark Adler
Expand Down Expand Up @@ -497,10 +511,12 @@ Source: http://curl.haxx.se/

--------------------------------------------------------------------------------
thirdparty/crcutil-*/: Apache 2.0 license
libraries: libcrcutil
Source: https://code.google.com/archive/p/crcutil/

--------------------------------------------------------------------------------
thirdparty/libunwind-*/: X11 License
libraries: libunwind
Source: http://savannah.nongnu.org/projects/libunwind/

Permission is hereby granted, free of charge, to any person obtaining
Expand All @@ -524,6 +540,7 @@ Source: http://savannah.nongnu.org/projects/libunwind/

--------------------------------------------------------------------------------
thirdparty/nvml-*/: BSD 3-clause license
libraries: libvmem
Source: https://github.com/pmem/nvml

Copyright (c) 2014-2015, Intel Corporation
Expand Down Expand Up @@ -621,6 +638,7 @@ Source: http://www.boost.org/

--------------------------------------------------------------------------------
thirdparty/thrift-*/: Apache License v2.0
libraries: libthrift
Source: https://thrift.apache.org

================================================================================
Expand All @@ -644,8 +662,10 @@ NOTE: optional build-time dependency

--------------------------------------------------------------------------------
thirdparty/gmock-*/: BSD 3-clause
libraries: libgmock
Source: https://github.com/google/googletest
NOTE: build-time dependency
TODO(KUDU-2696): gmock is unnecessarily linked into the kudu cli binary

--------------------------------------------------------------------------------
thirdparty/google-styleguide-*/: CC-By 3.0 (http://creativecommons.org/licenses/by/3.0/)
Expand Down

0 comments on commit 420eba8

Please sign in to comment.