Kudu is a columnar storage manager developed for the Hadoop platform. Kudu shares the common technical properties of Hadoop ecosystem applications: it runs on commodity hardware, is horizontally scalable, and supports highly available operation.
Kudu’s design sets it apart. Some of Kudu’s benefits include:
-
Fast processing of OLAP workloads.
-
Integration with MapReduce, Spark, and other Hadoop ecosystem components.
-
Tight integration with Apache Impala (incubating), making it a good, mutable alternative to using HDFS with Parquet. See Kudu Impala Integration.
-
Strong but flexible consistency model.
-
Strong performance for running sequential and random workloads simultaneously.
-
Efficient utilization of hardware resources.
-
High availability. Tablet Servers and Masters use the Raft Consensus Algorithm. Given a replication factor of
2f+1, ifftablet servers serving a given tablet fail, the tablet is still available.NoteHigh availability for masters is not supported during the public beta.
By combining all of these properties, Kudu targets support for families of applications that are difficult or impossible to implement on current-generation Hadoop storage technologies.
-
The KuduTableInputFormat has changed how it handles scan predicates, including how it serializes predicates to the job configuration object. The new configuration key is "kudu.mapreduce.encoded.predicate". Clients using the TableInputFormatConfigurator should not be affected.
-
The kudu-spark subproject was been renamed to follow naming conventions for scala kudu-spark_2.10
-
KUDU-1306 Scan token API for creating partition-aware scan descriptors. Can be used by clients and query engines to more easily execute parallel scans.
-
Gerrit 2848 Added a kudu datasource for spark which uses the kudu client directly instead of using mapreduce api. Includes predicate pushdowns for spark-sql and spark filters. Parallel retrieval for multiple tablets and column projections. Kudu integration with Spark Example
-
link:http://gerrit.cloudera.org:8080/#/c/2992/ Added ability to update and insert from spark using kudu datasource
-
The clients have longer default timeouts. For Java, the default operation timeout and the default admin operation timeout are now set to 30 seconds instead of 10. The default socket read timeout is now 10 seconds instead of 5. For the C++ client, the default admin timeout is now 30 seconds instead of 10, the default RPC timeout is now 10 seconds instead of 5, and the default scan timeout is now 30 seconds instead of 15.
-
Some default settings related to I/O behavior during flushes and compactions have been changed: The default for
flush_threshold_mbhas been increased from 64MB to 1000MB. The defaultcfile_do_on_finishhas been changed fromclosetoflush. Experiments using YCSB indicate that these values will provide better throughput for write-heavy applications on typical server hardware.
Kudu 0.8.0 delivers incremental features, improvements, and bug fixes over the previous versions.
To upgrade to Kudu 0.8.0, see Upgrade from 0.7.1 to 0.8.0.
-
0.8.0 clients are not fully compatible with servers running Kudu 0.7.1 or lower. In particular, scans that specify column predicates will fail. To work around this issue, upgrade all Kudu servers before upgrading clients.
-
KUDU-431 A simple Flume sink has been implemented.
-
KUDU-839 Java RowError now uses an enum error code.
-
Gerrit 2138 The handling of column predicates has been re-implemented in the server and clients.
-
KUDU-1379 Partition pruning has been implemented for C++ clients (but not yet for the Java client). This feature allows you to avoid reading a tablet if you know it does not serve the row keys you are querying.
-
Gerrit 2641 Kudu now uses
earliest-deadline-firstRPC scheduling and rejection. This changes the behavior of the RPC service queue to prevent unfairness when processing a backlog of RPC threads and to increase the likelihood that an RPC will be processed before it can time out.
-
KUDU-1337 Tablets from tables that were deleted might be unnecessarily re-bootstrapped when the leader gets the notification to delete itself after the replicas do.
-
KUDU-969 If a tablet server shuts down while compacting a rowset and receiving updates for it, it might immediately crash upon restart while bootstrapping that rowset’s tablet.
-
KUDU-1354 Due to a bug in Kudu’s MVCC implementation where row locks were released before the MVCC commit happened, flushed data would include out-of-order transactions, triggering a crash on the next compaction.
-
KUDU-1322 The C++ client now retries write operations if the tablet it is trying to reach has already been deleted.
-
Gerrit 2571 Due to a bug in the Java client, users were unable to close the
kudu-sparkshell because of lingering non-daemon threads.
-
Gerrit 2239 The concept of "feature flags" was introduced in order to manage compatibility between different Kudu versions. One case where this is helpful is if a newer client attempts to use a feature unsupported by the currently-running tablet server. Rather than receiving a cryptic error, the user gets an error message that is easier to interpret. This is an internal change for Kudu system developers and requires no action by users of the clients or API.
Kudu 0.7.1 is a bug fix release for 0.7.0.
-
KUDU-1325 fixes a tablet server crash that could occur during table deletion. In some cases, while a table was being deleted, other replicas would attempt to re-replicate tablets to servers that had already processed the deletion. This could trigger a race condition that caused a crash.
-
KUDU-1341 fixes a potential data corruption and crash that could happen shortly after tablet server restarts in workloads that repeatedly delete and re-insert rows with the same primary key. In most cases, this corruption affected only a single replica and could be repaired by re-replicating from another.
-
KUDU-1343 fixes a bug in the Java client that occurs when a scanner has to scan multiple batches from one tablet and then start scanning from another. In particular, this would affect any scans using the Java client that read large numbers of rows from multi-tablet tables.
-
KUDU-1345 fixes a bug where in some cases the hybrid clock could jump backwards, resulting in a crash followed by an inability to restart the affected tablet server.
-
KUDU-1360 fixes a bug in the kudu-spark module which prevented reading rows with
NULLvalues.
Kudu 0.7.0 is the first release done as part of the Apache Incubator and includes a number of changes, new features, improvements, and fixes.
The upgrade instructions can be found at Upgrade from 0.6.0 to 0.7.0.
-
The C++ client includes a new API,
KuduScanBatch, which performs better when a large number of small rows are returned in a batch. The old API ofvector<KuduRowResult>is deprecated.NoteThis change is API-compatible but not ABI-compatible. -
The default replication factor has been changed from 1 to 3. Existing tables will continue to use the replication factor they were created with. Applications that create tables may not work properly if they assume a replication factor of 1 and fewer than 3 replicas are available. To use the previous default replication factor, start the master with the configuration flag
--default_num_replicas=1. -
The Python client has been completely rewritten, with a focus on improving code quality and testing. The read path (scanners) has been improved by adding many of the features already supported by the C++ and Java clients. The Python client is no longer considered experimental.
-
With the goal of Spark integration in mind, a new
kuduRDDAPI has been added, which wrapsnewAPIHadoopRDDand includes a default source for Spark SQL.
-
The Java client includes new methods
countPendingErrors()andgetPendingErrors()onKuduSession. These methods allow you to count and retrieve outstanding row errors when configuring sessions withAUTO_FLUSH_BACKGROUND. -
New server-level metrics allow you to monitor CPU usage and context switching.
-
Kudu now builds on RHEL 7, CentOS 7, and SLES 12. Extra instructions are included for SLES 12.
-
The file block manager’s performance was improved, but it is still not recommended for real-world use.
-
The master now attempts to spread tablets more evenly across the cluster during table creation. This has no impact on existing tables, but will improve the speed at which under-replicated tabletsare re-replicated after a tablet server failure.
-
All licensing documents have been modified to adhere to ASF guidelines.
-
Kudu now requires an out-of-tree build directory. Review the build instructions for additional information.
-
The
C` client library is now explicitly built against the link:https://gcc.gnu.org/onlinedocs/libstdc/manual/using_dual_abi.html[old gcc5 ABI]. If you use gcc5 to build a Kudu application, your application must use the old ABI as well. This is typically achieved by defining the `_GLIBCXX_USE_CXX11_ABImacro at compile-time when building your application. For more information, see the previous link and link:http://developerblog.redhat.com/2015/02/05/gcc5-and-the-c11-abi/. -
The Python client is no longer considered experimental.
See also Limitations of the Kudu Public Beta. Where applicable, this list adds to or overrides that list.
-
Kudu 0.7 is known to work on RHEL 7 or 6.4 or newer, CentOS 7 or 6.4 or newer, Ubuntu Trusty, and SLES 12. Other operating systems may work but have not been tested.
The 0.6.0 release contains incremental improvements and bug fixes. The most notable changes are:
-
The Java client’s CreateTableBuilder and AlterTableBuilder classes have been renamed to CreateTableOptions and AlterTableOptions. Their methods now also return
thisobjects, allowing them to be used as builders. -
The Java client’s AbstractKuduScannerBuilder#maxNumBytes() setter is now called batchSizeBytes as is the corresponding property in AsyncKuduScanner. This makes it consistent with the C++ client.
-
The "kudu-admin" tool can now list and delete tables via its new subcommands "list_tables" and "delete_table <table_name>".
-
OSX is now supported for single-host development. Please consult its specific installation instructions in OS X.
See also Limitations of the Kudu Public Beta. Where applicable, this list adds to or overrides that list.
-
Kudu 0.6 is known to work on RHEL 6.4 or newer, CentOS 6.4 or newer, and Ubuntu Trusty. Other operating systems may work but have not been tested.
-
The Python client is still considered experimental.
See also Limitations of the Kudu Public Beta. Where applicable, this list adds to or overrides that list.
-
Kudu 0.5 is known to work on RHEL 7 or 6.4 or newer, CentOS 7 or 6.4 or newer, Ubuntu Trusty, and SLES 12. Other operating systems may work but have not been tested.
-
The Python client is considered experimental.
This release of Kudu is a public beta. Do not run this beta release on production clusters. During the public beta period, Kudu will be supported via a public JIRA and a public mailing list, which will be monitored by the Kudu development team and community members. Commercial support is not available at this time.
-
You can submit any issues or feedback related to your Kudu experience via either the JIRA system or the mailing list. The Kudu development team and community members will respond and assist as quickly as possible.
-
The Kudu team will work with early adopters to fix bugs and release new binary drops when fixes or features are ready. However, we cannot commit to issue resolution or bug fix delivery times during the public beta period, and it is possible that some fixes or enhancements will not be selected for a release.
-
We can’t guarantee time frames or contents for future beta code drops. However, they will be announced to the user group when they occur.
-
No guarantees are made regarding upgrades from this release to follow-on releases. While multiple drops of beta code are planned, we can’t guarantee their schedules or contents.
CREATE TABLE-
Impala supports creating and dropping tables using Kudu as the persistence layer. The tables follow the same internal / external approach as other tables in Impala, allowing for flexible data ingestion and querying.
INSERT-
Data can be inserted into Kudu tables in Impala using the same mechanisms as any other table with HDFS or HBase persistence.
UPDATE/DELETE-
Impala supports the
UPDATEandDELETESQL commands to modify existing data in a Kudu table row-by-row or as a batch. The syntax of the SQL commands is chosen to be as compatible as possible to existing solutions. In addition to simpleDELETEorUPDATEcommands, you can specify complex joins in theFROMclause of the query using the same syntax as a regularSELECTstatement. - Flexible Partitioning
-
Similar to partitioning of tables in Hive, Kudu allows you to dynamically pre-split tables by hash or range into a predefined number of tablets, in order to distribute writes and queries evenly across your cluster. You can partition by any number of primary key columns, by any number of hashes and an optional list of split rows. See Schema Design.
- Parallel Scan
-
To achieve the highest possible performance on modern hardware, the Kudu client within Impala parallelizes scans to multiple tablets.
- High-efficiency queries
-
Where possible, Impala pushes down predicate evaluation to Kudu, so that predicates are evaluated as close as possible to the data. Query performance is comparable to Parquet in many workloads.
Items in this list may be amended or superseded by limitations listed in the release notes for specific Kudu releases above.
-
Kudu is primarily designed for analytic use cases and, in the beta release, you are likely to encounter issues if a single row contains multiple kilobytes of data.
-
The columns which make up the primary key must be listed first in the schema.
-
Key columns cannot be altered. You must drop and recreate a table to change its keys.
-
Key columns must not be null.
-
Columns with
DOUBLE,FLOAT, orBOOLtypes are not allowed as part of a primary key definition. -
Type and nullability of existing columns cannot be changed by altering the table.
-
A table’s primary key cannot be changed.
-
Dropping a column does not immediately reclaim space. Compaction must run first. There is no way to run compaction manually, but dropping the table will reclaim the space immediately.
-
Ingest via Sqoop or Flume is not supported in the public beta. The recommended approach for bulk ingest is to use Impala’s
CREATE TABLE AS SELECTfunctionality or use the Kudu Java or C++ API. -
Tables must be manually pre-split into tablets using simple or compound primary keys. Automatic splitting is not yet possible. See Schema Design.
-
Tablets cannot currently be merged. Instead, create a new table with the contents of the old tables to be merged.
-
Replication and failover of Kudu masters is considered experimental. It is recommended to run a single master and periodically perform a manual backup of its data directories.
-
To use Kudu with Impala, you must install a special release of Impala called Impala_Kudu. Obtaining and installing a compatible Impala release is detailed in Kudu’s Impala Integration documentation.
-
To use Impala_Kudu alongside an existing Impala instance, you must install using parcels.
-
Updates, inserts, and deletes via Impala are non-transactional. If a query fails part of the way through, its partial effects will not be rolled back.
-
All queries will be distributed across all Impala hosts which host a replica of the target table(s), even if a predicate on a primary key could correctly restrict the query to a single tablet. This limits the maximum concurrency of short queries made via Impala.
-
No timestamp and decimal type support.
-
The maximum parallelism of a single query is limited to the number of tablets in a table. For good analytic performance, aim for 10 or more tablets per host or use large tables.
-
Impala is only able to push down predicates involving
=,⇐,>=, orBETWEENcomparisons between any column and a literal value, and<and>for integer columns only. For example, for a table with an integer keyts, and a string keyname, the predicateWHERE ts >= 12345will convert into an efficient range scan, whereaswhere name > 'lipcon'will currently fetch all data from the table and evaluate the predicate within Impala.
-
Authentication and authorization are not included in the public beta.
-
Data encryption is not included in the public beta.
-
Potentially-incompatible C++, Java and Python API changes may be required during the public beta.
-
ALTER TABLEis not yet fully supported via the client APIs. MoreALTER TABLEoperations will become available in future betas.
-
The Spark DataFrame implementation is not yet complete.
The following are known bugs and issues with the current beta release. They will be addressed in later beta releases.
-
Building Kudu from source using
gcc4.6 or 4.7 causes runtime and test failures. Be sure you are using a different version ofgccif you build Kudu from source. -
If the Kudu master is configured with the
-log_fsync_alloption, tablet servers and clients will experience frequent timeouts, and the cluster may become unusable. -
If a tablet server has a very large number of tablets, it may take several minutes to start up. It is recommended to limit the number of tablets per server to 100 or fewer. Consider this limitation when pre-splitting your tables. If you notice slow start-up times, you can monitor the number of tablets per server in the web UI.
Apache Kudu (incubating) is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
-
A Quickstart VM is provided to get you up and running quickly.
-
You can install Kudu using provided deb/yum packages.
-
You can install Kudu, in clusters managed by Cloudera Manager, using parcels or deb/yum packages.
-
You can build Kudu from source.
For full installation details, see Kudu Installation.