Skip to content

Commit

Permalink
Initial commit
Browse files Browse the repository at this point in the history
  • Loading branch information
Shubham Tagra committed Jun 28, 2016
0 parents commit 3ce3690
Show file tree
Hide file tree
Showing 44 changed files with 4,703 additions and 0 deletions.
89 changes: 89 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# RubiX

RubiX is a light-weight data caching framework that can be used by Big-Data engines.
RubiX can be extended to support any engine that accesses data in cloud stores using Hadoop FileSystem interface via plugins.
Using the same plugins, RubiX can also be extended to be used with any cloud store

### Usecase

RubiX provides disk or in-memory caching of data, which would otherwise be accessed over network when it resides in cloud store,
thereby improving performance.

### Supported Engines and Cloud Stores

- Presto: Amazon S3 is supported
- Hadoop-1: Any engine using hadoop-1, e.g. Hive can utilize RubiX. Amazon S3 is supported

### How to use it

RubiX has two components: a BookKeeper server and a FileSystem implementation that an engine should use.

List of things to be done to use RubiX are:

1. Start the BookKeeper server. It can be started via `hadoop jar` command, e.g.:
> hadoop jar rubix-bookkeeper-1.0.jar com.qubole.rubix.bookkeeper.BookKeeperServer
2. Engine side changes:
To use RubiX, you need to place the appropriate jars in the classpath and configure Engines to use RubiX filesystem to access the cloud store. Sections below show how to get started on RubiX with supported plugins

##### Using RubiX with Presto

1. Place rubix-bookkeeper.jar, rubix-core.jar, rubix-presto.jar in presto/plugin/hive-hadoop2/ directory.
All these jars are packaged in rubix-presto.tar under assembly module
2. Configuration changes
i. Set configuration to use RubiX filesystem in Presto.
ii. Set "hive.force-local-scheduling=true" in hive.properties
3. Start/Re-start the Presto server

##### Using RubiX with Hive

1. Add RubiX jars: rubix-bookkeeper.jar, rubix-core.jar, rubix-hadoop1 either to hadoop/lib directly or via `add jar` command.
All these jars are packaged in rubix-hadoop1.tar under assembly module
2. Configuration changes: Use following configs to start using RubiX
fs.s3n.impl=com.qubole.rubix.hadoop1.CachingNativeS3FileSystem
fs.s3.impl=com.qubole.rubix.hadoop1.CachingNativeS3FileSystem

### Configurations

##### BookKeeper server configurations
These configurations are to be providing as hadoop configs while started the BookKeeper server

| Configuration | Default | Description |
|------------------------------------------|------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| hadoop.cache.data.bookkeeper.port | 8899 | The port on which BookKeeper server will listen |
| hadoop.cache.data.bookkeeper.max-threads | unbounded | Maximum number of threads BookKeeper can launch |
| hadoop.cache.data.block-size | 1048576 | The size in bytes in which the file is logically divided internally. Higher value means lesser space requirement for metadata but can cause reading of more additional data than needed |
| hadoop.cache.data.dirprefix.list | /media/ephemeral | Prefixes for paths of directories used to store cached data. Final paths created by appending suffix in range [0, 5] followed by fcache. |
| hadoop.cache.data.fullness.percentage | 80 | Percentage of total disk space to use for caching and backing files are deleted in an LRU way. |
| hadoop.cache.data.expiration | unbounded | How long data is kept in cache |
##### FileSystem configurations
These configurations need to be provided by the engine which is going to use RubiX

| Configuration | Default | Description |
|--------------------------------------|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| hadoop.cache.data.enabled | true | Control using cache or not |
| hadoop.cache.data.strict.mode | false | By default RubiX tries not to fail read requests if there are some errors and tries to fallback to reading directly from remote source. Setting this config to true will fail read request if there were errors in RubiX. |
| hadoop.cache.data.location.blacklist | empty | Regex blacklisting locations that should not be cached |

### Monitoring

Client side monitoring is set up right now, stats are published to MBean named `rubix:name=stats`

Engines which provide interface to view jmx stats can see these stats. E.g. in Presto you can run this query to see the stats:
>
```
presto:default> select * from jmx.jmx."rubix:name=stats";
node | cachedreads | extrareadfromremote | hitrate | missrate | readfromcache | readfromremote | remotereads | warmuppenalty
-------------+-------------+---------------------+--------------------+--------------------+--------------------+-------------------+-------------+---------------
presto-vbox | 25 | 1.7881784439086914 | 0.5681818181818182 | 0.4318181818181818 | 18.379225730895996 | 18.55983543395996 | 19 | 0
(1 row)
```

### Building

mvn clean install

> Note that you will need thrift-0.9.0 installed
### Need Help?
You can post your queries to https://groups.google.com/forum/#!forum/rubix-users
76 changes: 76 additions & 0 deletions assembly/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<groupId>com.qubole.rubix</groupId>
<artifactId>rubix-root</artifactId>
<version>1.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>

<artifactId>assembly</artifactId>
<properties>
<main.basedir>${project.parent.basedir}</main.basedir>
</properties>

<dependencies>
<dependency>
<groupId>com.qubole.rubix</groupId>
<artifactId>rubix-bookkeeper</artifactId>
</dependency>

<dependency>
<groupId>com.qubole.rubix</groupId>
<artifactId>rubix-hadoop1</artifactId>
</dependency>

<dependency>
<groupId>com.qubole.rubix</groupId>
<artifactId>rubix-presto</artifactId>
</dependency>

<dependency>
<groupId>com.qubole.rubix</groupId>
<artifactId>rubix-core</artifactId>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<id>rubix-prestom</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<appendAssemblyId>false</appendAssemblyId>
<finalName>rubix-presto-${project.version}</finalName>
<descriptors>
<descriptor>src/main/resources/assemblies/rubix-presto.xml</descriptor>
</descriptors>
</configuration>
</execution>

<execution>
<id>rubix-hadoop1</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<appendAssemblyId>false</appendAssemblyId>
<finalName>rubix-hadoop1-${project.version}</finalName>
<descriptors>
<descriptor>src/main/resources/assemblies/rubix-hadoop1.xml</descriptor>
</descriptors>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>

</project>
19 changes: 19 additions & 0 deletions assembly/src/main/resources/assemblies/rubix-hadoop1.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3 http://maven.apache.org/xsd/assembly-1.1.3.xsd">
<id>tar</id>
<formats>
<format>tar</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>

<dependencySets>
<dependencySet>
<includes>
<include>com.qubole.rubix:rubix-bookkeeper</include>
<include>com.qubole.rubix:rubix-core</include>
<include>com.qubole.rubix:rubix-hadoop1</include>
</includes>
</dependencySet>
</dependencySets>
</assembly>
19 changes: 19 additions & 0 deletions assembly/src/main/resources/assemblies/rubix-presto.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3 http://maven.apache.org/xsd/assembly-1.1.3.xsd">
<id>tar</id>
<formats>
<format>tar</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>

<dependencySets>
<dependencySet>
<includes>
<include>com.qubole.rubix:rubix-bookkeeper</include>
<include>com.qubole.rubix:rubix-core</include>
<include>com.qubole.rubix:rubix-presto</include>
</includes>
</dependencySet>
</dependencySets>
</assembly>
134 changes: 134 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>com.qubole.rubix</groupId>
<artifactId>rubix-root</artifactId>
<packaging>pom</packaging>
<version>1.0-SNAPSHOT</version>
<name>rubix</name>

<scm>
<connection>scm:git:git://github.com/qubole/rubix.git</connection>
<url>http://github.com/qubole/rubix</url>
<developerConnection>scm:git:git://github.com/qubole/rubix.git</developerConnection>
</scm>

<inceptionYear>2016</inceptionYear>
<organization>
<name>Qubole</name>
<url>http://www.qubole.com/</url>
</organization>

<properties>
<main.basedir>${project.basedir}</main.basedir>
<dep.presto.version>0.142</dep.presto.version>
<dep.airlift.version>0.98</dep.airlift.version>
<commons-httpclient.version>3.0.1</commons-httpclient.version>
<dep.hadoop.version>1.2.1</dep.hadoop.version>
</properties>

<modules>
<module>rubix-bookkeeper</module>
<module>rubix-core</module>
<module>rubix-presto</module>
<module>rubix-hadoop1</module>
<module>assembly</module>
</modules>

<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.qubole.rubix</groupId>
<artifactId>rubix-bookkeeper</artifactId>
<version>${project.version}</version>
</dependency>

<dependency>
<groupId>com.qubole.rubix</groupId>
<artifactId>rubix-core</artifactId>
<version>${project.version}</version>
</dependency>

<dependency>
<groupId>com.qubole.rubix</groupId>
<artifactId>rubix-presto</artifactId>
<version>${project.version}</version>
</dependency>

<dependency>
<groupId>com.qubole.rubix</groupId>
<artifactId>rubix-hadoop1</artifactId>
<version>${project.version}</version>
</dependency>

<dependency>
<groupId>com.qubole.rubix</groupId>
<artifactId>assembly</artifactId>
<version>${project.version}</version>
</dependency>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>${dep.hadoop.version}</version>
</dependency>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${dep.hadoop.version}</version>
</dependency>

<dependency>
<groupId>org.weakref</groupId>
<artifactId>jmxutils</artifactId>
<version>1.18</version>
</dependency>

<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>18.0</version>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>

<plugin>
<artifactId>maven-release-plugin</artifactId>
<!-- Older release-plugin has problems deploying to release repo -->
<version>2.5.1</version>
</plugin>

<plugin>
<groupId>com.mycila</groupId>
<artifactId>license-maven-plugin</artifactId>
<version>2.3</version>
<configuration>
<header>${main.basedir}/src/license/header.txt</header>
<includes>
<include>**/*.java</include>
</includes>
</configuration>
<executions>
<execution>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
12 changes: 12 additions & 0 deletions rubix-bookkeeper/log4j.properties
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
log4j.rootLogger=debug, console, R
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.hadoop.log.CustomPatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm} %p %q %c{2}: %m%n

log4j.appender.R=org.apache.log4j.RollingFileAppender
log4j.appender.R.File=/media/ephemeral0/bks.log
log4j.appender.R.MaxFileSize=100KB
log4j.appender.R.MaxBackupIndex=2
log4j.appender.R.layout=org.apache.hadoop.log.CustomPatternLayout
log4j.appender.R.layout.ConversionPattern=%d{yy/MM/dd HH:mm} %p %q %c{2}: %m%n
Loading

0 comments on commit 3ce3690

Please sign in to comment.