Skip to content

Commit

Permalink
updated tensorframes and jpmml config
Browse files Browse the repository at this point in the history
  • Loading branch information
root committed Oct 24, 2016
1 parent 0a2e2cf commit 91ae933
Show file tree
Hide file tree
Showing 8 changed files with 318 additions and 1 deletion.
2 changes: 1 addition & 1 deletion bin/cli/pipeline-pyspark-shell.sh
Original file line number Diff line number Diff line change
@@ -1 +1 @@
pyspark --repositories $SPARK_REPOSITORIES --jars $SPARK_SUBMIT_JARS --packages $SPARK_SUBMIT_PACKAGES --py-files=$SPARK_SUBMIT_PYFILES
pyspark --repositories $SPARK_REPOSITORIES --jars $SPARK_SUBMIT_JARS --packages $SPARK_SUBMIT_PACKAGES --py-files $SPARK_SUBMIT_PYFILES
1 change: 1 addition & 0 deletions config/bash/pipeline.bashrc
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ export JPMML_SPARKML_VERSION=1.1.3
export PMML_MODEL_VERSION=1.3.3
export PMML_EVALUATOR_VERSION=1.3.2
export PMML_METRO_VERSION=1.3.3
export SPARK_SUBMIT_PYFILES=/root/pipeline/myapps/pmml/spark/1.6.1/lib/jpmml.py
######

# Dev Install Home (Tools)
Expand Down
Binary file not shown.
11 changes: 11 additions & 0 deletions config/jupyter/.ipython/profile_default/startup/README
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
This is the IPython startup directory

.py and .ipy files in this directory will be run *prior* to any code or files specified
via the exec_lines or exec_files configurables whenever you load this profile.

Files will be run in lexicographical order, so you can control the execution order of files
with a prefix, e.g.::

00-first.py
50-middle.py
99-last.ipy
293 changes: 293 additions & 0 deletions myapps/html/advancedspark.com/index.html.demo
Original file line number Diff line number Diff line change
@@ -0,0 +1,293 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta http-equiv="Content-Style-Type" content="text/css">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="description" content="">
<meta name="keywords" content="Apache, Apache Spark, Spark, Shark, Spork, SQL, SparkSQL, SparkR, Streaming, Storm, S4, SAMOA, MLbase, MLlib, MLI, MLOptimizer, Mahout, R, RStudio, Graph, GraphX, GraphLab, Giraph, Berkeley Data Analytics Stack, BDAS, Big Data, Hadoop, HBase, HDFS, S3, ZooKeeper, Parquet, RCFile, Avro, GZip, LZO, Snappy, SequenceFile, InputFormat, OutputFormat, InputSplit, Fast, Data Analytics, OLTP, OLAP, BlinkDB, Tachyon, Hive, Pig, Cascading, Oozie, Spork, Kite, UDF, DDL, DML, DAG, Directed Acyclic Graph, Scala, Java, Java 8, Python, Schema, Linear Algebra, Matrix, Breeze, Sparse, Dense, Columnar, Database, Analytics, Performance, MapReduce, Tez, Drill, Accumulo, Solr, SolrCloud, ElasticSearch, Lucene, Impala, YARN, Hue, Flume, Sqoop, Whirr, MapR, Cloudera, Hortonworks, Join, GroupBy, Union, Sort, Filter, Map, Reduce, Pipe, Streaming, Machine Learning, PageRank, Lambda Architecture, Ad-Hoc, Interactive, Real-time, Near real-time, Statistics, Linear Regression, Logistic Regression, Feature Selection, Stochastic Gradient Descent, Statistcal Analysis, Clustering, Classification, Collaborative Filtering, Recommendations, Item-Item Recommendations, User-Item Recommendations, Distributed Systems, Fault-Tolerance, Failover, Cluster, EMR, Elastic MapReduce, DynamoDB, RedShift, Kinesis, MQTT, Kafka, Twitter, ZeroMQ, Batch Processing, Akka, Functional Programming, Data Science, Play, ETL, Extract, Transform, Load, Cassandra, VNode, Virtual Node, Hinted Handoff, Circuit Breaker, Alternate Least Squares, Ganglia, Parallel, MPI, Distributed System, Lightning Fast Cluster Computing, TensorFlow, TensorFlow Serving, TensorFlow Distributed, TensorBoard, Multi-armed Bandit, A/B Testing">
<title>Spark After Dark</title>
<link href="css/bootstrap.min.css" rel="stylesheet" type="text/css" media="screen">
<link rel="icon" href="favicon.png" type="image/png" sizes="32x32">
<style>
body {
background-color: darkgray;
color: black;
}
li {
padding: 2px;
display: inline;
list-style-type: none;
position: relative;
width: 70px;
height: 50px;
max-width: 70px;
}
li .like {
position: absolute;
left: 10px;
z-index: 99;
display: none;
}
</style>

<!-- Start of fluxcapacitor Zendesk Widget script -->
<script>/*<![CDATA[*/window.zEmbed||function(e,t){var n,o,d,i,s,a=[],r=document.createElement("iframe");window.zEmbed=function(){a.push(arguments)},window.zE=window.zE||window.zEmbed,r.src="javascript:false",r.title="",r.role="presentation",(r.frameElement||r).style.cssText="display: none",d=document.getElementsByTagName("script"),d=d[d.length-1],d.parentNode.insertBefore(r,d),i=r.contentWindow,s=i.document;try{o=s}catch(e){n=document.domain,r.src='javascript:var d=document.open();d.domain="'+n+'";void(0);',o=s}o.open()._l=function(){var o=this.createElement("script");n&&(this.domain=n),o.id="js-iframe-async",o.src=e,this.t=+new Date,this.zendeskHost=t,this.zEQueue=a,this.body.appendChild(o)},o.write('<body onload="document._l();">'),o.close()}("//assets.zendesk.com/embeddable_framework/main.js","fluxcapacitor.zendesk.com");
/*]]>*/</script>
<!-- End of fluxcapacitor Zendesk Widget script -->

<script type="text/javascript" src="js/jquery-1.12.4.min.js"></script>
<script>
var myId = Math.floor((Math.random() * 100000) + 1);
var allItems = [];
var chosenItemIds = [];

function rate(itemId, rating) {
$("#like" + itemId).show()

chosenItemIds.push(itemId)

var encodedKey = btoa(myId)
var encodedValue = btoa(myId + "," + itemId + "," + rating + "," + location.hostname)

// Call the Kafka Rest Api
$.ajax({
url:'/item-rating-kafka',
type:'POST',
data:'{"records":[{"key":"' + encodedKey + '", "value":"' + encodedValue + '"}]}',
contentType:'application/vnd.kafka.binary.v1+json',
dataType:'json',
success: function(data, status){
//NoOp
}
})
}

function loadItems() {
$.getJSON( "/json/actors.json", function( data ) {
var items = [];
$.each( data, function( arrayIdx, item ) {
var id = item.id
var title = item.title
var imgSrc = item.img

// populate the global allItems
allItems[id] = item

items.push( "<li id='li" + id + "'><a href='javascript:rate(" + id + ", 1)'><img id='img" + id + "' alt='" + title + "' src='" + imgSrc + "' width='70px' height='50px' border='2'></a><img class='like' id='like" + id + "' alt='liked' src='img/like.png' width='50' height='43' border='2'></li>" );
});

$( "<ul/>", {
html: "<div id='items-inner'><hr><h3><center>Please Click 3 Actresses and 3 Actors Total </center></h3><center>" + items.join( "" ) + "</center></div>"
}).appendTo("#items");
});
}

function loadTopK() {
$("#topk").empty()

$.getJSON("/redis/GET/topk", function ( data ) {
var items = [];
var results = data.GET.split(",")
$.each(results, function(arrayIdx, resultId) {
var id = resultId
var title = allItems[id].title
var imgSrc = allItems[id].img

items.push( "<li id='li" + id + "'><img id='img" + id + "' alt='" + title + "' src='" + imgSrc + "' width='70px' height='50px' border='2'></li>" );
});

$( "<ul/>", {
html: "</br><center>" + items.join("") + "</center>"
}).appendTo( "#topk" );
});
}

function loadRecommendationsALS() {
$("#recommendations-als").empty()

$.getJSON("/redis/ZREVRANGE/::recommendations:" + myId + "/0/9", function(data) {
var recs = data.ZREVRANGE
var items = [];
$.each( recs, function( arrayIdx, recId ) {
var id = recId
var title = allItems[id].title
var imgSrc = allItems[id].img

items.push( "<li id='li" + id + "'><img id='img" + id + "' alt='" + title + "' src='" + imgSrc + "' width='70px' height='50px' border='2'></li>" );
});

$( "<ul/>", {
html: "</br><center>" + items.join( "" ) + "</center>"
}).appendTo( "#recommendations-als" );
});
}

function loadItemSimilarsALS(itemId) {
$("#item-similars-als").empty()
var items = [];
$.each( chosenItemIds, function( arrayIdx, itemId ) {
$.getJSON ("/serving/similars/" + itemId + "/0/9", function (data) {
var sims = data.results
$.each( sims, function( arrayIdx, simId ) {
var id = simId
var title = allItems[id].title
var imgSrc = allItems[id].img

items.push( "<li id='li" + id + "'><img id='img" + id + "' alt='" + title + "' src='" + imgSrc + "' width='70px' height='50px' border='2'></li>" );
});

$( "<ul/>", {
html: "</br><center>" + items.join( "" ) + "</center>"
}).appendTo( "#item-similars-als" );
});
});
}
</script>
</head>

<body>
<center>
<hr/>
<!-- Navigation Links
<a href="https://github.com/fluxcapacitor/pipeline/wiki/Architecture-Overview" target="_blank" id="architecture">Architecture</a> |
<a href="http://127.0.0.1:3123" target="_blank" id="zeppelin">Zeppelin</a> |
<a href="http://127.0.0.1:8754" target="_blank" id="jupyter">Jupyter/iPython</a> |
<a href="http://127.0.0.1/classify.html" target="_blank" id="tensorflow">TensorFlow</a> |
<a href="http://127.0.0.1:8754" target="_blank" id="sparkr">SparkR</a> |
<a href="http://127.0.0.1:6060" target="_blank" id="spark-admin">Spark Admin</a> |
<a href="http://127.0.0.1:9081" target="_blank" id="flink">Flink Admin</a> |
<a href="http://127.0.0.1:5601" target="_blank" id="kibana">Kibana</a> |
<a href="http://127.0.0.1:3060/admin" target="_blank" id="airflow">Airflow</a> |
<a href="http://127.0.0.1:7060" target="_blank" id="presto">Presto</a> |
<a href="http://127.0.0.1:6969/nifi" target="_blank" id="nifi">NiFi</a> |
<a href="/ganglia" target="_blank" id="ganglia">Ganglia</a> |
<a href="http://vectoross.io/demo/#/?host=127.0.0.1" target="_blank" id="vector">Vector</a>
-->
</center>
<hr/>
<center><img style="width: 320; height=400;" src="img/sparkafterdark.png"></center>

<script>
$("body").append("<h3><center>Welcome to Spark After Dark, Anonymous User ID <span style='color: blue'>" + myId + "</span>!</center></h3>")
</script>

<div id="items">
</div>

<hr/>
<!--
<h3>
<center>
Run Notebook to Generate <a href="http://127.0.0.1:3123/#/notebook/2AUUDPT56" target="_blank" id="topk-notebook">Non-Personalized Top 5 Items</a>
</center>
</h3>
-->
<h3>
<center>
Wait for Chris!! Then Click to Show <a href='javascript:loadTopK()'>Top 5 Items</a>
</center>
</h3>
<div id="topk">
</div>

<hr/>
<!--
<h3>
<center>
Run Notebook to Generate <a href="http://127.0.0.1:3123/#/notebook/2AUYFSKXN" target="_blank" id="als-notebook">Personalized User-to-Item Recommendations</a>
</center>
</h3>
-->
<h3>
<center>
Wait for Chris, then Click to Show <a href='javascript:loadRecommendationsALS()'>Recommendations</a>
</center>
</h3>
<div id="recommendations-als">
</div>

<hr/>
<!--
<h3>
<center>
Run Notebook to Generate <a href="http://127.0.0.1:3123/#/notebook/2BHDJNWAZ" target="_blank" id="user-to-user-similarity-notebook">User-to-User Similarity</a>
</center>
</h3>
<h3>
<center>
Change the User Id to Your User Id and Re-run Notebook
</center>
</h3>
-->
<hr/>
<!--
<h3>
<center>
Run Notebook to Generate <a href="http://127.0.0.1:3123/#/notebook/2BJQKR2G5" target="_blank" id="item-to-item-similarity-notebook">Item-to-Item Similarity</a>
</center>
</h3>

<h3>
<center>
Change the Item Id and Re-run Notebook
</center>
</h3>
-->
<hr/>
<!--
<h3>
<center>
Run Notebook to Generate <a href="http://127.0.0.1:3123/#/notebook/2BJB7GHA8" target="_blank" id="item-to-item-eigenface-similarity-notebook">Item-to-Item Eigenface Similarity</a>
</center>
</h3>
<h3>
<center>
Change the Item Id and Re-run Notebook
</center>
</h3>
--<
<hr/>
<!--
<script>
document.getElementById('zeppelin').href = window.location.protocol + "//" + window.location.hostname + ":3123";
document.getElementById('jupyter').href = window.location.protocol + "//" + window.location.hostname + ":8754";
document.getElementById('tensorflow').href = window.location.protocol + "//" + window.location.hostname + "/classify.html";
document.getElementById('sparkr').href = window.location.protocol + "//" + window.location.hostname + ":8754";
document.getElementById('spark-admin').href = window.location.protocol + "//" + window.location.hostname + ":6060";
document.getElementById('flink').href = window.location.protocol + "//" + window.location.hostname + ":9081";
document.getElementById('kibana').href = window.location.protocol + "//" + window.location.hostname + ":5601";
document.getElementById('airflow').href = window.location.protocol + "//" + window.location.hostname + ":3060/admin";
document.getElementById('presto').href = window.location.protocol + "//" + window.location.hostname + ":7060";
document.getElementById('nifi').href = window.location.protocol + "//" + window.location.hostname + ":6969/nifi";
document.getElementById('vector').href = "http://vectoross.io/demo/#/?host=" + window.location.hostname;
document.getElementById('topk-notebook').href = window.location.protocol + "//" + window.location.hostname + ":3123/#/notebook/2AUUDPT56";
document.getElementById('als-notebook').href = window.location.protocol + "//" + window.location.hostname + ":3123/#/notebook/2AUYFSKXN";
document.getElementById('user-to-user-similarity-notebook').href = window.location.protocol + "//" + window.location.hostname + ":3123/#/notebook/2BHDJNWAZ";
document.getElementById('item-to-item-similarity-notebook').href = window.location.protocol + "//" + window.location.hostname + ":3123/#/notebook/2BJQKR2G5";
document.getElementById('item-to-item-eigenface-similarity-notebook').href = window.location.protocol + "//" + window.location.hostname + ":3123/#/notebook/2BJB7GHA8";
</script>
-->
<script>
loadItems()
</script>
<!--
<script>
var startTime = new Date().getTime();
var interval = setInterval(function(){
if(new Date().getTime() - startTime > 20 * 60000){
clearInterval(interval);
return;
}
loadRealtimeRecommendationsALS();
loadRealtimeItemSimilarsALS();
}, 5000);
</script>
-->
<!--
<script>
alert("You're anonymous userID has changed to " + myId)
</script>
-->
</body>
</html>
12 changes: 12 additions & 0 deletions myapps/pmml/spark/1.6.1/lib/jpmml.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
from pyspark.mllib.common import _py2java

def toPMMLBytes(sc, data, pipelineModel):
javaData = _py2java(sc, data)
javaSchema = javaData.schema.__call__()

javaStages = sc._jvm.java.util.ArrayList()
for stage in pipelineModel.stages:
javaStages.add(stage._java_obj)
javaPipelineModel = sc._jvm.org.apache.spark.ml.PipelineModel(pipelineModel.uid, javaStages)

return sc._jvm.org.jpmml.sparkml.ConverterUtil.toPMMLByteArray(javaSchema, javaPipelineModel)
Binary file not shown.
Binary file modified myapps/spark/tensorframes/lib/tensorframes-assembly-0.2.4.jar
Binary file not shown.

0 comments on commit 91ae933

Please sign in to comment.