Dockerfile and CI (dmlc#74)

* Test CPP branch CI (dmlc#2) * Fix batching node-only graphs (dmlc#62) * fixing batching with graphs with no edges * oops forgot test * fix readme * Docker and Jenkins (dmlc#1) * docker ci cpu * install python packages * docker ci gpu * add readme * use dgl cpu image * run command in container as root * use python3 * fix test case * remove nose from docker file * docker folder readme * parallelize cpu and gpu * top level stages * comment out python2 related installation * fix * remove igraph * building for cpp * change building order * export env in test stage * withEnv * run docker container as root * fix test cases * fix test cases * minor * remove old build
ChenAris · Oct 7, 2018 · cdf7334 · cdf7334
1 parent 72f6345
commit cdf7334
Show file tree

Hide file tree

Showing 14 changed files with 223 additions and 73 deletions.
diff --git a/Jenkinsfile b/Jenkinsfile
@@ -1,33 +1,95 @@
 pipeline {
-    agent {
-        docker {
-            image 'pytorch/pytorch'
-        }
-    }
+    agent none
     stages {
-        stage('SETUP') {
-            steps {
-                sh 'easy_install nose'
-                sh 'apt-get update && apt-get install -y libxml2-dev'
-            }
-        }
-        stage('BUILD') {
-            steps {
-                dir('python') {
-                    sh 'python setup.py install'
+        stage('Build and Test') {
+            parallel {
+                stage('CPU') {
+                    agent {
+                        docker {
+                            image 'lingfanyu/dgl-cpu'
+                            args '-u root'
+                        }
+                    }
+                    stages {
+                        stage('SETUP') {
+                            steps {
+                                sh 'easy_install nose'
+                                sh 'git submodule init'
+                                sh 'git submodule update'
+                            }
+                        }
+                        stage('BUILD') {
+                            steps {
+                                sh 'if [ -d build ]; then rm -rf build; fi; mkdir build'
+                                dir('python') {
+                                    sh 'python3 setup.py install'
+                                }
+                                dir ('build') {
+                                    sh 'cmake ..'
+                                    sh 'make -j$(nproc)'
+                                }
+                            }
+                        }
+                        stage('TEST') {
+                            steps {
+                                withEnv(["DGL_LIBRARY_PATH=${env.WORKSPACE}/build"]) {
+                                    sh 'echo $DGL_LIBRARY_PATH'
+                                    sh 'nosetests tests -v --with-xunit'
+                                    sh 'nosetests tests/pytorch -v --with-xunit'
+                                }
+                            }
+                        }
+                    }
+                    post {
+                        always {
+                            junit '*.xml'
+                        }
+                    }
+                }
+                stage('GPU') {
+                    agent {
+                        docker {
+                            image 'lingfanyu/dgl-gpu'
+                            args '--runtime nvidia -u root'
+                        }
+                    }
+                    stages {
+                        stage('SETUP') {
+                            steps {
+                                sh 'easy_install nose'
+                                sh 'git submodule init'
+                                sh 'git submodule update'
+                            }
+                        }
+                        stage('BUILD') {
+                            steps {
+                                sh 'if [ -d build ]; then rm -rf build; fi; mkdir build'
+                                dir('python') {
+                                    sh 'python3 setup.py install'
+                                }
+                                dir ('build') {
+                                    sh 'cmake ..'
+                                    sh 'make -j$(nproc)'
+                                }
+                            }
+                        }
+                        stage('TEST') {
+                            steps {
+                                withEnv(["DGL_LIBRARY_PATH=${env.WORKSPACE}/build"]) {
+                                    sh 'echo $DGL_LIBRARY_PATH'
+                                    sh 'nosetests tests -v --with-xunit'
+                                    sh 'nosetests tests/pytorch -v --with-xunit'
+                                }
+                            }
+                        }
+                    }
+                    post {
+                        always {
+                            junit '*.xml'
+                        }
+                    }
                 }
             }
         }
-        stage('TEST') {
-            steps {
-                sh 'nosetests tests -v --with-xunit'
-                sh 'nosetests tests/pytorch -v --with-xunit'
-            }
-        }
-    }
-    post {
-        always {
-            junit '*.xml'
-        }
     }
 }
diff --git a/README.md b/README.md
@@ -12,7 +12,7 @@ Show below, there are three sets of APIs for different models.
 
 ## For Model developers
 - Always choose the API at the *highest* possible level.
-- Refer to [the default modules](examples/pytorch/util.py) to see how to register message and node update functions as well as readout functions; note how you can control sharing of parameters by adding a counter.
+- Refer to the [GCN example](examples/pytorch/gcn/gcn_batch.py) to see how to register message and node update functions;
 
 ## How to build (the `cpp` branch)
 

diff --git a/docker/Dockerfile.ci_cpu b/docker/Dockerfile.ci_cpu
@@ -0,0 +1,14 @@
+# CI docker CPU env
+# Adapted from github.com/dmlc/tvm/docker/Dockerfile.ci_cpu
+FROM ubuntu:16.04
+
+RUN apt-get update --fix-missing
+
+COPY install/ubuntu_install_core.sh /install/ubuntu_install_core.sh
+RUN bash /install/ubuntu_install_core.sh
+
+COPY install/ubuntu_install_python.sh /install/ubuntu_install_python.sh
+RUN bash /install/ubuntu_install_python.sh
+
+COPY install/ubuntu_install_python_package.sh /install/ubuntu_install_python_package.sh
+RUN bash /install/ubuntu_install_python_package.sh
diff --git a/docker/Dockerfile.ci_gpu b/docker/Dockerfile.ci_gpu
@@ -0,0 +1,22 @@
+# CI docker GPU env
+FROM nvidia/cuda:9.0-cudnn7-devel
+
+# Base scripts
+RUN apt-get update --fix-missing
+
+COPY install/ubuntu_install_core.sh /install/ubuntu_install_core.sh
+RUN bash /install/ubuntu_install_core.sh
+
+COPY install/ubuntu_install_python.sh /install/ubuntu_install_python.sh
+RUN bash /install/ubuntu_install_python.sh
+
+COPY install/ubuntu_install_python_package.sh /install/ubuntu_install_python_package.sh
+RUN bash /install/ubuntu_install_python_package.sh
+
+# Environment variables
+ENV PATH=/usr/local/nvidia/bin:${PATH}
+ENV PATH=/usr/local/cuda/bin:${PATH}
+ENV CPLUS_INCLUDE_PATH=/usr/local/cuda/include:${CPLUS_INCLUDE_PATH}
+ENV C_INCLUDE_PATH=/usr/local/cuda/include:${C_INCLUDE_PATH}
+ENV LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/nvidia/lib64:${LIBRARY_PATH}
+ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/nvidia/lib64:${LD_LIBRARY_PATH}
diff --git a/docker/README.md b/docker/README.md
@@ -0,0 +1,7 @@
+## Build docker image for CI
+
+### CPU image
+docker build -t dgl-cpu -f Dockerfile.ci_cpu .
+
+### GPU image
+docker build -t dgl-gpu -f Dockerfile.ci_gpu .
diff --git a/docker/install/ubuntu_install_core.sh b/docker/install/ubuntu_install_core.sh
@@ -0,0 +1,3 @@
+# install libraries for building c++ core on ubuntu
+apt update && apt install -y --no-install-recommends --force-yes \
+        apt-utils git build-essential make cmake wget unzip sudo libz-dev libxml2-dev
diff --git a/docker/install/ubuntu_install_python.sh b/docker/install/ubuntu_install_python.sh
@@ -0,0 +1,13 @@
+# install python and pip, don't modify this, modify install_python_package.sh
+# apt-get update && apt-get install -y python-dev python-pip
+
+# python 3.6
+apt-get update && yes | apt-get install software-properties-common
+add-apt-repository ppa:jonathonf/python-3.6
+apt-get update && apt-get install -y python3.6 python3.6-dev
+rm -f /usr/bin/python3 && ln -s /usr/bin/python3.6 /usr/bin/python3
+
+# Install pip
+cd /tmp && wget https://bootstrap.pypa.io/get-pip.py
+# python2 get-pip.py
+python3.6 get-pip.py
diff --git a/docker/install/ubuntu_install_python_package.sh b/docker/install/ubuntu_install_python_package.sh
@@ -0,0 +1,7 @@
+# install libraries for python package on ubuntu
+# pip2 install pylint numpy cython scipy nltk requests[security]
+pip3 install pylint numpy cython scipy nltk requests[security]
+
+# install DL Framework
+# pip2 install torch torchvision
+pip3 install torch torchvision
diff --git a/python/dgl/batched_graph.py b/python/dgl/batched_graph.py
@@ -174,7 +174,7 @@ def unbatch(graph):
     return [DGLGraph(graph_data=pttns[i],
                      node_frame=node_frames[i],
                      edge_frame=edge_frames[i]) for i in range(bsize)]
-    
+
 def batch(graph_list, node_attrs=ALL, edge_attrs=ALL):
     """Batch a list of DGLGraphs into one single graph.
 

diff --git a/python/dgl/graph.py b/python/dgl/graph.py
@@ -54,7 +54,7 @@ def __init__(self,
 
     def add_nodes(self, num, reprs=None):
         """Add nodes.
-        
+
         Parameters
         ----------
         num : int
@@ -69,7 +69,7 @@ def add_nodes(self, num, reprs=None):
 
     def add_edge(self, u, v, reprs=None):
         """Add one edge.
-        
+
         Parameters
         ----------
         u : int
@@ -85,7 +85,7 @@ def add_edge(self, u, v, reprs=None):
 
     def add_edges(self, u, v, reprs=None):
         """Add many edges.
-        
+
         Parameters
         ----------
         u : list, tensor
@@ -153,7 +153,7 @@ def has_node(self, vid):
             True if the node exists
         """
         return self.has_node(vid)
-    
+
     def __contains__(self, vid):
         """Same as has_node."""
         return self.has_node(vid)
@@ -290,7 +290,7 @@ def in_edges(self, v):
         ----------
         v : int, list, tensor
             The node(s).
-        
+
         Returns
         -------
         tensor
@@ -311,7 +311,7 @@ def out_edges(self, v):
         ----------
         v : int, list, tensor
             The node(s).
-        
+
         Returns
         -------
         tensor
@@ -332,7 +332,7 @@ def edges(self, sorted=False):
         ----------
         sorted : bool
             True if the returned edges are sorted by their src and dst ids.
-        
+
         Returns
         -------
         tensor
@@ -431,7 +431,7 @@ def from_networkx(self, nx_graph, node_attrs=None, edge_attrs=None):
 
         If 'id' edge attribute exists, the edge will be added follows
         the edge id order. Otherwise, order is undefined.
-        
+
         Parameters
         ----------
         nx_graph : networkx.DiGraph
@@ -1153,12 +1153,12 @@ def propagate(self,
         kwargs : keyword arguments, optional
             Arguments for pre-defined iterators.
         """
-        if isinstance(iterator, str):
+        if isinstance(traverser, str):
             # TODO Call pre-defined routine to unroll the computation.
             raise RuntimeError('Not implemented.')
         else:
             # NOTE: the iteration can return multiple edges at each step.
-            for u, v in iterator:
+            for u, v in traverser:
                 self.send_and_recv(u, v,
                         message_func, reduce_func, apply_node_func)