[doc] Switch parser to kramdown, normalize Headings

The switch to kramdown is necessary because I want to add tabs in the
documentation for code examples and Redcarpet does not allow markup
inside divs.

Before, some doc pages had "#" headings as toplevel headings while
others had "##" (which is the same as --- underlined headings). Now we
user level 2 everywhere. The page title is still a h1 heading.

docs/_config.yml

@@ -30,9 +30,7 @@ defaults:
       layout: docs
 
 highlighter: pygments
-markdown: redcarpet
-redcarpet:
-  # https://help.github.com/articles/github-flavored-markdown
-  extensions: ["no_intra_emphasis", "fenced_code_blocks", "autolink",
-               "tables", "with_toc_data", "strikethrough", "superscript",
-               "lax_spacing"]
+markdown: KramdownPygments
+
+kramdown:
+    toc_levels: 1..3

docs/_layouts/docs.html

@@ -8,6 +8,7 @@
 
     <link rel="stylesheet" href="//netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css">
     <link rel="stylesheet" href="css/syntax.css">
+    <link rel="stylesheet" href="css/codetabs.css">
 
     <!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries -->
     <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
@@ -46,9 +47,8 @@ <h1>Apache Flink {{ site.FLINK_VERSION_STABLE }} Documentation</h1>
 
                     <li>Programming Guides
                         <ul>
-                            <li><a href="java_api_guide.html">Java API</a></li>
-                            <li><a href="java_api_transformations.html">Java API Transformations</a></li>
-                            <li><a href="scala_api_guide.html">Scala API</a></li>
+                            <li><a href="programming_guide.html">Flink Programming Guide</a></li>
+                            <li><a href="dataset_transformations.html">DataSet Transformations</a></li>
                             <li><a href="hadoop_compatability.html">Hadoop Compatability</a></li>
                             <li><a href="iterations.html">Iterations</a></li>
                             <li><a href="spargel_guide.html">Spargel Graph API</a></li>
@@ -77,21 +77,21 @@ <h1>Apache Flink {{ site.FLINK_VERSION_STABLE }} Documentation</h1>
                             <li><a href="internal_overview.html">Overview</a></li>
                             <li><a href="internal_general_arch.html">General Architecture</a></li>
                             <li><a href="internal_add_operator.html">How-to: Adding a new Operator</a></li>
+                            <li><a href="internal_logging.html">How-to: Logging</a></li>
                         </ul>
                     </li>
                 </ul>
             </div>
             <div class="col-md-9">
                 <h1>{{ page.title }}</h1>
 
-                {{ page.content | tocify }}
-
                 {{ content }}
             </div>
         </div>
     </div>
 
     <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js"></script>
     <script src="//netdna.bootstrapcdn.com/bootstrap/3.1.1/js/bootstrap.min.js"></script>
+    <script src="js/codetabs.js"></script>
   </body>
 </html>

docs/_plugins/gh_link.rb

@@ -23,7 +23,7 @@ def render(context)
       name = name.to_s == '' ? file : name
       #refname = input[2].nil? ? file : input[2]
 
-      "<a href=#{config["FLINK_GITHUB_URL"]}/blob/#{gh_tag}/#{path}>#{name}</a>"
+      "[#{name}](#{config["FLINK_GITHUB_URL"]}/blob/#{gh_tag}/#{path})"
     end
   end
 end

docs/_plugins/kramdown_pygments.rb

@@ -0,0 +1,92 @@
+# We define the an additional option for the kramdown parser to look for
+module Kramdown
+  module Options
+      define(:kramdown_default_lang, Symbol, nil, <<EOF)
+Sets the default language for highlighting code blocks
+
+If no language is set for a code block, the default language is used
+instead. The value has to be one of the languages supported by pygments
+or nil if no default language should be used.
+
+Default: nil
+Used by: PygmentsHtml converter
+EOF
+  end
+end
+
+# This class is a plugin for kramdown, to make it use pygments instead of coderay
+# It has nothing to do with Jekyll, it is simply used by the custom converter below
+module Kramdown
+  module Converter
+    class PygmentsHtml < Html
+
+      begin
+        require 'pygments'
+      rescue LoadError
+        STDERR.puts 'You are missing a library required for syntax highlighting. Please run:'
+        STDERR.puts '  $ [sudo] gem install pygments'
+        raise FatalException.new("Missing dependency: Pygments")
+      end
+
+      def convert_codeblock(el, indent)
+        attr = el.attr.dup
+        lang = extract_code_language!(attr) || @options[:kramdown_default_lang]
+        code = pygmentize(el.value, lang)
+        code_attr = {}
+        code_attr['class'] = "language-#{lang}" if lang
+        "#{' '*indent}<div class=\"highlight\"><pre#{html_attributes(attr)}><code#{html_attributes(code_attr)}>#{code}</code></pre></div>\n"
+      end
+
+      def convert_codespan(el, indent)
+        attr = el.attr.dup
+        lang = extract_code_language!(attr) || @options[:kramdown_default_lang]
+        code = pygmentize(el.value, lang)
+        if lang
+          attr['class'] = "highlight"
+          if attr.has_key?('class')
+            attr['class'] += " language-#{lang}"
+          else
+            attr['class'] = "language-#{lang}"
+          end
+        end
+        "<code#{html_attributes(attr)}>#{code}</code>"
+      end
+
+      def pygmentize(code, lang)
+        if lang
+          Pygments.highlight(code,
+            :lexer => lang,
+            :options => { :startinline => true, :encoding => 'utf-8', :nowrap => true })
+        else
+          escape_html(code)
+        end
+      end
+    end
+  end
+end
+
+# This class is the actual custom Jekyll converter.
+class Jekyll::Converters::Markdown::KramdownPygments
+
+  def initialize(config)
+    require 'kramdown'
+    @config = config
+  rescue LoadError
+    STDERR.puts 'You are missing a library required for Markdown. Please run:'
+    STDERR.puts '  $ [sudo] gem install kramdown'
+    raise FatalException.new("Missing dependency: kramdown")
+  end
+
+  def convert(content)
+    html = Kramdown::Document.new(content, {
+        :auto_ids             => @config['kramdown']['auto_ids'],
+        :footnote_nr          => @config['kramdown']['footnote_nr'],
+        :entity_output        => @config['kramdown']['entity_output'],
+        :toc_levels           => @config['kramdown']['toc_levels'],
+        :smart_quotes         => @config['kramdown']['smart_quotes'],
+        :kramdown_default_lang => @config['kramdown']['default_lang'],
+        :input                => @config['kramdown']['input']
+    }).to_pygments_html
+    return html;
+  end
+end

docs/building.md

@@ -2,20 +2,20 @@
 title:  "Build Flink"
 ---
 
-
 In order to build Flink, you need the source code. Either download the source of a release or clone the git repository. In addition to that, you need Maven 3 and a JDK (Java Development Kit). Note that you can not build Flink with Oracle JDK 6 due to a unresolved bug in the Oracle Java compiler. It works well with OpenJDK 6 and all Java 7 and 8 compilers.
 
 To clone from git, enter:
-```
+
+~~~bash
 git clone {{ site.FLINK_GITHUB_URL }}
-```
+~~~
 
 The simplest way of building Flink is by running:
 
-```
+~~~bash
 cd incubator-flink
 mvn clean package -DskipTests
-```
+~~~
 
 This instructs Maven (`mvn`) to first remove all existing builds (`clean`) and then create a new Flink binary (`package`). The `-DskipTests` command prevents Maven from executing the unit tests.
 
@@ -27,36 +27,39 @@ This section covers building Flink for a specific Hadoop version. Most users do
 
 The problem is that Flink uses HDFS and YARN which are both dependencies from Apache Hadoop. There exist many different versions of Hadoop (from both the upstream project and the different Hadoop distributions). If a user is using a wrong combination of versions, exceptions like this one occur:
 
-```
+~~~bash
 ERROR: The job was not successfully submitted to the nephele job manager:
     org.apache.flink.nephele.executiongraph.GraphConversionException: Cannot compute input splits for TSV:
     java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException:
     Protocol message contained an invalid tag (zero).; Host Details :
-```
+~~~
 
 There are two main versions of Hadoop that we need to differentiate:
 - Hadoop 1, with all versions starting with zero or one, like 0.20, 0.23 or 1.2.1.
 - Hadoop 2, with all versions starting with 2, like 2.2.0.
 The main differentiation between Hadoop 1 and Hadoop 2 is the availability of Hadoop YARN (Hadoops cluster resource manager).
 
 **To build Flink for Hadoop 2**, issue the following command:
-```
+
+~~~bash
 mvn clean package -DskipTests -Dhadoop.profile=2
-```
+~~~
 
 The `-Dhadoop.profile=2` flag instructs Maven to build Flink with YARN support and the Hadoop 2 HDFS client.
 
 Usually, this flag is sufficient for full support of Flink for Hadoop 2-versions.
 However, you can also **specify a specific Hadoop version to build against**:
-```
+
+~~~bash
 mvn clean package -DskipTests -Dhadoop.profile=2 -Dhadoop.version=2.4.1
-```
+~~~
 
 
 **To build Flink against a vendor specific Hadoop version**, issue the following command:
-```
+
+~~~bash
 mvn clean package -DskipTests -Pvendor-repos -Dhadoop.profile=2 -Dhadoop.version=2.2.0-cdh5.0.0-beta-2
-```
+~~~
 
 The `-Pvendor-repos` activates a Maven [build profile](http://maven.apache.org/guides/introduction/introduction-to-profiles.html) that includes the repositories of popular Hadoop vendors such as Cloudera, Hortonworks, or MapR.
 
@@ -66,11 +69,11 @@ Maven will automatically build Flink with its YARN client if the `-Dhadoop.profi
 
 So if you are building Flink for Hadoop `2.0.0-alpha`, use the following command:
 
-```bash
+~~~bash
 -P\!include-yarn -Dhadoop.profile=2 -Dhadoop.version=2.0.0-alpha
-```
+~~~
 
-### Background
+## Background
 
 The builds with Maven are controlled by [properties](http://maven.apache.org/pom.html#Properties) and <a href="http://maven.apache.org/guides/introduction/introduction-to-profiles.html">build profiles</a>.
 There are two profiles, one for hadoop1 and one for hadoop2. When the hadoop2 profile is enabled, the system will also build the YARN client.

docs/cli.md

@@ -2,6 +2,10 @@
 title:  "Command-Line Interface"
 ---
 
+* This will be replaced by the TOC
+{:toc}
+
+
 Flink provides a command-line interface to run programs that are packaged
 as JAR files, and control their execution.  The command line interface is part
 of any Flink setup, available in local single node setups and in
@@ -20,7 +24,7 @@ The command line can be used to
 - provide information about a job, and
 - list running and waiting jobs.
 
-# Examples
+## Examples
 
 -   Run example program with no arguments.
 
@@ -61,11 +65,11 @@ The command line can be used to
 
         ./bin/flink cancel -i <jobID>
 
-# Usage
+## Usage
 
 The command line syntax is as follows:
 
-```
+~~~
 ./flink <ACTION> [OPTIONS] [ARGUMENTS]
 
 General options:
@@ -126,4 +130,4 @@ Action "cancel" cancels a submitted Flink program.
      -i,--jobid <jobID>             JobID of program to cancel
      
      -m,--jobmanager <host:port>    Option to connect to a different Flink master (JobManager).
-```
+~~~