doc: show changelog. for that, support page(source='something.rst')

Allow page target to have .rst source.

rm markdown goal's no-longer-so-useful --extensions option. (Would be happy to leave it in but doc'd as 'unused' instead or somesuch)

bring CHANGELOG.rst into the Pants doc site.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/46344769

Bugs closed: 612

Reviewed at https://rbcommons.com/s/twitter/r/1575/

examples/src/java/com/pants/examples/page.md

@@ -1,7 +1,7 @@
 README Files and Markdown
 =========================
 
-You can write program documentation in the popular Markdown format;
+You can write program documentation in the popular Markdown or ReST format;
 Pants eases publishing your docs to places where your users can read it.
 E.g., that `README.md` file in your source code is handy for editing;
 but you might want to generate a web page from that so folks can decide
@@ -10,12 +10,7 @@ whether they want to look at your source code.
 Markdown to HTML
 ----------------
 
-Pants uses the Python `Markdown` module; thus, in addition to the usual
-Gruber `Markdown` syntax, there are [other
-features](http://pythonhosted.org/Markdown/) Pants uses Python
-Markdown's `codehilite`, `extra`, `tables`, and `toc` extensions.
-
-To tell Pants about your Markdown file, use a
+To tell Pants about your Markdown or ReST file, use a
 <a pantsref="bdict_page">`page`</a>
 target in a `BUILD` file as in this excerpt from
 [examples/src/java/com/pants/examples/hello/main/BUILD](https://github.com/pantsbuild/pants/blob/master/examples/src/java/com/pants/examples/hello/main/BUILD):
@@ -32,8 +27,15 @@ your browser,
 
 Pants generates the HTML files in the `dist/markdown/` directory tree.
 
-Link to Another `page`
-----------------------
+Markdown Syntax
+---------------
+
+Pants uses the Python `Markdown` module; thus, in addition to the usual
+Gruber `Markdown` syntax, there are [other
+features](http://pythonhosted.org/Markdown/) Pants uses Python
+Markdown's `codehilite`, `extra`, `tables`, and `toc` extensions.
+
+### Link to Another `page`
 
 One `page` can link to another. Regular Markdown-link syntax works for
 regular links; but if you use `page`s to generate both `.html` files and
@@ -69,8 +71,7 @@ therein:
 
 Pants replaces the `pants('path/to:dest')` with the appropriate link.
 
-Include a File Snippet
-----------------------
+### Include a File Snippet
 
 Sometimes the best way to explain `HelloWorld.java` is to show an
 excerpt from `HelloWorld.java`. You can use the `!inc` markdown to do
@@ -109,6 +110,17 @@ end-at=*substring*<br>
 When excerpting the file to include, stop at a line containing
 *substring*.
 
+ReStructedText Syntax
+---------------------
+
+Pants can generate web content from
+[docutils reStructuredText](http://docutils.sourceforge.net/rst.html)-formatted text.
+
+To tell Pants that your `page` target's source is in reStructuredText format, you can either
+
+* give the `source` file an `.rst` file extension, or
+* pass `format='rst'` to the `page` target.
+
 Publishing
 ----------
 

src/docs/index.md

@@ -48,13 +48,14 @@ Using Pants With...
 News
 ----
 
++ [[Release History (Changelog)|pants('src/python/pants:changelog')]]
 + [[2014-09-16 Announcement|pants('src/docs:announce_201409')]]
   "Hello Pants Build"
 
 Advanced Documentation
 ----------------------
 
-+ [[Set up your Source Tree for Pants|pants('src/docs:announce_201409')]]
++ [[Set up your Source Tree for Pants|pants('src/docs:setup_repo')]]
 + [[Installing Pants|pants('src/docs:install')]]
 
 Pants Reference Documentation

src/python/pants/BUILD

@@ -112,3 +112,8 @@ python_library(
   name = 'version',
   sources = ['version.py'],
 )
+
+page(
+  name='changelog',
+  source='CHANGELOG.rst'
+)

src/python/pants/backend/core/targets/doc.py

@@ -74,12 +74,15 @@ def __init__(self,
                address=None,
                payload=None,
                source=None,
+               format=None,
                links=None,
                resources=None,
                provides=None,
                **kwargs):
     """
-    :param source: Source of the page in markdown format.
+    :param source: Path to page source file.
+    :param format: Page's format, ``md`` or ``rst``. By default, Pants infers from ``source`` file
+       extension: ``.rst`` is ReStructured Text; anything else is Markdown.
     :param links: Other ``page`` targets that this `page` links to.
     :type links: List of target specs
     :param provides: Optional "Addresses" at which this page is published.
@@ -88,9 +91,15 @@ def __init__(self,
     :param resources: An optional list of Resources objects.
     """
     payload = payload or Payload()
+    if not format:
+      if source and source.lower().endswith('.rst'):
+        format = 'rst'
+      else:
+        format = 'md'
     payload.add_fields({
       'sources': SourcesField(sources=[source],
                               sources_rel_path=address.spec_path),
+      'format': PrimitiveField(format),
       'links': PrimitiveField(links or []),
       'provides': self.ProvidesTupleField(provides or []),
     })
@@ -127,3 +136,8 @@ def provides(self):
     list.
     """
     return self.payload.provides
+
+  @property
+  def format(self):
+    """Returns this page's format, 'md' (Markdown) or 'rst' (ReStructured Text)."""
+    return self.payload.format

src/python/pants/backend/core/tasks/BUILD

@@ -178,6 +178,7 @@ python_library(
   sources = ['markdown_to_html.py'],
   resources = globs('templates/markdown/*.mustache'),
   dependencies = [
+    '3rdparty/python:docutils',
     '3rdparty/python:Markdown',
     '3rdparty/python:Pygments',
     ':common',

src/python/pants/backend/core/tasks/markdown_to_html.py

@@ -9,6 +9,7 @@
 import os
 import re
 
+from docutils.core import publish_parts
 import markdown
 from pkg_resources import resource_string
 from pygments import highlight
@@ -183,7 +184,23 @@ def page_to_html_path(page):
   return os.path.splitext(source_path)[0] + ".html"
 
 
+def rst_to_html(in_rst):
+  """Returns HTML rendering of an RST fragment.
+
+  :param in_rst: rst-formatted string
+  """
+  if not in_rst:
+    return ''
+  pp = publish_parts(in_rst, writer_name='html')
+  return_value = ''
+  if 'title' in pp and pp['title']:
+    return_value += '<title>{0}</title>\n<p style="font: 200% bold">{0}</p>\n'.format(pp['title'])
+  return_value += pp['body'].strip()
+  return return_value
+
+
 class MarkdownToHtml(Task):
+
   @classmethod
   def register_options(cls, register):
     register('--code-style', choices=list(get_all_styles()), default='friendly',
@@ -192,9 +209,6 @@ def register_options(cls, register):
              help='Open the generated documents in a browser.')
     register('--fragment', action='store_true',
              help='Generate a fragment of html to embed in a page.')
-    register('--extension', action='append', default=['.md', '.markdown'],
-             help='Process files with these extensions (as well as the standard extensions).')
-
 
   @classmethod
   def product_types(cls):
@@ -204,7 +218,6 @@ def __init__(self, *args, **kwargs):
     super(MarkdownToHtml, self).__init__(*args, **kwargs)
     self._templates_dir = os.path.join('templates', 'markdown')
     self.open = self.get_options().open
-    self.extensions = set(self.get_options().extension)
     self.fragment = self.get_options().fragment
     self.code_style = self.get_options().code_style
 
@@ -237,53 +250,58 @@ def is_page(target):
     wikigenmap = self.context.products.get('wiki_html')
     show = []
     for page in self.context.targets(is_page):
-      _, ext = os.path.splitext(page.source)
-      if ext in self.extensions:
-        def process_page(key, outdir, url_builder, config, genmap, fragment=False):
-          html_path = self.process(
+      def process_page(key, outdir, url_builder, config, genmap, fragment=False):
+        if page.format == 'rst':
+          html_path = self.process_rst(
+            os.path.join(outdir, page_to_html_path(page)),
+            os.path.join(page.payload.sources.rel_path, page.source),
+            self.fragment or fragment,
+          )
+        else:
+          html_path = self.process_md(
             os.path.join(outdir, page_to_html_path(page)),
             os.path.join(page.payload.sources.rel_path, page.source),
             self.fragment or fragment,
             url_builder,
             config,
             css=css
           )
-          self.context.log.info('Processed %s to %s' % (page.source, html_path))
-          relpath = os.path.relpath(html_path, outdir)
-          genmap.add(key, outdir, [relpath])
-          return html_path
-
-        def url_builder(linked_page, config=None):
-          dest = page_to_html_path(linked_page)
-          src_dir = os.path.dirname(page_to_html_path(page))
-          return linked_page.name, os.path.relpath(dest, src_dir)
-
-        page_path = os.path.join(outdir, 'html')
-        html = process_page(page, page_path, url_builder, lambda p: None, plaingenmap)
-        if css and not self.fragment:
-          plaingenmap.add(page, self.workdir, list(css_path))
-        if self.open and page in roots:
-          show.append(html)
-
-        if page.provides:
-          for wiki in page.provides:
-            def get_config(page):
-              # Take the first provided WikiArtifact. If a page is published to multiple places, it's
-              # undefined what the "proper" one is to link to. So we just take whatever is "first".
-              for wiki_artifact in page.payload.provides:
-                return wiki_artifact.config
-            basedir = os.path.join(self.workdir, str(hash(wiki)))
-            process_page((wiki, page), basedir, wiki.wiki.url_builder, get_config,
+        self.context.log.info('Processed %s to %s' % (page.source, html_path))
+        relpath = os.path.relpath(html_path, outdir)
+        genmap.add(key, outdir, [relpath])
+        return html_path
+
+      def url_builder(linked_page, config=None):
+        dest = page_to_html_path(linked_page)
+        src_dir = os.path.dirname(page_to_html_path(page))
+        return linked_page.name, os.path.relpath(dest, src_dir)
+
+      page_path = os.path.join(outdir, 'html')
+      html = process_page(page, page_path, url_builder, lambda p: None, plaingenmap)
+      if css and not self.fragment:
+        plaingenmap.add(page, self.workdir, list(css_path))
+      if self.open and page in roots:
+        show.append(html)
+
+      if page.provides:
+        for wiki in page.provides:
+          def get_config(page):
+            # Take the first provided WikiArtifact. If a page is published to multiple places, it's
+            # undefined what the "proper" one is to link to. So we just take whatever is "first".
+            for wiki_artifact in page.payload.provides:
+              return wiki_artifact.config
+          basedir = os.path.join(self.workdir, str(hash(wiki)))
+          process_page((wiki, page), basedir, wiki.wiki.url_builder, get_config,
                          wikigenmap, fragment=True)
 
     if show:
       binary_util.ui_open(*show)
 
   PANTS_LINK = re.compile(r'''pants\(['"]([^)]+)['"]\)(#.*)?''')
 
-  def process(self, output_path, source, fragmented, url_builder, get_config, css=None):
+  def process_md(self, output_path, source, fragmented, url_builder, get_config, css=None):
     def parse_url(spec):
-      match = MarkdownToHtml.PANTS_LINK.match(spec)
+      match = self.PANTS_LINK.match(spec)
       if match:
         address = SyntheticAddress.parse(match.group(1), relative_to=get_buildroot())
         page = self.context.build_graph.get_target(address)
@@ -310,9 +328,9 @@ def build_url(label):
     safe_mkdir(os.path.dirname(output_path))
     with codecs.open(output_path, 'w', 'utf-8') as output:
       source_path = os.path.join(get_buildroot(), source)
-      with codecs.open(source_path, 'r', 'utf-8') as input:
+      with codecs.open(source_path, 'r', 'utf-8') as source_stream:
         md_html = markdown.markdown(
-          input.read(),
+          source_stream.read(),
           extensions=['codehilite(guess_lang=False)',
                       'extra',
                       'tables',
@@ -325,15 +343,31 @@ def build_url(label):
           template = resource_string(__name__,
                                      os.path.join(self._templates_dir,
                                                   'fragment.mustache'))
-          generator = Generator(template,
-                                style_css=style_css,
-                                md_html=md_html)
+          generator = Generator(template, style_css=style_css, md_html=md_html)
           generator.write(output)
         else:
           style_link = os.path.relpath(css, os.path.dirname(output_path))
+          template = resource_string(__name__, os.path.join(self._templates_dir, 'page.mustache'))
+          generator = Generator(template, style_link=style_link, md_html=md_html)
+          generator.write(output)
+        return output.name
+
+  def process_rst(self, output_path, source, fragmented):
+    safe_mkdir(os.path.dirname(output_path))
+    with codecs.open(output_path, 'w', 'utf-8') as output:
+      source_path = os.path.join(get_buildroot(), source)
+      with codecs.open(source_path, 'r', 'utf-8') as source_stream:
+        rst_html = rst_to_html(source_stream.read())
+        if fragmented:
+          template = resource_string(__name__,
+                                     os.path.join(self._templates_dir,
+                                                  'fragment.mustache'))
+          generator = Generator(template,
+                                md_html=rst_html)
+          generator.write(output)
+        else:
           template = resource_string(__name__, os.path.join(self._templates_dir, 'page.mustache'))
           generator = Generator(template,
-                                style_link=style_link,
-                                md_html=md_html)
+                                md_html=rst_html)
           generator.write(output)
         return output.name

src/python/pants/docs/docsite.json

@@ -6,6 +6,7 @@
     "announce_201409": "dist/markdown/html/src/docs/announce_201409.html",
     "build_dictionary": "dist/builddict/build_dictionary.html",
     "build_files": "dist/markdown/html/src/docs/build_files.html",
+    "changelog": "dist/markdown/html/src/python/pants/CHANGELOG.html",
     "credits": "dist/markdown/html/src/python/pants/docs/credits.html",
     "dev": "dist/markdown/html/src/python/pants/docs/readme.html",
     "dev_tasks": "dist/markdown/html/src/python/pants/docs/dev_tasks.html",
@@ -90,6 +91,7 @@
         { "page": "setup_repo" },
         { "page": "build_dictionary" },
         { "page": "goals_reference" },
+        { "page": "changelog" },
         { "page": "dev",
           "children": [
             { "page": "howto_contribute" },

testprojects/src/java/com/pants/testproject/page/BUILD

@@ -15,3 +15,6 @@ page(name="circular",
        ':readme',
      ]
 )
+
+page(name="senserst",
+     source="sense.rst")

testprojects/src/java/com/pants/testproject/page/sense.rst

@@ -0,0 +1,27 @@
+Sense and Sensibility
+=====================
+
+*With some gratuitous markup*
+
+No sooner was her answer dispatched, than Mrs. Dashwood indulged herself in the pleasure of
+announcing to her son-in-law and his wife that she was provided with a house, and should
+incommode them no longer than till every thing were ready for her ``inhabiting`` it.
+
+They Heard Her With Surprise
+----------------------------
+
+Mrs. John Dashwood said nothing; but her husband **civilly** hoped
+that she would not be settled far from
+`Norland <http://www.calderdale.gov.uk/wtw/search/controlservlet?PageId=Detail&DocId=102275>`_.
+
+She had great satisfaction
+--------------------------
+
+...in replying that she was going into Devonshire.--Edward
+turned hastily towards her, on hearing this, and, in a voice of surprise
+and concern, which required no explanation to her, repeated,
+
+    Devonshire! Are you, indeed, going there? So far from hence! And to
+    what part of it?
+
+She explained the situation.