docs: Merge front-end-build-process with html-css.

This merges the "Static asset pipeline" article as a new section
at the bottom of the "HTML and CSS" article.

docs/overview/architecture-overview.md

@@ -115,7 +115,7 @@ For more details on the frontend, see our documentation on
 [translation](../translating/translating.md),
 [templates](../subsystems/html-css.html#html-templates),
 [directory structure](../overview/directory-structure.md), and
-[the static asset pipeline](../subsystems/front-end-build-process.md).
+[the static asset pipeline](../subsystems/html-css.html#static-asset-pipeline).
 
 [Jinja2]: http://jinja.pocoo.org/
 [Handlebars]: http://handlebarsjs.com/

docs/subsystems/dependencies.md

@@ -248,8 +248,8 @@ reasoning here.
   dependencies in the `yarn.lock` file; `yarn install` updates the
   `yarn.lock` files.
 * `tools/update-prod-static`.  This process is discussed in detail in
-  the [static asset pipeline](../subsystems/front-end-build-process.md) article,
-  but we don't use the `node_modules` directories directly in
+  the [static asset pipeline](../subsystems/html-css.html#static-asset-pipeline)
+  article, but we don't use the `node_modules` directories directly in
   production.  Instead, static assets are compiled using our static
   asset pipeline and it is the compiled assets that are served
   directly to users.  As a result, we don't ship the `node_modules`

docs/subsystems/html-css.md

@@ -134,6 +134,138 @@ whenever a template is changed.
 All user-facing strings (excluding pages only visible to sysadmins or
 developers) should be tagged for [translation][].
 
+## Static asset pipeline
+
+This section documents additional information that may be useful when
+developing new features for Zulip that require front-end changes,
+especially those that involve adding new files. For a more general
+overview, see the [new feature tutorial](../tutorials/new-feature-tutorial.md).
+
+Our [dependencies documentation](../subsystems/dependencies.md) has useful
+relevant background as well.
+
+### Primary build process
+
+Most of the existing JS in Zulip is written in
+[IIFE](https://www.google.com/#q=iife)-wrapped modules, one per file
+in the `static/js` directory. We will over time migrate these to
+Typescript modules.  Stylesheets are written in the Sass extension of
+CSS (with the scss syntax), they are converted from plain CSS and we
+have yet to take full advantage of the features Sass offers.  We use
+Webpack to transpile and build JS and CSS bundles that the browser can
+understand, one for each entry points specifed in
+`tools/webpack.assets.json`; source maps are generated in the process
+for better debugging experience.
+
+In development mode, bundles are built and served on the fly using
+webpack-dev-server with live reloading. In production mode (and when creating a
+release tarball using `tools/build-release-tarball`), the
+`tools/update-prod-static` tool (called by both `tools/build-release-tarball`
+and `tools/upgrade-zulip-from-git`) is responsible for orchestrating the
+webpack build, JS minification and a host of other steps for getting the assets
+ready for deployment.
+
+You can trace which source files are included in which HTML templates
+by comparing the `render_bundle` calls in the HTML templates under
+`templates/` with the bundles declared in `tools/webpack.assets.json`.
+
+### Adding static files
+
+To add a static file to the app (JavaScript, TypeScript, CSS/Sass, images, etc),
+first add it to the appropriate place under `static/`.
+
+- Third-party packages from the NPM repository should be added to
+  `package.json` for management by yarn, this allows them to be upgraded easily
+  and not bloat our codebase. Run `./tools/provision` for yarn to install the
+  new packages and update its lock file. You should also update
+  `PROVISION_VERSION` in `version.py` in the same commit. When adding modules
+  to `package.json`, please pin specific versions of them (don't using carets
+  `^`, tildes `~`, etc). We prefer fixed versions so that when the upstream
+  providers release new versions with incompatible APIs, it can't break Zulip.
+  We update those versions periodically to ensure we're running a recent
+  version of third-party libraries.
+- Third-party files that we have patched should all go in
+  `static/third/`. Tag the commit with "[third]" when adding or
+  modifying a third-party package.  Our goal is to the extent possible
+  to eliminate patched third-party code from the project.
+- Our own JavaScript and TypeScript files live under `static/js`.  Ideally,
+  new modules should be written in TypeScript (details on this policy below).
+- CSS/Sass files lives under `static/styles`.
+- Portico JavaScript ("portico" means for logged-out pages) lives under
+  `static/js/portico`.
+- Custom SVG graphics living under `static/assets/icons` are compiled into
+  custom icon webfonts by webfont-loader according to the
+  `static/assets/icons/template.hbs` template.
+
+For your asset to be included in a development/production bundle, it
+needs to be accessible from one of the entry points defined in
+`tools/webpack.assets.json`.
+
+* If you plan to only use the file within the app proper, and not on the login
+  page or other standalone pages, put it in the `app` bundle by importing it
+  in `static/js/bundles/app.js`.
+* If it needs to be available both in the app and all
+  logged-out/portico pages, import it to
+  `static/js/bundles/common.js` which itself is imported to the
+  `app` and `common` bundles.
+* If it's just used on a single standalone page (e.g. `/stats`),
+  create a new entry point in `tools/webpack.assets.json`. Use the
+  `bundle` macro (defined in `templates/zerver/base.html`) in the
+  relevant Jinja2 template to inject the compiled JS and CSS.
+
+If you want to test minified files in development, look for the
+`DEBUG =` line in `zproject/settings.py` and set it to `False`.
+
+### How it works in production
+
+A few useful notes are:
+* Zulip installs static assets in production in
+`/home/zulip/prod-static`.  When a new version is deployed, before the
+server is restarted, files are copied into that directory.
+* We use the VFL (Versioned File Layout) strategy, where each file in
+  the codebase (e.g. `favicon.ico`) gets a new name
+  (e.g. `favicon.c55d45ae8c58.ico`) that contains a hash in it.  Each
+  deployment, has a manifest file
+  (e.g. `/home/zulip/deployments/current/staticfiles.json`) that maps
+  codebase filenames to serving filenames for that deployment.  The
+  benefit of this VFL approach is that all the static files for past
+  deployments can coexist, which in turn eliminates most classes of
+  race condition bugs where browser windows opened just before a
+  deployment can't find their static assets.  It also is necessary for
+  any incremental rollout strategy where different clients get
+  different versions of the site.
+* Some paths for files (e.g. emoji) are stored in the
+  `rendered_content` of past messages, and thus cannot be removed
+  without breaking the rendering of old messages (or doing a
+  mass-rerender of old messages).
+
+### CommonJS/Typescript modules
+
+Webpack provides seemless interoperability between different module
+systems such as CommonJS, AMD and ES6. Our JS files are written in the
+CommonJS format, which specifies public functions and variables as
+properties of the special `module.exports` object.  We also currently
+assign said object to the global `window` variable, which is a hack
+allowing us to use modules without importing them with the `require()`
+statement.
+
+New modules should ideally be written in TypeScript (though in cases
+where one is moving code from an existing JavaScript module, the new
+commit should just move the code, not translate it to TypeScript).
+
+TypeScript provides more accurate information to development tools,
+allowing for better refactoring, auto-completion and static
+analysis. TypeScript uses an ES6-like module system.  Any declaration
+can be made public by adding the `export` keyword. Consuming
+variables, functions, etc exported from another module should be done
+with the `import` statement as oppose to accessing them from the
+global `window` scope.  Internally our typescript compiler is
+configured to transpile TS to the ES6 module system.
+
+Read more about these module systems here:
+* [TypeScript modules](https://www.typescriptlang.org/docs/handbook/modules.html)
+* [CommonJS](https://nodejs.org/api/modules.html#modules_modules)
+
 [Jinja2]: http://jinja.pocoo.org/
 [Handlebars]: http://handlebarsjs.com/
 [trans]: http://jinja.pocoo.org/docs/dev/templates/#i18n

docs/subsystems/index.rst

@@ -17,7 +17,6 @@ Subsystems Documentation
    caching
    realms
    management-commands
-   front-end-build-process
    schema-migrations
    migration-renumbering
    hashchange-system

tools/linter_lib/custom_check.py

@@ -617,7 +617,7 @@
      'good_lines': ['<button type="button" class="close close-alert-word-status" aria-label="{{t \'Close\' }}">'],
      'bad_lines': ['<button aria-label="foo"></button>']},
     {'pattern': 'script src="http',
-     'description': "Don't directly load dependencies from CDNs.  See docs/subsystems/front-end-build-process.md",
+     'description': "Don't directly load dependencies from CDNs.  See docs/subsystems/html-css.md",
      'exclude': set(["templates/corporate/billing.html", "templates/zerver/hello.html",
                      "templates/corporate/upgrade.html"]),
      'good_lines': ["{{ render_bundle('landing-page') }}"],

zerver/lib/storage.py

@@ -1,4 +1,5 @@
-# Useful reading is https://zulip.readthedocs.io/en/latest/subsystems/front-end-build-process.html
+# Useful reading is:
+# https://zulip.readthedocs.io/en/latest/subsystems/html-css.html#front-end-build-process
 
 import os
 from typing import Optional