wip update packaging

Finish packaging updates

wip refactor packaging, core, and dev modes

wip packaging updates

reorg

wip packaging setup

wip packaging

cleanup

Move mock packages out of jupyterlab dir

Always use yarn for developers

wip update dev and core mode

clean up commands and extension

cleanup

update docs

Update .yarnrc

update ci scripts

Update ensure repo

fix integrity check

Update gitignore

Switch back to jlpm

wip

Update scripts

cleanup

cleanup

cleanup

cleanup

no need for egg-info hack anymore

clean up handling of mock packages

update selenium check

Refactor travis

Cleanup

reorder matrix

Fix handling of link and unlinking extensions

Fix handling of postcss

Cache the miniconda dir

fix travis yml syntax

fix travis yml syntax

Update group names

fix miniconda handling

Update docs

Fix handling of miniconda path

reorder path

tweak karma settings

cleanup

Remove manifest in favor of package_data

fix path addition

more setup cleanup

Add staging data to package data

Make sure our data_files work

add comment

use glob2 to get proper files

defer using glob2

cleanup

Inline handling of ** globs

Add a node_modules blocker

use fresh environments on every build

fix removal of conda envs

use env create for requirements file

ensure selenium is installed

Fix jupyterlab#3231

front load the longer job

Fix jupyterlab#3227

Uniform handling of error messages for extension commands

Let process errors propagate

Fix #jupyterlab#3264

Fix validation and add tests

Fix property name

fix handling of app_dir

Remove pdb from py.test so builds don't time out

Use npm to run scripts in the re-usable build utils

Clean up build:update

Fix glob handling

Use declarative specs for data files and package files

Add detection of extension module shadowing

Fix package_data_spec

Add workaround for git clean behavior on windows

remove debug statement

Clean up setupbase

Address comments

Remove outdated file check

glob handling cleanup

Incorporate jupyter-packaging #20

clean up setup.py

Make sure build:src works

Reinstate build:src

Fix spelling

wip add test manager

fix pip install .

Clean up selenium check implementation and temp usage

Invert requirements

clean up handling of launcher versions in script

fix version spec

.gitignore

@@ -2,10 +2,12 @@ MANIFEST
 build
 dist
 lib
-jupyterlab/build
+jupyterlab/static
 jupyterlab/schemas
 jupyterlab/themes
 jupyterlab/geckodriver
+dev_mode/schemas
+dev_mode/themes
 
 node_modules
 .cache

.travis.yml

@@ -7,12 +7,20 @@ addons:
   firefox: latest
 cache:
   yarn: true
+  directories:
+    - $HOME/miniconda
+    - $HOME/.cache/pip
 env:
   matrix:
-    - GROUP=tests
-    - GROUP=coverage
-    - GROUP=other
+    - GROUP=py3
+    - GROUP=py2
+    - GROUP=js
+    - GROUP=integrity
+    - GROUP=cli
+    - GROUP=docs
+    - GROUP=js_cov
   global:
+  - MINICONDA_DIR="$HOME/miniconda"
   - GH_REF: github.com/jupyterlab/jupyterlab.git
   - secure: MWpTI6cj3/Bnmtrr0Oqlp2JeWqDneB9aEjlQDaRxLOkqVbxhqDcYW9qAgZZP+sq29vT5oVMWzyCirteKxJfG2vy3HQE1XNLhz82Sf/7sE6DQ51gohl0CcOeA/uA8hCXEw97hneFWsZgHKqSoch7nVDsE3qfYgO+930jHlnxYApJGP9hZFv2Q2NVa6+99kipEYS4BY/yBDYKy6/t4kXcnBrUlNaPtdjnXcrY9esLZ7EQtkaG5VqcQVIBaLJKGF5Q7Aufj5nCFaZ6hZDF1Bi/AbmIbVWFyiT+22i8DZK6YwenECckyzoWkl+bEhYepWsgBKh/BDgPBAmPWKHgU5V4apDaGqZBhF7FP6H02AdZYYuCwl47jyakqvWLZW7oDmorL+HsWG5HQ3m0tMT2ywdbwNOiD39tiPPXjsvROh5ys9vL6NzQvxILCeEOnzcZrFuxi2LGEZfnlqRIjkh1llUAvNc3mOycRLWDOwVQa2+U59qDRXCSY2RD+MOfcdFUGengVujTMaAPMBUa3E33/ZIOOKJtR5TIajYZvd9B2uDlz02QfvTK+hrTaNYJjRZ8WCaeSM/CIKdoLw+29MNO6eqtchw0/vNvM8c9EkhrhMQKcY04OecVhmZkemFhd4SD5l92VX3z3xSxLkmazfNkj3CigWDXNxfDd2ORoGjA46Pga8RM=
 install:

.yarnrc

@@ -1,3 +1,3 @@
-yarn-path "./jupyterlab/yarn.js"
+yarn-path "./jupyterlab/staging/yarn.js"
 workspaces-experimental true
 registry "https://registry.npmjs.org"

CONTRIBUTING.md

@@ -23,7 +23,7 @@ All source code is written in [TypeScript](http://www.typescriptlang.org/Handboo
 
 ## Setting Up a Development Environment
 
-### Installing Node.js and npm
+### Installing Node.js and jlpm
 
 Building the JupyterLab from its GitHub source code requires Node.js version
 4+.
@@ -82,14 +82,13 @@ jupyter serverextension enable --py jupyterlab
 
 Notes:
 
-* The `jlpm` command is a JupyterLab-provided, locked version of the [yarn](https://yarnpkg.com/en/).  If you have `yarn` installed already, you can use
-the `yarn` command when developing, and it will use the local version of `yarn`
- in `jupyterlab/yarn.js` when run in the repository or a built application
- directory.
+* The `jlpm` command is a JupyterLab-provided, locked version of the [yarn](https://yarnpkg.com/en/) package manager.  If you have `yarn` installed 
+already, you can use the `yarn` command when developing, and it will use the 
+local version of `yarn` in `jupyterlab/yarn.js` when run in the repository or 
+a built application directory.
 
-* At times, it may be necessary to clean your local repo with the command `npm run clean:slate`.  This will clean the repository, and re-install and
-rebuild.  Note that we use `npm` in this one instance because the `jlpm`
-command may not be available at the time.
+* At times, it may be necessary to clean your local repo with the command `jlpm run clean:slate`.  This will clean the repository, and re-install and
+rebuild.
 
 * If `pip` gives a `VersionConflict` error, it usually means that the installed
 version of `jupyterlab_launcher` is out of date. Run `pip install --upgrade
@@ -156,22 +155,13 @@ provides additional architecture information.
 The repository consists of many npm packages that are managed using the lerna
 build tool.  The npm package source files are in the `packages/` subdirectory.
 
-**Prerequisites**
-
-- [node](http://nodejs.org/) (preferably version 5 or later)
-- Jupyter notebook server version 4.3 or later (to run examples)
-
-```bash
-npm install --save jupyterlab
-```
-
 ### Build the NPM Packages from Source
 
 ```bash
 git clone https://github.com/jupyterlab/jupyterlab.git
 cd jupyterlab
 pip install -e .
-jlpm install
+jlpm
 jlpm run build:packages
 ```
 
@@ -211,11 +201,11 @@ the initial build if the assets are already built.
 ## Build Utilities
 
 There are a series of build utilities for maintaining the repository.
-To get a suggested version for a library use `npm run get:dependency foo`.
-To update the version of a library across the repo use `npm run update:dependency foo@^x.x`.
-To remove an unwanted dependency use `npm run remove:dependency foo`.
+To get a suggested version for a library use `jlpm run get:dependency foo`.
+To update the version of a library across the repo use `jlpm run update:dependency foo@^x.x`.
+To remove an unwanted dependency use `jlpm run remove:dependency foo`.
 
-The key utility is `npm run integrity`, which ensures the integrity of
+The key utility is `jlpm run integrity`, which ensures the integrity of
 the packages in the repo. It will:
 
 - Ensure the core package version dependencies match everywhere.
@@ -228,10 +218,10 @@ in the repository at once, instead of 50+ individual builds.
 
 The integrity script also allows you to automatically add a dependency for
 a package by importing from it in the TypeScript file, and then running:
-`npm run integrity && npm install` from the repo root.
+`jlpm run integrity` from the repo root.
 
 We also have scripts for creating and removing packages in `packages/`,
-`npm run create:package` and `npm run remove:package`.
+`jlpm run create:package` and `jlpm run remove:package`.
 
 
 ## Notes

README.md

@@ -56,15 +56,10 @@ Note: If installing using `pip install --user`, you must add the user-level
  `bin` directory to your `PATH` environment variable in order to launch
  `jupyter lab`.
 
-Note: JupyterLab can be installed from a git checkout using `pip`.  Note
-that you will have to build the application after installing to get the static 
-assets.  `pip` will warn you that the static assets are not included when 
-installing, and it will fail to build a cached wheel if it decides to try 
-and make one.  Example:
+JupyterLab can be installed from a git checkout using `pip`.  Example:
 
 ```bash
 pip install git+git://github.com/jupyterlab/jupyterlab.git
-jupyter lab build   # Requires nodejs
 jupyter serverextension enable --py jupyterlab --sys-prefix
 ```
 

appveyor.yml

@@ -12,7 +12,7 @@ environment:
 
 # build cache to preserve files/folders between builds
 cache:
-  - '%LOCALAPPDATA%\\Yarn' # yarn cache
+  - '%LOCALAPPDATA%\\Yarn' # jlpm cache
 
 # scripts that run after cloning repository
 install:
@@ -37,8 +37,8 @@ install:
   - 'jlpm versions'
   - 'jlpm config current'
   - 'jlpm cache list'
-  - 'jlpm install'
-  - 'jlpm run build:main'
+  - 'jlpm'
+  - 'jlpm run build'
   - 'jupyter lab build'
 
 build: off
@@ -47,9 +47,13 @@ build: off
 test_script:
   # Run integrity first we we see the message.
   - 'jlpm run integrity'
+  - 'python -m jupyterlab.selenium_check'
+  - 'pip install jupyterlab_launcher==0.6'
   - 'py.test'
   - 'git status'
   - 'jlpm run build'
   - 'jlpm run build:test'
   - 'jlpm test || jlpm test || jlpm test'
-  - 'python -m jupyterlab.selenium_check'
+  - 'python -m jupyterlab.selenium_check --dev-mode'
+
+

buildutils/src/create-package.ts

@@ -37,5 +37,6 @@ inquirer.prompt(questions).then(answers => {
   data.name = name;
   data.description = description;
   utils.writePackageData(jsonPath, data);
-  utils.run('jlpm run integrity');
+  // Use npm here so this file can be used outside of JupyterLab.
+  utils.run('npm run integrity');
 });

buildutils/src/ensure-repo.ts

@@ -113,7 +113,7 @@ function ensureJupyterlab(): string[] {
   let version = String(childProcess.execSync(cmd)).trim();
 
   let basePath = path.resolve('.');
-  let corePath = path.join(basePath, 'jupyterlab', 'package.json');
+  let corePath = path.join(basePath, 'dev_mode', 'package.json');
   let corePackage = utils.readJSONFile(corePath);
 
   corePackage.jupyterlab.extensions = {};
@@ -183,7 +183,14 @@ function ensureIntegrity(): boolean {
   let messages: { [key: string]: string[] } = {};
 
   // Pick up all the package versions.
-  utils.getLernaPaths().forEach(pkgPath => {
+  let paths = utils.getLernaPaths();
+
+  // These two are not part of the workspaces but should be kept
+  // in sync.
+  paths.push('./jupyterlab/tests/mock_packages/extension');
+  paths.push('./jupyterlab/tests/mock_packages/mimeextension');
+
+  paths.forEach(pkgPath => {
     // Read in the package.json.
     let data: any;
     try {

buildutils/src/patch-release.ts

@@ -27,16 +27,20 @@ if (!fs.existsSync(packagePath)) {
 
 // Perform the patch operations.
 console.log('Patching', target, '...');
-utils.run('jlpm run build:packages');
-utils.run('jlpm version patch', { cwd: packagePath });
-utils.run('jlpm publish', { cwd: packagePath});
+// Use npm here so this file can be used outside of JupyterLab.
+utils.run('npm run build:packages');
+utils.run('npm version patch', { cwd: packagePath });
+utils.run('npm publish', { cwd: packagePath});
+
+// Update the static folder.
+utils.run('npm run build:update');
 
 // Extract the new package info.
 let data = utils.readJSONFile(path.join(packagePath, 'package.json'));
 let name = data.name;
 let version = data.version;
 
-utils.run('jlpm run integrity');
+utils.run('npm run integrity');
 utils.run('git commit -a -m "Release ' + name + '@' + version + '"');
 utils.run('git tag ' + name + '@' + version);
 

buildutils/src/remove-package.ts

@@ -45,7 +45,7 @@ fs.removeSync(path.dirname(packagePath));
 
 // Update the core jupyterlab build dependencies.
 try {
-  utils.run('jlpm run integrity');
+  utils.run('npm run integrity');
 } catch (e) {
   if (!process.env.TRAVIS_BRANCH) {
     console.error(e);

buildutils/src/update-core-mode.ts

@@ -0,0 +1,35 @@
+/*-----------------------------------------------------------------------------
+| Copyright (c) Jupyter Development Team.
+| Distributed under the terms of the Modified BSD License.
+|----------------------------------------------------------------------------*/
+
+import * as fs from 'fs-extra';
+import * as path from 'path';
+import * as utils from './utils';
+
+// Get the dev mode package.json file.
+let data = utils.readJSONFile('./dev_mode/package.json');
+
+// Update the values that need to change and write to staging.
+data['scripts']['build'] = 'webpack';
+data['scripts']['watch'] = 'webpack --watch';
+data['scripts']['build:prod'] = "webpack --define process.env.NODE_ENV=\"'production'\"";
+data['jupyterlab']['outputDir'] = '..';
+data['jupyterlab']['staticDir'] = '../static';
+data['jupyterlab']['linkedPackages'] = {};
+
+let staging = './jupyterlab/staging';
+utils.writePackageData(path.join(staging, 'package.json'), data);
+
+// Update our index file and webpack file.
+fs.copySync('./dev_mode/index.js', './jupyterlab/staging/index.js');
+fs.copySync('./dev_mode/webpack.config.js',
+            './jupyterlab/staging/webpack.config.js');
+
+
+// Update the jlpm.lock file.
+utils.run('jlpm', { cwd: staging });
+
+
+// Build the core assets.
+utils.run('jlpm run build:prod', { cwd: staging });

clean.py

@@ -0,0 +1,16 @@
+import os
+import subprocess
+
+here = os.path.abspath(os.path.dirname(__file__))
+
+
+# Workaround for https://github.com/git-for-windows/git/issues/607
+if os.name == 'nt':
+    for (root, dnames, files) in os.walk(here):
+        if 'node_modules' in dnames:
+            subprocess.check_call(['rmdir', '/s', '/q', 'node_modules'],
+                cwd=root, shell=True)
+            dnames.remove('node_modules')
+
+
+subprocess.check_call(['git', 'clean', '-dfx'], cwd=here)

docs/extensions_dev.md

@@ -93,29 +93,19 @@ jupyter lab --watch
 ```
 
 This will cause the application to incrementally rebuild when one of the
-linked packages changes.  You can open another terminal and run
-
-```bash
-npm run watch
-```
-
-to automatically recompile the TypeScript files while editing.  Note that
-only the compiled JavaScript files (and the CSS files) are watched by the 
-WebPack process.  
-
-The is also a `jupyter lab --fast-watch` that allows you to skip the
-initial build if the app is already built.  
+linked packages changes.  Note that only compiled JavaScript files (and the 
+CSS files) are watched by the WebPack process.  
 
 Note that the application is built against **released** versions of the
 core JupyterLab extensions.  If your extension depends on JupyterLab
 packages, it should be compatible with the dependencies in the
-`jupyterlab/package.app.json` file.  If you must
+`jupyterlab/static/package.json` file.  If you must
 install a extension into a development branch of JupyterLab, you
 have to graft it into the source tree of JupyterLab itself.
 This may be done using the command
 
 ```
-jlpm run add:sibling <path-or-url> && npm install
+jlpm run add:sibling <path-or-url>
 ```
 
 in the JupyterLab root directory, where `<path-or-url>` refers either to an