Make Webpacker the default JavaScript compiler for Rails 6 (rails#33079)

* Use Webpacker by default on new apps

* Stop including coffee-rails by default

* Drop using a js_compressor by default

* Drop extra test for coffeescript inclusion by default

* Stick with skip_javascript to signify skipping webpack

* Don't install a JS runtime by default any more

* app/javascript will be the new default directory for JS

* Make it clear that this is just for configuring the default Webpack framework setup now

* Start using the Webpack tag in the default layout

* Irrelevant test

* jQuery is long gone

* Stop having asset pipeline compile default application.js

* Add rails-ujs by default to the Webpack setup

* Add Active Storage JavaScript to application.js pack by default

* Consistent quoting

* Add Turbolinks to default pack

* Add Action Cable to default pack

Need some work on how to set the global consumer that channels will
work with. @javan?

* Require all channels by default and use a separate consumer stub

* Channel generator now targets Webpack style

* Update task docs to match new generator style

* Use uniform import style

* Drop the JS assets generator

It was barely helpful as it was. It’s no longer helpful in a Webpacked
world. Sayonara!

* Add app/javascript to the stats directories

* Simpler import style

Which match the other imports.

* Address test failures from dropping JS compilation (and compression)

* webpacker-default: Modify `AssetsGeneratorTest`

Before:

```
$ bin/test test/generators/assets_generator_test.rb
Run options: --seed 46201

F

Failure:
AssetsGeneratorTest#test_assets [/Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/assets_generator_test.rb:12]:
Expected file "app/assets/javascripts/posts.js" to exist, but does not

bin/test /Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/assets_generator_test.rb:10

.

Finished in 0.031343s, 63.8101 runs/s, 95.7152 assertions/s.
2 runs, 3 assertions, 1 failures, 0 errors, 0 skips
```

After:

```
$ bin/test test/generators/assets_generator_test.rb
Run options: --seed 43571

..

Finished in 0.030370s, 65.8545 runs/s, 65.8545 assertions/s.
2 runs, 2 assertions, 0 failures, 0 errors, 0 skips
```

* webpacker-default: Modify `ChannelGeneratorTest`

Before:

```
$ bin/test test/generators/channel_generator_test.rb
Run options: --seed 8986

.F

Failure:
ChannelGeneratorTest#test_channel_with_multiple_actions_is_created [/Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:43]:
Expected file "app/assets/javascripts/channels/chat.js" to exist, but does not

bin/test /Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:34

.F

Failure:
ChannelGeneratorTest#test_channel_is_created [/Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:29]:
Expected file "app/assets/javascripts/channels/chat.js" to exist, but does not

bin/test /Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:22

E

Error:
ChannelGeneratorTest#test_cable_js_is_created_if_not_present_already:
Errno::ENOENT: No such file or directory @ apply2files - /Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/fixtures/tmp/app/assets/javascripts/cable.js

bin/test /Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:60

F

Failure:
ChannelGeneratorTest#test_channel_suffix_is_not_duplicated [/Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:87]:
Expected file "app/assets/javascripts/channels/chat.js" to exist, but does not

bin/test /Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:80

F

Failure:
ChannelGeneratorTest#test_channel_on_revoke [/Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:77]:
Expected file "app/assets/javascripts/cable.js" to exist, but does not

bin/test /Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:68

Finished in 0.064384s, 108.7227 runs/s, 481.4861 assertions/s.
7 runs, 31 assertions, 4 failures, 1 errors, 0 skips
```

After:

```
$ bin/test test/generators/channel_generator_test.rb
Run options: --seed 44857

.......

Finished in 0.060243s, 116.1961 runs/s, 697.1764 assertions/s.
7 runs, 42 assertions, 0 failures, 0 errors, 0 skips
```

* Fix shared generator tests.

* webpacker-default: Modify `ControllerGeneratorTest`

The JS assets generator was dropped. ref. rails@46215b1

* Revert "Simpler import style". It's currently failing with an error of "TypeError: undefined is not an object (evaluating '__WEBPACK_IMPORTED_MODULE_2_activestorage___default.a.start')". Waiting for @javan to have a look.

This reverts commit 5d3ebb7.

* require webpacker in test app

* Add webpacker without making the build hang/timeout. (rails#33640)

* use yarn workspaces to allow for installing unreleased packages and only generate js/bootsnap when required

* no longer need to have webpacker in env templates as webpacker moved this config to yml file

* Fix rubocop violation

* Got the test passing for the running scaffold

* update expected lines of code

* update middleware tests to account for webpacker

* disable js in plugins be default to get the tests passing (rails#34009)

* clear codeclimate report issues

* Anything newer than currently released is good

* Use Webpacker development version during development of Rails

* Edge should get development webpacker as well

* Add changelog entry for Webpacker change

.gitignore

@@ -14,3 +14,5 @@ debug.log
 node_modules/
 package-lock.json
 pkg/
+/tmp/
+/yarn.lock

.travis.yml

@@ -42,6 +42,11 @@ before_install:
   - "[[ $GEM != 'av:ujs' ]] || nvm install node"
   - "[[ $GEM != 'av:ujs' ]] || node --version"
   - "[[ $GEM != 'av:ujs' ]] || (cd actionview && npm install)"
+  - "[[ $GEM != 'ar:mysql2' ]] || [[ $MYSQL == 'mariadb' ]] || sudo mysql -e \"use mysql; update user set authentication_string='' where User='root'; update user set plugin='mysql_native_password';FLUSH PRIVILEGES;\""
+  - "[[ $GEM != 'ar:mysql2' ]] || [[ $MYSQL == 'mariadb' ]] || sudo mysql_upgrade"
+  - "[[ $GEM != 'ar:mysql2' ]] || [[ $MYSQL == 'mariadb' ]] || sudo service mysql restart"
+  - "[[ $GEM != 'railties' ]] || (curl -o- -L https://yarnpkg.com/install.sh | bash)"
+  - "[[ $GEM != 'railties' ]] || export PATH=$HOME/.yarn/bin:$PATH"
 
 before_script:
   # Set Sauce Labs username and access key. Obfuscated, purposefully not encrypted.

.yarnrc

@@ -0,0 +1,2 @@
+workspaces-experimental true
+--add.prefer-offline true

Gemfile

@@ -15,7 +15,7 @@ gem "rack-cache", "~> 1.2"
 gem "coffee-rails"
 gem "sass-rails"
 gem "turbolinks", "~> 5"
-
+gem "webpacker", github: "rails/webpacker", require: ENV["SKIP_REQUIRE_WEBPACKER"] != "true"
 # require: false so bcrypt is loaded only when has_secure_password is used.
 # This is to avoid Active Model (and by extension the entire framework)
 # being dependent on a binary library.

Gemfile.lock

@@ -30,6 +30,15 @@ GIT
     queue_classic (3.2.0.RC1)
       pg (>= 0.17, < 2.0)
 
+GIT
+  remote: https://github.com/rails/webpacker.git
+  revision: 48d9fd52c3e9637dbb21e551b48c84c0f2dbb2ed
+  specs:
+    webpacker (4.0.0.pre.pre.2)
+      activesupport (>= 4.2)
+      rack-proxy (>= 0.6.1)
+      railties (>= 4.2)
+
 PATH
   remote: .
   specs:
@@ -364,6 +373,8 @@ GEM
       rack (>= 0.4)
     rack-protection (2.0.3)
       rack
+    rack-proxy (0.6.5)
+      rack
     rack-test (1.1.0)
       rack (>= 1.0, < 3)
     rails-dom-testing (2.0.3)
@@ -565,6 +576,7 @@ DEPENDENCIES
   uglifier (>= 1.3.0)
   w3c_validators
   wdm (>= 0.1.0)
+  webpacker!
   websocket-client-simple!
 
 BUNDLED WITH

actioncable/lib/rails/generators/channel/USAGE

@@ -1,14 +1,12 @@
 Description:
 ============
-    Stubs out a new cable channel for the server (in Ruby) and client (in CoffeeScript).
+    Stubs out a new cable channel for the server (in Ruby) and client (in JavaScript).
     Pass the channel name, either CamelCased or under_scored, and an optional list of channel actions as arguments.
 
-    Note: Turn on the cable connection in app/assets/javascripts/cable.js after generating any channels.
-
 Example:
 ========
     rails generate channel Chat speak
 
-    creates a Chat channel class and CoffeeScript asset:
+    creates a Chat channel class and JavaScript asset:
         Channel:    app/channels/chat_channel.rb
-        Assets:     app/assets/javascripts/channels/chat.coffee
+        Assets:     app/javascript/channels/chat_channel.js

actioncable/lib/rails/generators/channel/channel_generator.rb

@@ -16,10 +16,11 @@ def create_channel_file
 
         if options[:assets]
           if behavior == :invoke
-            template "assets/cable.js", "app/assets/javascripts/cable.js"
+            template "javascript/index.js", "app/javascript/channels/index.js"
+            template "javascript/consumer.js", "app/javascript/channels/consumer.js"
           end
 
-          js_template "assets/channel", File.join("app/assets/javascripts/channels", class_path, "#{file_name}")
+          js_template "javascript/channel", File.join("app/javascript/channels", class_path, "#{file_name}_channel")
         end
 
         generate_application_cable_files

actioncable/lib/rails/generators/channel/templates/assets/channel.js.tt → actioncable/lib/rails/generators/channel/templates/javascript/channel.js.tt

@@ -1,4 +1,6 @@
-App.<%= class_name.underscore %> = App.cable.subscriptions.create("<%= class_name %>Channel", {
+import consumer from "./consumer"
+
+consumer.subscriptions.create("<%= class_name %>Channel", {
   connected: function() {
     // Called when the subscription is ready for use on the server
   },

actioncable/lib/rails/generators/channel/templates/assets/cable.js.tt → actioncable/lib/rails/generators/channel/templates/javascript/consumer.js.tt

@@ -1,13 +1,6 @@
 // Action Cable provides the framework to deal with WebSockets in Rails.
 // You can generate new channels where WebSocket features live using the `rails generate channel` command.
-//
-//= require action_cable
-//= require_self
-//= require_tree ./channels
 
-(function() {
-  this.App || (this.App = {});
+import ActionCable from "actioncable"
 
-  App.cable = ActionCable.createConsumer();
-
-}).call(this);
+export default ActionCable.createConsumer()

actioncable/lib/rails/generators/channel/templates/javascript/index.js.tt

@@ -0,0 +1,5 @@
+// Load all the channels within this directory and all subdirectories. 
+// Channel files must be named *_channel.js.
+
+const channels = require.context('.', true, /\_channel\.js$/)
+channels.keys().forEach(channels)

activestorage/test/dummy/app/assets/config/manifest.js

@@ -1,5 +1,3 @@
 
 //= link_tree ../images
-//= link_directory ../javascripts .js
 //= link_directory ../stylesheets .css
-//= link active_storage_manifest.js

activestorage/test/dummy/config/environments/production.rb

@@ -25,8 +25,7 @@
   # Apache or NGINX already handles this.
   config.public_file_server.enabled = ENV["RAILS_SERVE_STATIC_FILES"].present?
 
-  # Compress JavaScripts and CSS.
-  config.assets.js_compressor = :uglifier
+  # Compress CSS using a preprocessor.
   # config.assets.css_compressor = :sass
 
   # Do not fallback to assets pipeline if a precompiled asset is missed.

activestorage/test/dummy/config/webpacker.yml

@@ -0,0 +1,72 @@
+# Note: You must restart bin/webpack-dev-server for changes to take effect
+
+default: &default
+  source_path: app/javascript
+  source_entry_path: packs
+  public_output_path: packs
+  cache_path: tmp/cache/webpacker
+  check_yarn_integrity: false
+
+  # Additional paths webpack should lookup modules
+  # ['app/assets', 'engine/foo/app/assets']
+  resolved_paths: []
+
+  # Reload manifest.json on all requests so we reload latest compiled packs
+  cache_manifest: false
+
+  extensions:
+    - .js
+    - .sass
+    - .scss
+    - .css
+    - .module.sass
+    - .module.scss
+    - .module.css
+    - .png
+    - .svg
+    - .gif
+    - .jpeg
+    - .jpg
+
+development:
+  <<: *default
+  compile: true
+
+  # Verifies that versions and hashed value of the package contents in the project's package.json
+  check_yarn_integrity: true
+
+  # Reference: https://webpack.js.org/configuration/dev-server/
+  dev_server:
+    https: false
+    host: localhost
+    port: 3035
+    public: localhost:3035
+    hmr: false
+    # Inline should be set to true if using HMR
+    inline: true
+    overlay: true
+    compress: true
+    disable_host_check: true
+    use_local_ip: false
+    quiet: false
+    headers:
+      'Access-Control-Allow-Origin': '*'
+    watch_options:
+      ignored: /node_modules/
+
+
+test:
+  <<: *default
+  compile: true
+
+  # Compile test packs to a separate directory
+  public_output_path: packs-test
+
+production:
+  <<: *default
+
+  # Production depends on precompilation of packs prior to booting for performance.
+  compile: false
+
+  # Cache manifest.json for performance
+  cache_manifest: true

package.json

@@ -0,0 +1,14 @@
+{
+  "private": true,
+  "workspaces": [
+    "actioncable",
+    "activestorage",
+    "actionview",
+    "tmp/templates/app_template",
+    "railties/test/fixtures/tmp/bukkits/**/test/dummy",
+    "railties/test/fixtures/tmp/bukkits/**/spec/dummy"
+  ],
+  "dependencies": {
+    "webpack": "^4.17.1"
+  }
+}

railties/CHANGELOG.md

@@ -1,3 +1,11 @@
+*   Use Webpacker by default to manage app-level JavaScript through the new app/javascript directory.
+    Sprockets is now solely in charge, by default, of compiling CSS and other static assets.
+    Action Cable channel generators will create ES6 stubs rather than use CoffeeScript.
+    Active Storage, Action Cable, Turbolinks, and Rails-UJS are loaded by a new application.js pack.
+    Generators no longer generate JavaScript stubs.
+
+    *DHH*, *Lachlan Sylvester*
+
 *   Refactors `migrations_paths` command option in generators
     to `database` (aliased as `db`). Now, the migrations paths
     will be read from the specified database configuration in the

railties/Rakefile

@@ -28,6 +28,9 @@ namespace :test do
     require "bundler/setup" unless defined?(Bundler)
     require "active_support"
 
+    # Only generate the template app once.
+    require_relative "test/isolation/abstract_unit"
+
     failing_files = []
 
     dirs = (ENV["TEST_DIR"] || ENV["TEST_DIRS"] || "**").split(",")

railties/lib/rails/generators.rb

@@ -33,8 +33,6 @@ module Generators
       rails: {
         actions: "-a",
         orm: "-o",
-        javascripts: "-j",
-        javascript_engine: "-je",
         resource_controller: "-c",
         scaffold_controller: "-c",
         stylesheets: "-y",

railties/lib/rails/generators/app_base.rb

@@ -68,10 +68,7 @@ def self.add_shared_options_for(name)
         class_option :skip_listen,         type: :boolean, default: false,
                                            desc: "Don't generate configuration that depends on the listen gem"
 
-        class_option :skip_coffee,         type: :boolean, default: false,
-                                           desc: "Don't use CoffeeScript"
-
-        class_option :skip_javascript,     type: :boolean, aliases: "-J", default: false,
+        class_option :skip_javascript,     type: :boolean, aliases: "-J", default: name == "plugin",
                                            desc: "Skip JavaScript files"
 
         class_option :skip_turbolinks,     type: :boolean, default: false,
@@ -327,59 +324,30 @@ def convert_database_option_for_jruby
       def assets_gemfile_entry
         return [] if options[:skip_sprockets]
 
-        gems = []
-        gems << GemfileEntry.version("sass-rails", "~> 5.0",
-                                     "Use SCSS for stylesheets")
-
-        if !options[:skip_javascript]
-          gems << GemfileEntry.version("uglifier",
-                                     ">= 1.3.0",
-                                     "Use Uglifier as compressor for JavaScript assets")
-        end
-
-        gems
+        GemfileEntry.version("sass-rails", "~> 5.0", "Use SCSS for stylesheets")
       end
 
       def webpacker_gemfile_entry
-        return [] unless options[:webpack]
+        return [] if options[:skip_javascript]
 
-        comment = "Transpile app-like JavaScript. Read more: https://github.com/rails/webpacker"
-        GemfileEntry.new "webpacker", nil, comment
+        if options.dev? || options.edge?
+          GemfileEntry.github "webpacker", "rails/webpacker", nil, "Use development version of Webpacker"
+        else
+          GemfileEntry.new "webpacker", nil, "Transpile app-like JavaScript. Read more: https://github.com/rails/webpacker"
+        end
       end
 
       def jbuilder_gemfile_entry
         comment = "Build JSON APIs with ease. Read more: https://github.com/rails/jbuilder"
         GemfileEntry.new "jbuilder", "~> 2.5", comment, {}, options[:api]
       end
 
-      def coffee_gemfile_entry
-        GemfileEntry.version "coffee-rails", "~> 4.2", "Use CoffeeScript for .coffee assets and views"
-      end
-
       def javascript_gemfile_entry
-        if options[:skip_javascript] || options[:skip_sprockets]
+        if options[:skip_javascript] || options[:skip_turbolinks]
           []
         else
-          gems = [javascript_runtime_gemfile_entry]
-          gems << coffee_gemfile_entry unless options[:skip_coffee]
-
-          unless options[:skip_turbolinks]
-            gems << GemfileEntry.version("turbolinks", "~> 5",
-             "Turbolinks makes navigating your web application faster. Read more: https://github.com/turbolinks/turbolinks")
-          end
-
-          gems
-        end
-      end
-
-      def javascript_runtime_gemfile_entry
-        comment = "See https://github.com/rails/execjs#readme for more supported runtimes"
-        if defined?(JRUBY_VERSION)
-          GemfileEntry.version "therubyrhino", nil, comment
-        elsif RUBY_PLATFORM.match?(/mingw|mswin/)
-          GemfileEntry.version "duktape", nil, comment
-        else
-          GemfileEntry.new "mini_racer", nil, comment, { platforms: :ruby }, true
+          [ GemfileEntry.version("turbolinks", "~> 5",
+             "Turbolinks makes navigating your web application faster. Read more: https://github.com/turbolinks/turbolinks") ]
         end
       end
 
@@ -452,9 +420,9 @@ def run_bundle
       end
 
       def run_webpack
-        if !(webpack = options[:webpack]).nil?
+        unless options[:skip_javascript]
           rails_command "webpacker:install"
-          rails_command "webpacker:install:#{webpack}" unless webpack == "webpack"
+          rails_command "webpacker:install:#{options[:webpack]}" if options[:webpack] && options[:webpack] != "webpack"
         end
       end