Merge pull request rspec#1946 from rspec/aggregate-failures

Aggregate failures Integration

.jrubyrc

@@ -0,0 +1,5 @@
+# Remove `.java` lines from JRuby stacktraces. Necessary for a passing travis build
+# on JRuby in 1.8 mode for the new failure aggregator specs. Our generated test
+# fixture doesn't know how to deal with excess java lines so it's best to ignore
+# those lines.
+backtrace.mask=true

Changelog.md

@@ -40,6 +40,8 @@ Enhancements:
   (e.g. `.rspec` or `~/.rspec` or `ENV['SPEC_OPTS']`) so they can
   easily find the source of the problem. (Myron Marston, #1940)
 * Add pending message contents to the json formatter output. (Jon Rowe, #1949)
+* Add shared group backtrace to the output displayed by the built-in
+  formatters for pending examples that have been fixed. (Myron Marston, #1946)
 
 Bug Fixes:
 

features/.nav

@@ -46,6 +46,7 @@
   - default_path.feature
 - expectation_framework_integration:
   - configure_expectation_framework.feature
+  - failure_aggregation.feature
 - mock_framework_integration:
   - use_rspec.feature
   - use_flexmock.feature

features/expectation_framework_integration/aggregating_failures.feature

@@ -0,0 +1,277 @@
+Feature: Aggregating Failures
+
+  RSpec::Expectations provides [`aggregate_failures`](../../../rspec-expectations/docs/aggregating-failures), an API that allows you to group a set of expectations and see all the failures at once, rather than it aborting on the first failure. RSpec::Core improves on this feature in a couple of ways:
+
+    * RSpec::Core provides much better failure output, adding code snippets and backtraces to the sub-failures, just like it does for any normal failure.
+    * RSpec::Core provides [metadata](../metadata/user-defined-metadata) integration for this feature. Each example that is tagged with `:aggregate_failures` will be wrapped in an `aggregate_failures` block. You can also use `config.define_derived_metadata` to apply this to every example automatically.
+
+  The metadata form is quite convenient, but may not work well for end-to-end tests that have multiple distinct steps. For example, consider a spec for an HTTP client workflow that (1) makes a request, (2) expects a redirect, (3) follows the redirect, and (4) expects a particular response.  You probably want the `expect(response.status).to be_between(300, 399)` expectation to immediately abort if it fails, because you can't perform the next step (following the redirect) if that is not satisfied. For these situations, we encourage you to use the `aggregate_failures` block form to wrap each set of expectations that represents a distinct step in the test workflow.
+
+  Background:
+    Given a file named "lib/client.rb" with:
+      """ruby
+      Response = Struct.new(:status, :headers, :body)
+
+      class Client
+        def self.make_request
+          Response.new(404, { "Content-Type" => "text/plain" }, "Not Found")
+        end
+      end
+      """
+
+  Scenario: Use `aggregate_failures` block form
+    Given a file named "spec/use_block_form_spec.rb" with:
+      """ruby
+      require 'client'
+
+      RSpec.describe Client do
+        it "returns a successful response" do
+          response = Client.make_request
+
+          aggregate_failures "testing reponse" do
+            expect(response.status).to eq(200)
+            expect(response.headers).to include("Content-Type" => "application/json")
+            expect(response.body).to eq('{"message":"Success"}')
+          end
+        end
+      end
+      """
+    When I run `rspec spec/use_block_form_spec.rb`
+    Then it should fail and list all the failures:
+      """
+      Failures:
+
+        1) Client returns a successful response
+           Got 3 failures from failure aggregation block "testing reponse".
+           # ./spec/use_block_form_spec.rb:7
+
+           1.1) Failure/Error: expect(response.status).to eq(200)
+
+                  expected: 200
+                       got: 404
+
+                  (compared using ==)
+                # ./spec/use_block_form_spec.rb:8
+
+           1.2) Failure/Error: expect(response.headers).to include("Content-Type" => "application/json")
+                  expected {"Content-Type" => "text/plain"} to include {"Content-Type" => "application/json"}
+                  Diff:
+                  @@ -1,2 +1,2 @@
+                  -[{"Content-Type"=>"application/json"}]
+                  +"Content-Type" => "text/plain",
+                # ./spec/use_block_form_spec.rb:9
+
+           1.3) Failure/Error: expect(response.body).to eq('{"message":"Success"}')
+
+                  expected: "{\"message\":\"Success\"}"
+                       got: "Not Found"
+
+                  (compared using ==)
+                # ./spec/use_block_form_spec.rb:10
+      """
+
+  Scenario: Use `:aggregate_failures` metadata
+    Given a file named "spec/use_metadata_spec.rb" with:
+      """ruby
+      require 'client'
+
+      RSpec.describe Client do
+        it "returns a successful response", :aggregate_failures do
+          response = Client.make_request
+
+          expect(response.status).to eq(200)
+          expect(response.headers).to include("Content-Type" => "application/json")
+          expect(response.body).to eq('{"message":"Success"}')
+        end
+      end
+      """
+    When I run `rspec spec/use_metadata_spec.rb`
+    Then it should fail and list all the failures:
+      """
+      Failures:
+
+        1) Client returns a successful response
+           Got 3 failures:
+
+           1.1) Failure/Error: expect(response.status).to eq(200)
+
+                  expected: 200
+                       got: 404
+
+                  (compared using ==)
+                # ./spec/use_metadata_spec.rb:7
+
+           1.2) Failure/Error: expect(response.headers).to include("Content-Type" => "application/json")
+                  expected {"Content-Type" => "text/plain"} to include {"Content-Type" => "application/json"}
+                  Diff:
+                  @@ -1,2 +1,2 @@
+                  -[{"Content-Type"=>"application/json"}]
+                  +"Content-Type" => "text/plain",
+                # ./spec/use_metadata_spec.rb:8
+
+           1.3) Failure/Error: expect(response.body).to eq('{"message":"Success"}')
+
+                  expected: "{\"message\":\"Success\"}"
+                       got: "Not Found"
+
+                  (compared using ==)
+                # ./spec/use_metadata_spec.rb:9
+      """
+
+  Scenario: Enable failure aggregation globally using `define_derived_metadata`
+    Given a file named "spec/enable_globally_spec.rb" with:
+      """ruby
+      require 'client'
+
+      RSpec.configure do |c|
+        c.define_derived_metadata do |meta|
+          meta[:aggregate_failures] = true
+        end
+      end
+
+      RSpec.describe Client do
+        it "returns a successful response" do
+          response = Client.make_request
+
+          expect(response.status).to eq(200)
+          expect(response.headers).to include("Content-Type" => "application/json")
+          expect(response.body).to eq('{"message":"Success"}')
+        end
+      end
+      """
+    When I run `rspec spec/enable_globally_spec.rb`
+    Then it should fail and list all the failures:
+      """
+      Failures:
+
+        1) Client returns a successful response
+           Got 3 failures:
+
+           1.1) Failure/Error: expect(response.status).to eq(200)
+
+                  expected: 200
+                       got: 404
+
+                  (compared using ==)
+                # ./spec/enable_globally_spec.rb:13
+
+           1.2) Failure/Error: expect(response.headers).to include("Content-Type" => "application/json")
+                  expected {"Content-Type" => "text/plain"} to include {"Content-Type" => "application/json"}
+                  Diff:
+                  @@ -1,2 +1,2 @@
+                  -[{"Content-Type"=>"application/json"}]
+                  +"Content-Type" => "text/plain",
+                # ./spec/enable_globally_spec.rb:14
+
+           1.3) Failure/Error: expect(response.body).to eq('{"message":"Success"}')
+
+                  expected: "{\"message\":\"Success\"}"
+                       got: "Not Found"
+
+                  (compared using ==)
+                # ./spec/enable_globally_spec.rb:15
+      """
+
+  Scenario: Nested failure aggregation works
+    Given a file named "spec/nested_failure_aggregation_spec.rb" with:
+      """ruby
+      require 'client'
+
+      RSpec.describe Client do
+        it "returns a successful response", :aggregate_failures do
+          response = Client.make_request
+
+          expect(response.status).to eq(200)
+
+          aggregate_failures "testing headers" do
+            expect(response.headers).to include("Content-Type" => "application/json")
+            expect(response.headers).to include("Content-Length" => "21")
+          end
+
+          expect(response.body).to eq('{"message":"Success"}')
+        end
+      end
+      """
+    When I run `rspec spec/nested_failure_aggregation_spec.rb`
+    Then it should fail and list all the failures:
+      """
+      Failures:
+
+        1) Client returns a successful response
+           Got 3 failures:
+
+           1.1) Failure/Error: expect(response.status).to eq(200)
+
+                  expected: 200
+                       got: 404
+
+                  (compared using ==)
+                # ./spec/nested_failure_aggregation_spec.rb:7
+
+           1.2) Got 2 failures from failure aggregation block "testing headers".
+                # ./spec/nested_failure_aggregation_spec.rb:9
+
+                1.2.1) Failure/Error: expect(response.headers).to include("Content-Type" => "application/json")
+                         expected {"Content-Type" => "text/plain"} to include {"Content-Type" => "application/json"}
+                         Diff:
+                         @@ -1,2 +1,2 @@
+                         -[{"Content-Type"=>"application/json"}]
+                         +"Content-Type" => "text/plain",
+                       # ./spec/nested_failure_aggregation_spec.rb:10
+
+                1.2.2) Failure/Error: expect(response.headers).to include("Content-Length" => "21")
+                         expected {"Content-Type" => "text/plain"} to include {"Content-Length" => "21"}
+                         Diff:
+                         @@ -1,2 +1,2 @@
+                         -[{"Content-Length"=>"21"}]
+                         +"Content-Type" => "text/plain",
+                       # ./spec/nested_failure_aggregation_spec.rb:11
+
+           1.3) Failure/Error: expect(response.body).to eq('{"message":"Success"}')
+
+                  expected: "{\"message\":\"Success\"}"
+                       got: "Not Found"
+
+                  (compared using ==)
+                # ./spec/nested_failure_aggregation_spec.rb:14
+      """
+
+  Scenario: Mock expectation failures are aggregated as well
+    Given a file named "spec/mock_expectation_failure_spec.rb" with:
+      """ruby
+      require 'client'
+
+      RSpec.describe "Aggregating Failures", :aggregate_failures do
+        it "has a normal expectation failure and a message expectation failure" do
+          client = double("Client")
+          expect(client).to receive(:put).with("updated data")
+          allow(client).to receive(:get).and_return(Response.new(404, {}, "Not Found"))
+
+          response = client.get
+          expect(response.status).to eq(200)
+        end
+      end
+      """
+    When I run `rspec spec/mock_expectation_failure_spec.rb`
+    Then it should fail and list all the failures:
+      """
+      Failures:
+
+        1) Aggregating Failures has a normal expectation failure and a message expectation failure
+           Got 2 failures:
+
+           1.1) Failure/Error: expect(response.status).to eq(200)
+
+                  expected: 200
+                       got: 404
+
+                  (compared using ==)
+                # ./spec/mock_expectation_failure_spec.rb:10
+
+           1.2) Failure/Error: expect(client).to receive(:put).with("updated data")
+                  (Double "Client").put("updated data")
+                      expected: 1 time with arguments: ("updated data")
+                      received: 0 times
+                # ./spec/mock_expectation_failure_spec.rb:6
+
+      """

features/step_definitions/additional_cli_steps.rb

@@ -187,4 +187,16 @@
   step "I run `#{cmd}`"
 end
 
+Then(/^it should fail and list all the failures:$/) do |string|
+  step %q{the exit status should not be 0}
+  expect(normalize_whitespace_and_backtraces(all_output)).to include(normalize_whitespace_and_backtraces(string))
+end
+
+module WhitespaceNormalization
+  def normalize_whitespace_and_backtraces(text)
+    text.lines.map { |line| line.sub(/\s+$/, '').sub(/:in .*$/, '') }.join
+  end
+end
+
+World(WhitespaceNormalization)
 World(FormatterSupport)

lib/rspec/core.rb

@@ -119,20 +119,13 @@ def self.configure
   #     end
   #
   def self.current_example
-    thread_local_metadata[:current_example]
+    RSpec::Support.thread_local_data[:current_example]
   end
 
   # Set the current example being executed.
   # @api private
   def self.current_example=(example)
-    thread_local_metadata[:current_example] = example
-  end
-
-  # @private
-  # A single thread local variable so we don't excessively pollute that
-  # namespace.
-  def self.thread_local_metadata
-    Thread.current[:_rspec] ||= { :shared_example_group_inclusions => [] }
+    RSpec::Support.thread_local_data[:current_example] = example
   end
 
   # @private

lib/rspec/core/configuration.rb

@@ -367,6 +367,8 @@ def initialize
         @libs = []
         @derived_metadata_blocks = FilterableItemRepository::QueryOptimized.new(:any?)
         @threadsafe = true
+
+        define_built_in_hooks
       end
 
       # @private
@@ -1732,6 +1734,12 @@ def value_for(key)
         @preferred_options.fetch(key) { yield }
       end
 
+      def define_built_in_hooks
+        around(:example, :aggregate_failures => true) do |ex|
+          aggregate_failures(nil, :from_around_hook => true, &ex)
+        end
+      end
+
       def assert_no_example_groups_defined(config_option)
         return unless RSpec.world.example_groups.any?
 

lib/rspec/core/example_group.rb

@@ -231,7 +231,7 @@ def self.define_example_method(name, extra_options={})
       # @see DSL#describe
       def self.define_example_group_method(name, metadata={})
         idempotently_define_singleton_method(name) do |*args, &example_group_block|
-          thread_data = RSpec.thread_local_metadata
+          thread_data = RSpec::Support.thread_local_data
           top_level   = self == ExampleGroup
 
           if top_level
@@ -705,17 +705,22 @@ def description
 
       # @private
       def self.current_backtrace
-        RSpec.thread_local_metadata[:shared_example_group_inclusions].reverse
+        shared_example_group_inclusions.reverse
       end
 
       # @private
       def self.with_frame(name, location)
-        current_stack = RSpec.thread_local_metadata[:shared_example_group_inclusions]
+        current_stack = shared_example_group_inclusions
         current_stack << new(name, location)
         yield
       ensure
         current_stack.pop
       end
+
+      # @private
+      def self.shared_example_group_inclusions
+        RSpec::Support.thread_local_data[:shared_example_group_inclusions] ||= []
+      end
     end
   end