Merge with release/0.8.0

CHANGELOG.md

@@ -1,18 +1,40 @@
 ## [Unreleased][unreleased]
 
+## [0.8.0] - 2016/04/13
+
+This release includes support for PostgreSQL as Kong's primary datastore!
+
 ### Breaking changes
 
 - Remove support for the long deprecated `/consumers/:consumer/keyauth/` and `/consumers/:consumer/basicauth/` routes (deprecated in `0.5.0`). The new routes (available since `0.5.0` too) use the real name of the plugin: `/consumers/:consumer/key-auth` and `/consumers/:consumer/basic-auth`.
 
 ### Added
 
-- Support for PostgreSQL as Kong's primary datastore. [#331](https://github.com/Mashape/kong/issues/331) [#1054](https://github.com/Mashape/kong/issues/1054)
+- Support for PostgreSQL 9.4+ as Kong's primary datastore. [#331](https://github.com/Mashape/kong/issues/331) [#1054](https://github.com/Mashape/kong/issues/1054)
 - Configurable Cassandra reading/writing consistency. [#1026](https://github.com/Mashape/kong/pull/1026)
-- Admin API: oncluding pending and running timers count in the response to `/`. [#992](https://github.com/Mashape/kong/pull/992)
+- Admin API: including pending and running timers count in the response to `/`. [#992](https://github.com/Mashape/kong/pull/992)
 - Plugins
   - **New correlation-id plugin**: assign unique identifiers to the requests processed by Kong. Courtesy of [@opyate](https://github.com/opyate). [#1094](https://github.com/Mashape/kong/pull/1094)
-  - JWT
-    - Add support for RS256 signed tokens thanks to [@kdstew](https://github.com/kdstew)! [#1053](https://github.com/Mashape/kong/pull/1053)
+  - LDAP: add support for LDAP authentication. [#1133](https://github.com/Mashape/kong/pull/1133)
+  - StatsD: add support for StatsD logging. [#1142](https://github.com/Mashape/kong/pull/1142)
+  - JWT: add support for RS256 signed tokens thanks to [@kdstew](https://github.com/kdstew)! [#1053](https://github.com/Mashape/kong/pull/1053)
+  - ACL: appends `X-Consumer-Groups` to the request, so the upstream service can check what groups the consumer belongs to. [#1154](https://github.com/Mashape/kong/pull/1154)
+  - Galileo (mashape-analytics): increase batch sending timeout to 30s. [#1091](https://github.com/Mashape/kong/pull/1091)
+- Added `ttl_on_failure` option in the cluster configuration, to configure the TTL of failed nodes. [#1125](https://github.com/Mashape/kong/pull/1125)
+
+### Fixed
+
+- Introduce a new `port` option when connecting to your Cassandra cluster instead of using the CQL default (9042). [#1139](https://github.com/Mashape/kong/issues/1139)
+- Plugins
+  - Request/Response Transformer: add missing migrations for upgrades from ` <= 0.5.x`. [#1064](https://github.com/Mashape/kong/issues/1064)
+  - OAuth2
+    - Error responses comply to RFC 6749. [#1017](https://github.com/Mashape/kong/issues/1017)
+    - Handle multipart requests. [#1067](https://github.com/Mashape/kong/issues/1067)
+    - Make access_tokens correctly expire. [#1089](https://github.com/Mashape/kong/issues/1089)
+
+> **internal**
+> - replace globals with singleton pattern thanks to [@mars](https://github.com/mars).
+> - fixed resolution mismatches when using deep paths in the path resolver thanks to [siddharthkchatterjee](https://github.com/siddharthkchatterjee)
 
 ## [0.7.0] - 2016/02/24
 
@@ -521,7 +543,8 @@ First version running with Cassandra.
 - CLI `bin/kong` script.
 - Database migrations (using `db.lua`).
 
-[unreleased]: https://github.com/mashape/kong/compare/0.7.0...next
+[unreleased]: https://github.com/mashape/kong/compare/0.8.0...next
+[0.8.0]: https://github.com/mashape/kong/compare/0.7.0...0.8.0
 [0.7.0]: https://github.com/mashape/kong/compare/0.6.1...0.7.0
 [0.6.1]: https://github.com/mashape/kong/compare/0.6.0...0.6.1
 [0.6.0]: https://github.com/mashape/kong/compare/0.5.4...0.6.0

Makefile

@@ -22,8 +22,8 @@ dev: install
       echo $$rock already installed, skipping ; \
     fi \
 	done;
-	bin/kong config -c kong.yml -e TEST
-	bin/kong config -c kong.yml -e DEVELOPMENT
+	bin/kong config -c kong.yml -e TEST -s TEST
+	bin/kong config -c kong.yml -e DEVELOPMENT -s DEVELOPMENT
 	bin/kong migrations -c $(DEVELOPMENT_CONF) up
 
 clean:

UPGRADE.md

@@ -22,6 +22,46 @@ $ kong reload [-c configuration_file]
 
 **Reminder**: `kong reload` leverages the Nginx `reload` signal and seamlessly starts new workers taking over the old ones until they all have been terminated. This will guarantee you no drop in your current incoming traffic.
 
+## Upgrade to `0.8.x`
+
+No important breaking changes for this release, just be careful to not use the long deprecated routes `/consumers/:consumer/keyauth/` and `/consumers/:consumer/basicauth/` as instructed in the Changelog. As always, also make sure to check the configuration file for new properties (this release allows you to configure the read/write consistency of Cassandra).
+
+Let's talk about **PostgreSQL**. To use it instead of Cassandra, follow those steps:
+
+* Get your hands on a 9.4+ server (being compatible with Postgres 9.4 allows you to use [Amazon RDS](https://aws.amazon.com/rds/))
+* Create a database, (maybe a user too?), let's say `kong`
+* Update your Kong configuration:
+
+```yaml
+# as always, be careful about your YAML formatting
+database: postgres
+postgres:
+  host: "127.0.0.1"
+  port: 5432
+  user: kong
+  password: kong
+  database: kong
+```
+
+As usual, migrations should run from kong start, but as a reminder and just in case, here are some tips:
+
+Reset the database with (careful, you'll lose all data):
+```
+$ kong migrations reset --config kong.yml
+```
+
+Run the migrations manually with:
+```
+$ kong migrations up --config kong.yml
+```
+
+If needed, list your migrations for debug purposes with:
+```
+$ kong migrations list --config kong.yml
+```
+
+**Note**: This release does not provide a mean to migrate from Cassandra to PostgreSQL. Additionally, we recommend that you **do not** use `kong reload` if you switch your cluster from Cassandra to PostgreSQL. Instead, we recommend that you migrate by spawning a new cluster and gradually redirect your traffic before decomissioning your old nodes.
+
 ## Upgrade to `0.7.x`
 
 If you are running a source installation, you will need to upgrade OpenResty to its `1.9.7.*` version. The good news is that this family of releases does not need to patch the NGINX core anymore to enable SSL support. If you install Kong from one of the distribution packages, they already include the appropriate OpenResty, simply download and install the appropriate package for your platform.

kong-0.7.0-0.rockspec → kong-0.8.0-0.rockspec

@@ -1,9 +1,9 @@
 package = "kong"
-version = "0.7.0-0"
+version = "0.8.0-0"
 supported_platforms = {"linux", "macosx"}
 source = {
   url = "git://github.com/Mashape/kong",
-  tag = "0.7.0"
+  tag = "0.8.0"
 }
 description = {
   summary = "Kong is a scalable and customizable API Management Layer built on top of Nginx.",
@@ -20,7 +20,7 @@ dependencies = {
   "yaml ~> 1.1.2-1",
   "lapis ~> 1.3.1-1",
   "stringy ~> 0.4-1",
-  "lua-cassandra ~> 0.5.0",
+  "lua-cassandra ~> 0.5.1",
   "pgmoon ~> 1.4.0",
   "multipart ~> 0.3-2",
   "lua-path ~> 0.2.3-1",
@@ -34,7 +34,8 @@ dependencies = {
   "lrexlib-pcre ~> 2.7.2-1",
   "lua-llthreads2 ~> 0.1.3-1",
   "luacrypto >= 0.3.2-1",
-  "luasyslog >= 1.0.0-2"
+  "luasyslog >= 1.0.0-2",
+  "lua_pack ~> 1.0.4-0"
 }
 build = {
   type = "builtin",
@@ -242,6 +243,12 @@ build = {
     ["kong.plugins.hmac-auth.api"] = "kong/plugins/hmac-auth/api.lua",
     ["kong.plugins.hmac-auth.daos"] = "kong/plugins/hmac-auth/daos.lua",
 
+    ["kong.plugins.ldap-auth.handler"] = "kong/plugins/ldap-auth/handler.lua",
+    ["kong.plugins.ldap-auth.access"] = "kong/plugins/ldap-auth/access.lua",
+    ["kong.plugins.ldap-auth.schema"] = "kong/plugins/ldap-auth/schema.lua",
+    ["kong.plugins.ldap-auth.ldap"] = "kong/plugins/ldap-auth/ldap.lua",
+    ["kong.plugins.ldap-auth.asn1"] = "kong/plugins/ldap-auth/asn1.lua",
+
     ["kong.plugins.syslog.handler"] = "kong/plugins/syslog/handler.lua",
     ["kong.plugins.syslog.schema"] = "kong/plugins/syslog/schema.lua",
 
@@ -250,7 +257,11 @@ build = {
 
     ["kong.plugins.datadog.handler"] = "kong/plugins/datadog/handler.lua",
     ["kong.plugins.datadog.schema"] = "kong/plugins/datadog/schema.lua",
-    ["kong.plugins.datadog.statsd_logger"] = "kong/plugins/datadog/statsd_logger.lua"
+    ["kong.plugins.datadog.statsd_logger"] = "kong/plugins/datadog/statsd_logger.lua",
+
+    ["kong.plugins.statsd.handler"] = "kong/plugins/statsd/handler.lua",
+    ["kong.plugins.statsd.schema"] = "kong/plugins/statsd/schema.lua",
+    ["kong.plugins.statsd.statsd_logger"] = "kong/plugins/statsd/statsd_logger.lua"
   },
   install = {
     conf = { "kong.yml" },

kong.yml

@@ -87,17 +87,30 @@
   ## Key for encrypting network traffic within Kong. Must be a base64-encoded 16-byte key.
   # encrypt: "foo"
 
+  ######
+  ## The TTL (time to live), in seconds, of a node in the cluster when it stops sending healthcheck pings, maybe
+  ## because of a failure. If the node is not able to send a new healthcheck before the expiration, then new nodes
+  ## in the cluster will stop attempting to connect to it on startup.
+  # ttl_on_failure: 3600
+
 ######
 ## Specify which database to use. Only "cassandra" is currently available.
 # database: cassandra
 
 ######
 ## PostgreSQL configuration
 # postgres:
-#    host: "127.0.0.1"
-#    port: 5432
-#    user: kong
-#    database: kong
+  # host: "127.0.0.1"
+  # port: 5432
+
+  ######
+  ## Name of the database used by Kong. Will be created if it does not exist.
+  # database: kong
+
+  #####
+  ## User authentication settings
+  # user: ""
+  # password: ""
 
 ######
 ## Cassandra configuration (keyspace, authentication, client-to-node encryption)
@@ -107,6 +120,10 @@
   # contact_points:
   #   - "127.0.0.1:9042"
 
+  ## Port on which your cluster's peers (other than your contact_points)
+  ## are listening on.
+  # port: 9042
+
   ######
   ## Name of the keyspace used by Kong. Will be created if it does not exist.
   # keyspace: kong

kong/cli/services/serf.lua

@@ -160,7 +160,7 @@ function Serf:_add_node()
   local _, err = self._dao_factory.nodes:insert({
     name = name,
     cluster_listening_address = stringy.strip(addr)
-  }, {ttl = 3600})
+  }, {ttl = self._configuration.cluster.ttl_on_failure})
   if err then
     return false, err
   end

kong/constants.lua

@@ -13,7 +13,7 @@ return {
     "http-log", "key-auth", "hmac-auth", "basic-auth", "ip-restriction",
     "mashape-analytics", "request-transformer", "response-transformer",
     "request-size-limiting", "rate-limiting", "response-ratelimiting", "syslog",
-    "loggly", "datadog", "runscope"
+    "loggly", "datadog", "runscope", "ldap-auth", "statsd"
   },
   -- Non standard headers, specific to Kong
   HEADERS = {
@@ -23,7 +23,10 @@ return {
     CONSUMER_ID = "X-Consumer-ID",
     CONSUMER_CUSTOM_ID = "X-Consumer-Custom-ID",
     CONSUMER_USERNAME = "X-Consumer-Username",
-    CREDENTIAL_USERNAME = "X-Credential-Username"
+    CREDENTIAL_USERNAME = "X-Credential-Username",
+    RATELIMIT_LIMIT = "X-RateLimit-Limit",
+    RATELIMIT_REMAINING = "X-RateLimit-Remaining",
+    CONSUMER_GROUPS = "X-Consumer-Groups"
   },
   RATELIMIT = {
     PERIODS = {

kong/core/cluster.lua

@@ -79,7 +79,7 @@ local function send_keepalive(premature)
       ngx.log(ngx.ERR, tostring(err))
     elseif #nodes == 1 then
       local node = nodes[1]
-      local _, err = singletons.dao.nodes:update(node, node)
+      local _, err = singletons.dao.nodes:update(node, node, {ttl=singletons.configuration.cluster.ttl_on_failure})
       if err then
         ngx.log(ngx.ERR, tostring(err))
       end

kong/core/reports.lua

@@ -1,6 +1,7 @@
 local syslog = require "kong.tools.syslog"
 local cache = require "kong.tools.database_cache"
 local utils = require "kong.tools.utils"
+local singletons = require "kong.singletons"
 local unique_str = utils.random_string()
 local enabled = false
 
@@ -29,7 +30,7 @@ local function send_ping(premature)
   if elapsed and elapsed == 0 then
     local reqs = cache.get(cache.requests_key())
     if not reqs then reqs = 0 end
-    syslog.log({signal = "ping", requests = reqs, unique_id = unique_str})
+    syslog.log({signal = "ping", requests = reqs, unique_id = unique_str, database = singletons.configuration.database})
     cache.incr(cache.requests_key(), -reqs) -- Reset counter
   end
   create_timer(INTERVAL, send_ping)

kong/dao/cassandra_db.lua

@@ -26,6 +26,9 @@ function CassandraDB:new(options)
     prepared_shm = "cassandra_prepared",
     contact_points = options.contact_points,
     keyspace = options.keyspace,
+    protocol_options = {
+      default_port = options.port
+    },
     query_options = {
       prepare = true
     },
@@ -298,7 +301,7 @@ function CassandraDB:count(table_name, tbl, schema)
   end
 end
 
-function CassandraDB:update(table_name, schema, constraints, filter_keys, values, nils, full, options)
+function CassandraDB:update(table_name, schema, constraints, filter_keys, values, nils, full, model, options)
   -- must check unique constaints manually too
   local err = check_unique_constraints(self, table_name, constraints, values, filter_keys, true)
   if err then
@@ -309,6 +312,30 @@ function CassandraDB:update(table_name, schema, constraints, filter_keys, values
     return nil, err
   end
 
+  -- Cassandra TTL on update is per-column and not per-row, and TTLs cannot be updated on primary keys.
+  -- Not only that, but TTL on other rows can only be incremented, and not decremented. Because of all
+  -- of these limitations, the only way to make this happen is to do an upsert operation.
+  -- This implementation can be changed once Cassandra closes this issue: https://issues.apache.org/jira/browse/CASSANDRA-9312
+  if options and options.ttl then
+    if schema.primary_key and #schema.primary_key == 1 and filter_keys[schema.primary_key[1]] then
+      local row, err = self:find(table_name, schema, filter_keys)
+      if err then
+        return nil, err
+      elseif row then
+        for k, v in pairs(row) do
+          if not values[k] then
+            model[k] = v -- Populate the model to be used later for the insert
+          end
+        end
+
+        -- Insert without any contraint check, since the check has already been executed
+        return self:insert(table_name, schema, model, {unique={}, foreign={}}, options)
+      end
+    else
+      return nil, "Cannot update TTL on entities that have more than one primary_key"
+    end
+  end
+
   local sets, args, where = {}, {}
   for col, value in pairs(values) do
     local field = schema.fields[col]

kong/dao/dao.lua

@@ -265,7 +265,7 @@ function DAO:update(tbl, filter_keys, options)
     fix(old, values, self.schema)
   end
 
-  local res, err = self.db:update(self.table, self.schema, self.constraints, primary_keys, values, nils, full_update, options)
+  local res, err = self.db:update(self.table, self.schema, self.constraints, primary_keys, values, nils, full_update, model, options)
   if err then
     return nil, err
   elseif res then

kong/dao/migrations/postgres.lua

@@ -1,12 +1,14 @@
 return {
   {
     name = "2015-01-12-175310_skeleton",
-    up = [[
-      CREATE TABLE IF NOT EXISTS schema_migrations(
-        id text PRIMARY KEY,
-        migrations varchar(100)[]
-      );
-    ]],
+    up = function(db, properties)
+      return db:queries [[
+        CREATE TABLE IF NOT EXISTS schema_migrations(
+          id text PRIMARY KEY,
+          migrations varchar(100)[]
+        );
+      ]]
+    end,
     down = [[
       DROP TABLE schema_migrations;
     ]]