Merge branch 'release/8.8.1'

COPYING.adoc

@@ -1,14 +1,16 @@
-image:https://i.creativecommons.org/l/by-sa/4.0/88x31.png[CC BY-SA] Synacor, Inc., 2016
+include::settings.adoc[]
 
-(C) 2016-2017 by Synacor, Inc.
+image:https://i.creativecommons.org/l/by-sa/4.0/88x31.png[CC BY-SA] Synacor, Inc., {copyright-year}
+
+(C) {copyright-year} by Synacor, Inc.
 
 This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
 International License unless another license agreement between you and
 Synacor, Inc. provides otherwise. To view a copy of this license, visit
 https://creativecommons.org/licenses/by-sa/4.0 or send a letter to Creative
 Commons, PO Box 1866, Mountain View, CA 94042, USA.
 
-Synacor, Inc., 2016 +
+Synacor, Inc., {copyright-year}
 40 La Riviere Drive, Suite 300 +
 Buffalo, New York 14202
 

LICENSE.adoc

@@ -1,15 +1,16 @@
+include::settings.adoc[]
 
-image:https://i.creativecommons.org/l/by-sa/4.0/88x31.png[CC BY-SA] Synacor, Inc., 2016
+image:https://i.creativecommons.org/l/by-sa/4.0/88x31.png[CC BY-SA] Synacor, Inc., {copyright-year}
 
-(C) 2016 by Synacor, Inc.
+(C) {copyright-year} by Synacor, Inc.
 
 This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
 International License unless another license agreement between you and
 Synacor, Inc. provides otherwise. To view a copy of this license, visit
 https://creativecommons.org/licenses/by-sa/4.0 or send a letter to Creative
 Commons, PO Box 1866, Mountain View, CA 94042, USA.
 
-Synacor, Inc., 2016 +
+Synacor, Inc., {copyright-year} +
 40 La Riviere Drive, Suite 300 +
 Buffalo, New York 14202
 

backuprestore.adoc

@@ -1525,3 +1525,29 @@ zmplayredo --fromTime "2008/10/17 08:00:00:000"
 All data is brought forward to the current time and the standby site is
 set up and running. Data from 8:30:00 to 8:35:00 is lost but that is
 expected when the restore process is being carried out.
+
+
+== Notes on Ephemeral Data
+
+As of ZCS 8.8, ephemeral data is not backed up as part of the backup process.
+Since auth tokens are ephemeral attributes, the implication is that clients
+accessing accounts restored after deletion will need to re-authenticate;
+auth tokens generated prior to the backup will no longer work.
+
+The Jira ticket ZMS-614 has been opened to track the potential future development
+of ephemeral data backups.
+
+=== Backups with SSDB Backend
+
+If SSDB is used as the ephemeral backend, a backup will not include any ephemeral
+attributes. All authentication data is lost, including the last logon timestamp.
+As a result, the "Last Login" column in Accounts section of the admin console
+will display "Never logged in" for restored accounts. If a specific need for this
+data arises, all authentication events are logged in the audit.log file. Customers
+considering migrating to an SSDB backend should be aware of this limitation.
+
+=== Backups with LDAP backend
+
+If the ephemeral backend is LDAP, a backup will not include auth tokens or CSRF
+tokens, but it will include the last logon timestamp. Upon account restore,
+the appropriate "Last Login" value in the admin console will be restored.

ephemeraldata.adoc

@@ -82,12 +82,20 @@ Ephemeral data migration is a one-way process. The `zmmigrateattrs` script does
 
 === Changes to zmprov
 
-Due to changes in the way multi-valued ephemeral data is stored, the attributes `zimbraAuthTokens` and `zimbraCsrfTokenData` are no longer returned as part of the `zmprov ga <account>` response. The value of `zimbraLastLogonTimestamp` is returned as before.
+Due to changes in the way multi-valued ephemeral data is stored, the attributes `zimbraAuthTokens` and `zimbraCsrfTokenData` are no longer returned as part of the `zmprov ga <account>` response. The value of `zimbraLastLogonTimestamp` is returned as before, although
+only if the -l flag is not used, as adding the -l flag will restrict the server to accessing attributes in LDAP only.
+It is still possible to modify these attributes using the `zmprov ma <account>` command, regardless of the ephemeral backend. In order to do this, the provided attribute value must match its LDAP format: `tokenId|expiration|serverVersion` for auth tokens; `data:crumb:expiration` for CSRF tokens.
 
 === Migration In-Progress Flag
 
 The `zmmigrateattrs` script sets a flag in `SSDB` at the beginning of the migration process and unsets it when migration completes. When this flag is set, the server will look for an account's ephemeral data in both the `SSDB` and LDAP backends, as the data may reside in either one. In the event of an error during migration, the flag will remain set until migration is re-run successfully. For testing or debugging purposes, this flag can be set or unset via command line using the `--set-flag` and `--unset-flag` options of `zmmigrateattrs`.
 
 === Migration CSV Output
 
-Each run of `zmmigrateattrs` generates a CSV file in `/opt/zimbra/data/tmp/` directory. The file contains migration info for every migrated account. If any migrations fail, a cutdown CSV file report detailing only the errors is also created in the same directory. The name(s) of the file(s) are logged at the end of the run.
+Each run of `zmmigrateattrs` generates a CSV file in `/opt/zimbra/data/tmp/` directory. The file contains migration info for every migrated account. If any migrations fail, a cutdown CSV file report detailing only the errors is also created in the same directory. The name(s) of the file(s) are logged at the end of the run.
+
+=== Account Deletion Behavior
+
+Ephemeral data deletion behavior differs slightly between SSDB and LDAP backends. With SSDB as the backend, account deletion results in
+the `zimbraLastLogonTimestamp` attribute being explicitly deleted from SSDB. `zimbraAuthTokens` and `zimbraCsrfTokenData`, however, are
+left to be expired by SSDB when the token lifetimes are reached (default of 2 days). Conversely, ephemeral data in LDAP is wiped immediately as part of the account deletion process.

imap_upstream.adoc

@@ -0,0 +1,186 @@
+= Zimbra Upstream IMAP Configuration
+
+:toc:
+
+Starting with version 8.8.1 of the {product-name} Suite the following
+IMAP related configuration options are available:
+
+* zimbraReverseProxyUpstreamImapServers
+
+  The pool of servers that are available to the proxy for handling IMAP sessions.
+  When this value is empty the Nginx Lookup Extension will select the mailbox server that hosts the account.
+
+*  zimbraImapLoadBalancingAlgorithm
+
+  Configures the load-balancing algorithm used to select an IMAP server from the pool of available zimbraReverseProxyUpstreamImapServers.
+
+The available balancing algorithms are as follows:
+
+* ClientIpHash
+
+  Selects an IMAP server server based on a hash of the client IP address.
+  The hashing algorithm used is a simple modulus of the client's ip address as an integer with the number of
+   configured IMAP servers. This will always return the same 'index' for the same client ip address.
+
+   Note: Extra configuration is necessary if a NAT proxy is being utilized in the environment.
+
+* custom
+
+  Selects a registered load balancing algorithm with the following format:
+
+      custom:{handler-algorithm} [arg1 arg2 ...]
+
+[NOTE]
+If the custom load balancing class can't be located the default
+ClientIpHash will be returned.
+
+= IMAPD-related Configuration & Files
+
+This section summarizes all of the configuration options related to
+the `zimbra-imapd` as well as all other files that are installed for
+the new service.
+
+== Global Configuration Settings
+
+* These control the embedded IMAP(S) servers that are run inside of
+  `mailboxd`.
+** `zimbraImapServerEnabled`
+** `zimbraImapBindPort`
+** `zimbraImapSSLServerEnabled`
+** `zimbraImapSSLBindPort`
+* These control the IMAP(S) servers that are run by `zimbra-imapd`.
+** `zimbraRemoteImapServerEnabled`
+** `zimbraRemoteImapBindPort`
+** `zimbraRemoteImapSSLServerEnabled`
+** `zimbraRemoteImapSSLBindPort`
+* This maintains the list of available IMAP(S) servers:
+** `zimbraReverseProxyUpstreamImapServers`
+
+== Server Configuration Settions
+
+If the `zimbra-imapd` service is enabled on a given server, it will be added to
+the multi-valued attribute `zimbraServiceEnabled`.
+
+For example, on an IMAP-only server:
+
+[EXAMPLE]
+----
+$ zmprov gs `zmhostname` zimbraServiceEnabled
+# name HOST.DOMAIN
+zimbraServiceEnabled: imapd
+zimbraServiceEnabled: stats
+----
+
+== Files that are installed for zimbra-imapd
+
+These are the files that are specifically installed by the
+zimbra-imapd DEB/RPM:
+
+* `/opt/zimbra/bin/zmimapdctl`
+* `/opt/zimbra/conf/imapd.log4j.properties`
+* `/opt/zimbra/lib/jars/oauth-1.4.jar`
+
+In addition, the service makes use of other jar files that are already
+installed into `/opt/zimbra/lib/jars` via the `zimbra-core` package.
+
+== Files that are created during installation
+
+`zmcertmgr` is invoked by `zmsetup.pl` to create the following
+SSL-related files:
+
+* `imapd.crt`
+* `imapd.key`
+* `imapd.keystore`
+
+== Log files
+
+The following log files (in `/opt/zimbra/log`) are created by the
+`zimbra-imapd` service via configuration contained in
+`/opt/zimbra/conf/imapd.log4j.properties`.
+
+* `imapd-audit.log`
+* `imapd.log`
+
+
+== Process Files
+
+The following files are created directly by the `zmimapdctl` script:
+
+* `imapd.out`
+* `imapd.pid`
+
+== Other Files
+
+Activity statistics are collected in `/opt/zimbra/zmstat/imapd.csv`.
+
+
+= Migrating to an imapd pool
+
+When migrating to a dedicated pool of IMAP servers, several steps must
+be followed to ensure a smooth migration process. Failure to follow
+these steps may result in service interruptions in the form of dropped
+connections of existing IMAP clients.
+
+== Memcached considerations
+
+Prior to changing any configuration, it should be decided how existing
+routes in `memcached` will be handled. By default, routes are cached in
+`memcached` for 1 day, as specified by the
+`zimbraMemcachedClientExpirySeconds` LDAP attribute. If untouched, it
+will take this long for the cached routes to expire and for the lookup
+extension to send IMAP traffic to the newly provisioned imapd
+servers. There are two things that can be done to change this
+behavior:
+
+1. Flush `memcached` after the imapd pool configuration is
+complete. This will cause all existing IMAP sessions to be restarted
+on the new servers.
+2. One day prior to switching to the imapd pool, modify the value
+`zimbraMemcachedClientExpirySeconds` of to a shorter interval, such as
+30 minutes, and reload the `memcached` configuration with the command
+`zmprov rmcc all`. Doing so will allow embedded IMAP to be turned off
+within a shorter timeframe without risking service interruptions.
+
+== Migration steps
+
+1. Set up the pool of imapd servers. This pool should be sized
+appropriately to accommodate expected IMAP traffic levels. It is
+recommended that you test each server to ensure that it is functioning
+correctly; this can be done via command line with openssl, or by
+configuring an existing IMAP client to point directly at the
+server. These servers should not be listed in the
+`zimbraReverseProxyUpstreamImapServers` attribute yet.
+2. (optional) Specify a load-balancing algorithm using the
+`zimbraImapLoadBalancingAlgorithm` attribute. The default is
+`ClientIPHash`.  If custom load-balancing algorithms are written,
+they can be specified as well.
+3. Add the imapd servers to `zimbraReverseProxyUpstreamImapServers`:
++
+----
+zmprov mcf +zimbraReverseProxyUpstreamImapServers <server1> \
+  +zimbraReverseProxyUpstreamImapServers <server2> \
+ ...
+----
++
+4. Flush the config cache on lookup servers: `zmprov -a fc config`
+5. If `zimbraMemcachedClientExpirySeconds` was decreased prior to this
+change, wait the corresponding amount of time for the existing routes
+to expire. This will allow routes in memcached to switch over to the
+new routes. Otherwise, flush memcached manually. It is recommended
+that this be done outside peak hours if possible.
+6. Set `zimbraImapServerEnabled` And `zimbraImapSSLServerEnabled` to
+`FALSE` at the global config level:
++
+----
+zmprov mcf zimbraImapServerEnabled FALSE
+zmprov mcf zimbraImapSSLServerEnabled FALSE
+----
+[NOTE] If these settings were overridden at the at the server level,
+you will need to modify them on the mailbox servers via `zmprov ms <server>..`
++
+7. Reset `zimbraMemcachedClientExpirySeconds` to the original value if
+necessary.
+
+
+
+

imapd.adoc

@@ -0,0 +1,15 @@
+= Zimbra IMAPD Server
+:toc:
+
+The Zimbra IMAPD server is an optionally installed dedicated server that handles IMAP(S) traffic.
+It can be installed on the same node as a *mailbox server* or on a separate node.
+When installed on the same node as a *mailbox server* the mailbox's internal IMAPD process may be disabled.  By default it is left enabled.
+
+In installations which have very heavy IMAP usage it is the recommended practice to install IMAPD on separate nodes from the *mailbox* processes to allow for horizontal scaling of IMAPD resources independently from the mailbox nodes.
+
+Whenever a new IMAPD node is installed in a ZCS cluster that is not a mailboxd it MUST BE added to the '''zimbraHttpThrottleSafeIPs''' configuration item or the DosFilter will
+throttle the new server.  Failure to do so will cause SOAP traffic from the Remote IMAPD node to be throttled leading to unexpected communication errors.
+
+Alternatively the DoSFilter can be effectively disabled by adding the ip address subnet that the new machine lives on to the '''zimbraHttpThrottleSafeIPs''' configuration item.
+
+Lastly, when an IMAPD node is added to a ZCS cluster, the globalconfig LDAP cache must be flushed on all ZCS servers listed in '''zimbraReverseProxyAvailableLookupTargets'''. This is necessary to ensure that the new node is added to '''zimbraReverseProxyUpstreamImapServers''' attribute; without this step, the lookup extension on these servers will not be aware of the newly-provisioned IMAP node. To do this, run the command `zmprov flushCache -a config`. To verify that this has taken effect, make sure that the new IMAPD node is listed in the output of `zmprov gacf zimbraReverseProxyUpstreamImapServers`, when run from a lookup target server.

index.adoc

@@ -1,13 +1,4 @@
-:document-title: Administrator Guide
-:product-name: Zimbra Collaboration
-:product-abbrev: ZCS
-= {product-name} {document-title}
-v8.8.0, July, 2017
-:icons: font
-:title-logo-image: images/zimbra-logo.jpg
-:showlinks:
-:source-highlighter: coderay
-:toc: left
+include::settings.adoc[]
 
 :leveloffset: +1
 
@@ -33,6 +24,20 @@ include::mailboxserver.adoc[]
 
 <<<
 
+include::imapd.adoc[]
+
+<<<
+
+:leveloffset: +1
+include::imap_upstream.adoc[]
+
+<<<
+
+include::remote_imap.adoc[]
+
+<<<
+
+:leveloffset: -1
 include::ldap.adoc[]
 
 <<<

legal.adoc

@@ -1,17 +1,19 @@
+include::settings.adoc[]
+
 [colophon]
 = License
 
-image:https://i.creativecommons.org/l/by-sa/4.0/88x31.png[CC BY-SA] Synacor, Inc., 2016
+image:https://i.creativecommons.org/l/by-sa/4.0/88x31.png[CC BY-SA] Synacor, Inc., {copyright-year}
 
-(C) 2016 by Synacor, Inc. {product-name} {document-title}
+(C) {copyright-year} by Synacor, Inc. {product-name} {document-title}
 
 This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
 International License unless another license agreement between you and
 Synacor, Inc. provides otherwise. To view a copy of this license, visit
 https://creativecommons.org/licenses/by-sa/4.0 or send a letter to Creative
 Commons, PO Box 1866, Mountain View, CA 94042, USA.
 
-Synacor, Inc., 2016 +
+Synacor, Inc., {copyright-year} +
 40 La Riviere Drive, Suite 300 +
 Buffalo, New York 14202