Merge remote-tracking branch 'IQSS/develop' into QDR/6886-external_st…

…atus_labels

conf/docker-aio/0prep_deps.sh

@@ -4,10 +4,10 @@ if [ ! -d dv/deps ]; then
 fi
 wdir=`pwd`
 
-if [ ! -e dv/deps/payara-5.2021.4.zip ]; then
+if [ ! -e dv/deps/payara-5.2021.5.zip ]; then
 	echo "payara dependency prep"
 	# no more fiddly patching :)
-	wget https://s3-eu-west-1.amazonaws.com/payara.fish/Payara+Downloads/5.2021.4/payara-5.2021.4.zip  -O dv/deps/payara-5.2021.4.zip
+	wget https://s3-eu-west-1.amazonaws.com/payara.fish/Payara+Downloads/5.2021.5/payara-5.2021.5.zip  -O dv/deps/payara-5.2021.5.zip
 fi
 
 if [ ! -e dv/deps/solr-8.8.1dv.tgz ]; then

conf/docker-aio/c8.dockerfile

@@ -24,7 +24,7 @@ COPY disableipv6.conf /etc/sysctl.d/
 RUN rm /etc/httpd/conf/*
 COPY httpd.conf /etc/httpd/conf 
 RUN cd /opt ; tar zxf /tmp/dv/deps/solr-8.8.1dv.tgz 
-RUN cd /opt ; unzip /tmp/dv/deps/payara-5.2021.4.zip ; ln -s /opt/payara5 /opt/glassfish4
+RUN cd /opt ; unzip /tmp/dv/deps/payara-5.2021.5.zip ; ln -s /opt/payara5 /opt/glassfish4
 
 # this copy of domain.xml is the result of running `asadmin set server.monitoring-service.module-monitoring-levels.jvm=LOW` on a default glassfish installation (aka - enable the glassfish REST monitir endpoint for the jvm`
 # this dies under Java 11, do we keep it?

doc/release-notes/5.6-release-notes.md

@@ -0,0 +1,130 @@
+# Dataverse Software 5.6
+
+This release brings new features, enhancements, and bug fixes to the Dataverse Software. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project.
+
+## Release Highlights
+
+### Anonymized Access in Support of Double Blind Review
+
+Dataverse installations can select whether or not to allow users to create anonymized private URLs and can control which specific identifying fields are anonymized. If this is enabled, users can create private URLs that do not expose identifying information about dataset depositors, allowing for double blind reviews of datasets in the Dataverse installation.
+
+### Guestbook Responses API
+
+A new API to retrieve Guestbook responses has been added. This makes it easier to retrieve the records for large guestbooks and also makes it easier to integrate with external systems.
+
+### Dataset Semantic API (Experimental)
+
+Dataset metadata can be retrieved, set, and updated using a new, flatter JSON-LD format - following the format of an OAI-ORE export (RDA-conformant Bags), allowing for easier transfer of metadata to/from other systems (i.e. without needing to know Dataverse's metadata block and field storage architecture). This new API also allows for the update of terms metadata (#5899).
+
+This development was supported by the [Research Data Alliance](https://rd-alliance.org), DANS, and Sciences PO and follows the recommendations from the [Research Data Repository Interoperability Working Group](http://dx.doi.org/10.15497/RDA00025).
+
+### Dataset Migration API (Experimental)
+
+Datasets can now imported following the format of an OAI-ORE export (RDA-conformant Bags), allowing for easier migration from one Dataverse installation to another, and migration from other systems. This experimental, superuser only, endpoint also allows keeping the existing persistent identifier (where the authority and shoulder match those for which the software is configured) and publication dates.
+
+This development was supported by DANS and the [Research Data Alliance](https://rd-alliance.org) and follows the recommendations from the [Research Data Repository Interoperability Working Group](http://dx.doi.org/10.15497/RDA00025).
+
+### Direct Upload API Now Available for Adding Multiple Files Metadata to the Dataset
+
+Users can now add metadata of multiple files to the dataset once the files exists in the s3 bucket using the direct upload API. This makes direct uploads more efficient and reduces server load by only updating the dataset once instead of once per file. For more information, see the [Direct DataFile Upload/Replace API section](https://guides.dataverse.org/en/5.6/developers/s3-direct-upload-api.html) of the Dataverse Software Guides.
+
+## Major Use Cases
+
+Newly-supported major use cases in this release include:
+
+- Users can create Private URLs that anonymize dataset metadata, allowing for double blind peer review. (Issue #1724, PR #7908)
+- Users can download Guestbook records using a new API. (Issue #7767, PR #7931)
+- Users can update terms metadata using the new semantic API. (Issue #5899, PR #7414)
+- Users can retrieve, set, and update metadata using a new, flatter JSON-LD format. (Issue #6497, PR #7414)
+- Administrators can use the Traces API to retreive information about specific types of user activity  (Issue #7952, PR #7953)
+
+## Notes for Dataverse Installation Administrators
+
+### New Database Constraint
+
+A new DB Constraint has been added in this release. Full instructions on how to identify whether or not your database needs any cleanup before the upgrade can be found in the [Dataverse software GitHub repository](https://github.com/IQSS/dataverse/blob/develop/scripts/issues/7451/PRE-RELEASE-INFO.txt). This information was also emailed out to Dataverse installation contacts.
+
+### Payara 5.2021.5 (or Higher) Required
+
+Some changes in this release require an upgrade to Payara 5.2021.5 or higher. (See the upgrade section).
+
+Instructions on how to update can be found in the [Payara documentation](https://docs.payara.fish/community/docs/5.2021.4/documentation/user-guides/upgrade-payara.html) We've included the necessary steps below, but we recommend that you review the Payara upgrade instructions as it could be helpful during any troubleshooting.
+
+### Enhancement to DDI Metadata Exports
+
+To increase support for internationalization and to improve compliance with CESSDA requirements, DDI exports now have a holdings element with a URI attribute whose value is the URL form of the dataset PID.
+
+## New JVM Options and DB Settings
+
+:AnonymizedFieldTypeNames can be used to enable creation of anonymized Private URLs and to specify which fields will be anonymized.
+
+## Notes for Tool Developers and Integrators
+
+### Semantic API
+
+The new Semantic API is especially helpful in data migrations and getting metadata into a Dataverse installation. Learn more in the [Developers Guide](https://guides.dataverse.org/en/5.6/developers/).
+
+## Complete List of Changes
+
+For the complete list of code changes in this release, see the [5.6 Milestone](https://github.com/IQSS/dataverse/milestone/97?closed=1) in Github.
+
+For help with upgrading, installing, or general questions please post to the [Dataverse Community Google Group](https://groups.google.com/forum/#!forum/dataverse-community) or email [email protected].
+
+## Installation
+
+If this is a new installation, please see our [Installation Guide](https://guides.dataverse.org/en/5.6/installation/).
+
+## Upgrade Instructions
+
+0\. These instructions assume that you've already successfully upgraded from Dataverse Software 4.x to Dataverse Software 5 following the instructions in the [Dataverse Software 5 Release Notes](https://github.com/IQSS/dataverse/releases/tag/v5.0). After upgrading from the 4.x series to 5.0, you should progress through the other 5.x releases before attempting the upgrade to 5.6.
+
+The steps below include a required upgrade to Payara 5.2021.5 or higher. (It is a simple matter of reusing your existing domain directory with the new distribution). But we also recommend that you review the Payara upgrade instructions as it could be helpful during any troubleshooting: [Payara documentation](https://docs.payara.fish/community/docs/5.2021.5/documentation/user-guides/upgrade-payara.html)
+
+If you are running Payara as a non-root user (and you should be!), **remember not to execute the commands below as root**. Use `sudo` to change to that user first. For example, `sudo -i -u dataverse` if `dataverse` is your dedicated application user.  
+
+In the following commands we assume that Payara 5 is installed in `/usr/local/payara5`. If not, adjust as needed.
+
+`export PAYARA=/usr/local/payara5`
+
+(or `setenv PAYARA /usr/local/payara5` if you are using a `csh`-like shell)
+
+1\. Undeploy the previous version 
+
+- `$PAYARA/bin/asadmin list-applications`
+- `$PAYARA/bin/asadmin undeploy dataverse<-version>`
+
+2\. Stop Payara
+
+- `service payara stop`
+- `rm -rf $PAYARA/glassfish/domains/domain1/generated`
+
+3\. Move the current Payara directory out of the way
+
+- `mv $PAYARA $PAYARA.MOVED`
+
+4\. Download the new Payara version (5.2021.5+), and unzip it in its place
+
+5\. Replace the brand new payara/glassfish/domains/domain1 with your old, preserved `domain1`
+
+6\. Start Payara
+
+- `service payara start`
+
+7\. Deploy this version.
+
+- `$PAYARA/bin/asadmin deploy dataverse-5.6.war`
+
+8\. Restart payara
+
+- `service payara stop`
+- `service payara start`
+
+## Additional Release Steps
+
+**If your installation relies on the database-side stored procedure for generating sequential numeric identifiers:**
+
+Note that you can skip this step if your installation uses the default-style, randomly-generated six alphanumeric character-long identifiers for your datasets! This is the case with most Dataverse installations.
+
+The underlying database framework has been modified in this release, to make it easier for installations  to create custom procedures for generating identifier strings that suit their needs. Your current configuration will  be automatically updated by the database upgrade (Flyway) script incorporated in the release. No manual configuration changes should be necessary. However, after the upgrade, we recommend that you confirm that your installation can still create new datasets, and that they are still assigned sequential numeric identifiers. In the unlikely chance that this is no longer working, please re-create the stored procedure following the steps described in the documentation for the `:IdentifierGenerationStyle` setting in the *Configuration* section of the Installation Guide for this release (v5.6).
+
+(Running the script supplied there will NOT overwrite the position on the sequence you are currently using!)

doc/release-notes/7786-ddi_changes.md

@@ -1,9 +1,9 @@
-Note: These notes cover several related PRs (#7984, #7958, #7959 respectively for the three bullets below.) If some are not merged before the next release, these notes should be adjusted.
+Note: These notes cover several related PRs (#7958, #7959 respectively for the three bullets below.) If some are not merged before the next release, these notes should be adjusted.
 
 ### Enhancements to DDI Metadata Exports
 
 Several changes have been made to the DDI exports to improve support for internationalization and to improve compliance with CESSDA requirements. These changes include:
-* Addition of a holdings element with a URI attribute whose value is the URL form of the dataset PID
+
 * Addition of xml:lang attributes specifying the dataset metadata language at the document level and for individual elements such as title and description
 * Specification of controlled vocabulary terms in duplicate elements in multiple languages (in the installation default langauge and, if different, the dataset metadata language)
 

doc/sphinx-guides/source/admin/dataverses-datasets.rst

@@ -41,7 +41,7 @@ Recursively assigns the users and groups having a role(s),that are in the set co
     curl -H "X-Dataverse-key: $API_TOKEN" http://$SERVER/api/admin/dataverse/$dataverse-alias/addRoleAssignmentsToChildren
     
 Configure a Dataverse Collection to store all new files in a specific file store
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 To direct new files (uploaded when datasets are created or edited) for all datasets in a given Dataverse collection, the store can be specified via the API as shown below, or by editing the 'General Information' for a Dataverse collection on the Dataverse collection page. Only accessible to superusers. ::
  
@@ -110,6 +110,8 @@ Mints a new identifier for a dataset previously registered with a handle. Only a
 
     curl -H "X-Dataverse-key: $API_TOKEN" -X POST http://$SERVER/api/admin/$dataset-id/reregisterHDLToPID
     
+.. _send-metadata-to-pid-provider:
+
 Send Dataset metadata to PID provider
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

doc/sphinx-guides/source/conf.py

@@ -65,9 +65,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '5.5'
+version = '5.6'
 # The full version, including alpha/beta/rc tags.
-release = '5.5'
+release = '5.6'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

doc/sphinx-guides/source/developers/dataset-migration-api.rst

@@ -7,6 +7,7 @@ This experimental migration API offers an additional option with some potential
 
 * metadata can be specified using the json-ld format used in the OAI-ORE metadata export
 * existing publication dates and PIDs are maintained (currently limited to the case where the PID can be managed by the Dataverse software, e.g. where the authority and shoulder match those the software is configured for)
+* updating the PID at the provider can be done immediately or later (with other existing APIs)
 * adding files can be done via the standard APIs, including using direct-upload to S3
 
 This API consists of 2 calls: one to create an initial Dataset version, and one to 'republish' the dataset through Dataverse with a specified publication date.
@@ -44,6 +45,14 @@ The call above creates a Dataset. Once it is created, other APIs can be used to
   export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
   export SERVER_URL=https://demo.dataverse.org
  
-  curl -H 'Content-Type: application/jsonld' -H X-Dataverse-key:$API_TOKEN -X POST -d '{"schema:datePublished": "2020-10-26","@context":{ "schema":"http://schema.org/"}}' "$SERVER_URL/api/datasets/{id}/actions/:releasemigrated"
+  curl -H 'Content-Type: application/ld+json' -H X-Dataverse-key:$API_TOKEN -X POST -d '{"schema:datePublished": "2020-10-26","@context":{ "schema":"http://schema.org/"}}' "$SERVER_URL/api/datasets/{id}/actions/:releasemigrated"
 
 datePublished is the only metadata supported in this call.
+
+An optional query parameter: updatepidatprovider (default is false) can be set to true to automatically update the metadata and targetUrl of the PID at the provider. With this set true, the result of this call will be that the PID redirects to this dataset rather than the dataset in the source repository.
+
+.. code-block:: bash
+
+  curl -H 'Content-Type: application/ld+json' -H X-Dataverse-key:$API_TOKEN -X POST -d '{"schema:datePublished": "2020-10-26","@context":{ "schema":"http://schema.org/"}}' "$SERVER_URL/api/datasets/{id}/actions/:releasemigrated?updatepidatprovider=true"
+
+If the parameter is not added and set to true, other existing APIs can be used to update the PID at the provider later, e.g. :ref:`send-metadata-to-pid-provider`

doc/sphinx-guides/source/developers/dev-environment.rst

@@ -85,9 +85,9 @@ To install Payara, run the following commands:
 
 ``cd /usr/local``
 
-``sudo curl -O -L https://s3-eu-west-1.amazonaws.com/payara.fish/Payara+Downloads/5.2021.4/payara-5.2021.4.zip``
+``sudo curl -O -L https://s3-eu-west-1.amazonaws.com/payara.fish/Payara+Downloads/5.2021.5/payara-5.2021.5.zip``
 
-``sudo unzip payara-5.2021.4.zip``
+``sudo unzip payara-5.2021.5.zip``
 
 ``sudo chown -R $USER /usr/local/payara5``
 

doc/sphinx-guides/source/installation/prerequisites.rst

@@ -44,7 +44,7 @@ On RHEL/derivative you can make Java 11 the default with the ``alternatives`` co
 Payara
 ------
 
-Payara 5.2021.4 is recommended. Newer versions might work fine, regular updates are recommended.
+Payara 5.2021.5 is recommended. Newer versions might work fine, regular updates are recommended.
 
 Installing Payara
 =================
@@ -55,8 +55,8 @@ Installing Payara
 
 - Download and install Payara (installed in ``/usr/local/payara5`` in the example commands below)::
 
-	# wget https://s3-eu-west-1.amazonaws.com/payara.fish/Payara+Downloads/5.2021.4/payara-5.2021.4.zip
-	# unzip payara-5.2021.4.zip
+	# wget https://s3-eu-west-1.amazonaws.com/payara.fish/Payara+Downloads/5.2021.5/payara-5.2021.5.zip
+	# unzip payara-5.2021.5.zip
 	# mv payara5 /usr/local
 
 If you intend to install and run Payara under a service account (and we hope you do), chown -R the Payara hierarchy to root to protect it but give the service account access to the below directories: