Merge pull request IQSS#6552 from IQSS/develop

Merge v4.19 into master

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,15 @@
+**What this PR does / why we need it**:
+
+**Which issue(s) this PR closes**:
+
+Closes #
+
+**Special notes for your reviewer**:
+
+**Suggestions on how to test this**:
+
+**Does this PR introduce a user interface change?**:
+
+**Is there a release notes update needed for this change?**:
+
+**Additional documentation**:

conf/docker-aio/0prep_deps.sh

@@ -8,7 +8,7 @@ if [ ! -e dv/deps/glassfish4dv.tgz ]; then
 	mkdir -p /tmp/dv-prep/gf
 	cd /tmp/dv-prep/gf
 	wget http://download.java.net/glassfish/4.1/release/glassfish-4.1.zip
-	wget http://search.maven.org/remotecontent?filepath=org/jboss/weld/weld-osgi-bundle/2.2.10.Final/weld-osgi-bundle-2.2.10.Final-glassfish4.jar -O weld-osgi-bundle-2.2.10.Final-glassfish4.jar
+	wget https://search.maven.org/remotecontent?filepath=org/jboss/weld/weld-osgi-bundle/2.2.10.Final/weld-osgi-bundle-2.2.10.Final-glassfish4.jar -O weld-osgi-bundle-2.2.10.Final-glassfish4.jar
 	unzip glassfish-4.1.zip
 	rm glassfish4/glassfish/modules/weld-osgi-bundle.jar
 	mv weld-osgi-bundle-2.2.10.Final-glassfish4.jar glassfish4/glassfish/modules

conf/solr/7.3.1/updateSchemaMDB.sh

@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/bin/bash
 set -euo pipefail
 
 # This script updates the <field> and <copyField> schema configuration necessary to properly
@@ -61,6 +61,12 @@ echo "Retrieve schema data from ${DATAVERSE_URL}/api/admin/index/solr/schema"
 TMPFILE=`mktemp`
 curl -f -sS "${DATAVERSE_URL}/api/admin/index/solr/schema${UNBLOCK_KEY}" > $TMPFILE
 
+### Fail gracefull if Dataverse is not ready yet.
+if [[ "`wc -l ${TMPFILE}`" < "3" ]]; then
+  echo "Dataverse responded with empty file. When running on K8s: did you bootstrap yet?"
+  exit 123
+fi
+
 ### Processing
 echo "Writing ${TARGET}/schema_dv_mdb_fields.xml"
 echo "<fields>" > ${TARGET}/schema_dv_mdb_fields.xml
@@ -76,4 +82,4 @@ rm ${TMPFILE}*
 
 ### Reloading
 echo "Triggering Solr RELOAD at ${SOLR_URL}/solr/admin/cores?action=RELOAD&core=collection1"
-curl -f -sS "${SOLR_URL}/solr/admin/cores?action=RELOAD&core=collection1"
+curl -f -sS "${SOLR_URL}/solr/admin/cores?action=RELOAD&core=collection1"

doc/mergeParty/readme.md

@@ -1,5 +1,5 @@
-# Merge Party Readme
-Welcome to the merge party! This document is intended to give a short overview of why we need this party, when was changed and how to change it. There's much work to do, so we'll keep it short. Hopefully.
+# Merge Party
+Welcome to the merge party! This document is intended to give a short overview of why we need this party, when was it changed and how to change it. There's much work to do, so we'll keep it short, hopefully.
 
 ## What Just Happened
 In order to allow users to log into Dataverse using credentials from other systems (e.g. institutional Shibboleth server), we had to refactor out the internal user management sub-system (formerly known as "DataverseUser") and introduce a new user system. The existing system was taken out of Dataverse but kept in the .war file, as we also need to support standalone instances.
@@ -16,7 +16,7 @@ From a merge standpoint, this means that code that previously referenced `Datave
 
 Most of these changes have been done by Michael/Phil - otherwise, the `auth` branch would not compile.
 
-Since the guest user does not live in the database, it does not have an id. Moreover, JPA classes cannot link directly to it\*. But have no fear - all users (and, really, all `RoleAssignee`s, which are users or groups) have an identifier. When you need to reference a user (and later, a group) just use the identifier (it's of type `String`). When needing to convert an identifier to a user, call `RoleAssigneeServiceBean.getRoleAssignee( identifier )` in the general case, or `AuthenticationServiceBean.getAuthenticatedUser(identifier)` if you're certain the identifier is of an authenticated user.
+The guest user does not live in the database so it does not have an id. Moreover, JPA classes cannot link directly to it\*. But have no fear - all users (and, really, all `RoleAssignee`s, which are users or groups) have an identifier. When you need to reference a user (and later, a group) just use the identifier (it's of type `String`). When needing to convert an identifier to a user, call `RoleAssigneeServiceBean.getRoleAssignee( identifier )` in the general case, or `AuthenticationServiceBean.getAuthenticatedUser(identifier)` if you're certain the identifier is of an authenticated user.
 
 
 \* We have debated this for a while, since we could have created a dummy record, like we've done so far. We went with this solution, as it is cleaner, can't be messed up by SQL scripts, and will make even more sense once groups arrive.
@@ -84,4 +84,4 @@ A new script that sets up the users and the dataverses, sets the system up for b
 
 ## Undoing the undoing the merge
 When merging back to master, we need to undo commit 8ae3e6a482b87b52a1745bb06f340875803d2c5b (a.k.a 8ae3e6a), which is the commit that undid the erroneous merge.
-More at http://www.christianengvall.se/undo-pushed-merge-git/
+More at http://www.christianengvall.se/undo-pushed-merge-git/

doc/release-notes/4.19-release-notes.md

@@ -0,0 +1,125 @@
+# Dataverse 4.19
+
+This release brings new features, enhancements, and bug fixes to Dataverse. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project.
+
+## Release Highlights
+
+### Open ID Connect Support
+
+Dataverse now provides basic support for any OpenID Connect (OIDC) compliant authentication provider.
+
+Prior to supporting this standard, new authentication methods needed to be added by pull request. OIDC support provides a standardized way for authentication, sharing user information, and more. You are able to use any compliant provider just by loading a configuration file, without touching the codebase. While the usual prominent providers like Google and others feature OIDC support there are plenty of other options to easily attach your installation to a custom authentication provider, using enterprise grade software. 
+
+See the [OpenID Connect Login Options documentation](http://guides.dataverse.org/en/4.19/installation/oidc.html) in the Installation Guide for more details.
+
+This is to be extended with support for attribute mapping, group syncing and more in future versions of the code.  
+
+### Python Installer
+
+We are introducing a new installer script, written in Python. It is intended to eventually replace the old installer (written in Perl). For now it is being offered as an (experimental) alternative.
+
+See [README_python.txt](https://github.com/IQSS/dataverse/blob/v4.19/scripts/installer/README_python.txt) in scripts/installer and/or in the installer bundle for more information.
+
+## Major Use Cases
+
+Newly-supported use cases in this release include:
+
+- Dataverse installation administrators will be able to experiment with a Python Installer (Issue #3937, PR #6484)
+- Dataverse installation administrators will be able to set up an OIDC-compliant login options by editing a configuration file and with no need for a code change (Issue #6432, PR #6433)
+- Following setup by a Dataverse administration, users will be able to log in using OIDC-compliant methods (Issue #6432, PR #6433)
+- Users of the Search API will see additional fields in the JSON output (Issues #6300, #6396, PR #6441)
+- Users loading the support form will now be presented with the math challenge as expected and will be able to successfully send an email to support (Issue #6307, PR #6462)
+- Users of https://mybinder.org can now spin up Jupyter Notebooks and other computational environments from Dataverse DOIs (Issue #4714, PR #6453)
+
+## Notes for Dataverse Installation Administrators
+
+### Security vulnerability in Solr
+
+A serious security issue has recently been identified in multiple versions of Solr search engine, including v.7.3 that Dataverse is currently using. Follow the instructions below to verify that your installation is safe from a potential attack. You can also consult the following link for a detailed description of the issue:
+
+<A HREF="https://github.com/veracode-research/solr-injection#7-cve-2019-xxxx-rce-via-velocity-template-by-_s00py">RCE in Solr via Velocity Template</A>.
+
+The vulnerability allows an intruder to execute arbitrary code on the system running Solr. Fortunately, it can only be exploited if Solr API access point is open to direct access from public networks (aka, "the outside world"), which is NOT needed in a Dataverse installation.
+
+We have always recommended having Solr (port 8983) firewalled off from public access in our installation guides. But we recommend that you double-check your firewall settings and verify that the port is not accessible from outside networks. The simplest quick test is to try the following URL in your browser:
+
+      `http://<your Solr server address>:8983`
+
+and confirm that you get "access denied" or that it times out, etc. 
+
+In most cases, when Solr runs on the same server as the Dataverse web application, you will only want the port accessible from localhost. We also recommend that you add the following arguments to the Solr startup command: `-j jetty.host=127.0.0.1`. This will make Solr accept connections from localhost only; adding redundancy, in case of the firewall failure.
+
+In a case where Solr needs to run on a different host, make sure that the firewall limits access to the port only to the Dataverse web host(s), by specific ip address(es).
+
+We would also like to reiterate that it is simply never a good idea to run Solr as root! Running the process as a non-privileged user would substantially minimize any potential damage even in the event that the instance is compromised.
+
+### Citation and Geospatial Metadata Block Updates
+
+We updated two metadata blocks in this release. Updating these metadata blocks is mentioned in the step-by-step upgrade instructions below. 
+
+### Run ReExportall
+
+We made changes to the JSON Export in this release (#6246). If you'd like these changes to reflected in your JSON exports, you should run ReExportall as part of the upgrade process. We've included this in the step-by-step instructions below.
+
+### BinderHub
+
+https://mybinder.org now supports spinning up Jupyter Notebooks and other computational environments from Dataverse DOIs.
+
+### Widgets update for OpenScholar
+
+We updated the code for widgets so that they will keep working in OpenScholar sites after the upcoming upgrade OpenScholar upgrade to Drupal 8. If users of your dataverse have embedded widgets on an Openscholar site that upgrades to Drupal 8, you will need to run this Dataverse version (or later) for the widgets to keep working.
+
+### Payara tech preview
+
+Dataverse 4 has always run on Glassfish 4.1 but changes in this release (PR #6523) should open the door to upgrading to Payara 5 eventually. Production installations of Dataverse should remain on Glassfish 4.1 but feedback from any experiments running Dataverse on Payara 5 is welcome via the [usual channels](https://dataverse.org/contact).
+
+## Notes for Tool Developers and Integrators
+
+### Search API
+
+The boolean parameter `query_entities` has been removed from the Search API. The former "true" behavior of "whether entities are queried via direct database calls (for developer use)" is now always true.
+
+Additional fields are now available via the Search API, mostly related to information about specific dataset versions.
+
+## Complete List of Changes
+
+For the complete list of code changes in this release, see the <a href="https://github.com/IQSS/dataverse/milestone/86?closed=1">4.19 milestone</a> in Github.
+
+For help with upgrading, installing, or general questions please post to the <a href="https://groups.google.com/forum/#!forum/dataverse-community">Dataverse Google Group</a> or email [email protected].
+
+## Installation
+
+If this is a new installation, please see our <a href="http://guides.dataverse.org/en/4.19/installation/">Installation Guide</a>.
+
+## Upgrade
+
+1. Undeploy the previous version.
+
+- &lt;glassfish install path&gt;/glassfish4/bin/asadmin list-applications
+- &lt;glassfish install path&gt;/glassfish4/bin/asadmin undeploy dataverse
+
+2. Stop glassfish and remove the generated directory, start.
+
+- service glassfish stop
+- remove the generated directory: rm -rf &lt;glassfish install path&gt;glassfish4/glassfish/domains/domain1/generated
+- service glassfish start
+
+3. Deploy this version.
+
+- &lt;glassfish install path&gt;/glassfish4/bin/asadmin deploy &lt;path&gt;dataverse-4.19.war
+
+4. Restart glassfish.
+
+5. Update Citation Metadata Block
+
+- `wget https://github.com/IQSS/dataverse/releases/download/v4.19/citation.tsv`
+- `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @citation.tsv -H "Content-type: text/tab-separated-values"`
+
+6. Update Geospatial Metadata Block
+
+- `wget https://github.com/IQSS/dataverse/releases/download/v4.19/geospatial.tsv`
+- `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @geospatial.tsv -H "Content-type: text/tab-separated-values"`
+
+7. (Optional) Run ReExportall to update JSON Exports  
+
+   <http://guides.dataverse.org/en/4.19/admin/metadataexport.html?highlight=export#batch-exports-through-the-api>

doc/sphinx-guides/source/_static/installation/files/etc/init.d/solr

@@ -7,7 +7,7 @@
 
 SOLR_DIR="/usr/local/solr/solr-7.3.1"
 SOLR_COMMAND="bin/solr"
-SOLR_ARGS="-m 1g"
+SOLR_ARGS="-m 1g -j jetty.host=127.0.0.1"
 SOLR_USER=solr
 
 case $1 in
@@ -33,4 +33,3 @@ case $1 in
         exit 1
         ;;
 esac
-