Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid Sqlalchemy errors from prior harvest rollbacks #445

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Avoid Sqlalchemy errors from prior harvest rollbacks #445

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
322110e
Validate before DB transaction boundary
bonnland May 18, 2021
18befbc
Avoid InvalidRequestError and connection timeouts
bonnland May 20, 2021
07c371e
Merge branch 'fix-validation' of https://github.com/bonnland/ckanext-…
bonnland May 20, 2021
ef23ce6
Revert ckanharvester code
bonnland May 20, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Revert ckanharvester code
  • Loading branch information
@bonnland
bonnland committed May 20, 2021
commit ef23ce6dd055b72e7a98940eb48ae820aef54c94
38 changes: 0 additions & 38 deletions ckanext/harvest/harvesters/ckanharvester.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@
from ckan.logic import ValidationError, NotFound, get_action
from ckan.lib.helpers import json
from ckan.plugins import toolkit
import ckan.lib.plugins as lib_plugins

from ckanext.harvest.model import HarvestObject
from .base import HarvesterBase
Expand Down Expand Up @@ -548,43 +547,6 @@ def get_extra(key, package_dict):

package_dict = self.modify_package_dict(package_dict, harvest_object)

# validate packages if needed
validate_packages = True
if validate_packages:
if 'type' not in package_dict:
package_plugin = lib_plugins.lookup_package_plugin()
try:
# use first type as default if user didn't provide type
package_type = package_plugin.package_types()[0]
except (AttributeError, IndexError):
package_type = 'dataset'
# in case a 'dataset' plugin was registered w/o fallback
package_plugin = lib_plugins.lookup_package_plugin(package_type)
package_dict['type'] = package_type
else:
package_plugin = lib_plugins.lookup_package_plugin(package_dict['type'])


errors = {}
# if package has been previously imported
try:
existing_package_dict = self._find_existing_package(package_dict)

if not 'metadata_modified' in package_dict or \
package_dict['metadata_modified'] > existing_package_dict.get('metadata_modified'):
schema = package_plugin.update_package_schema()
data, errors = lib_plugins.plugin_validate(
package_plugin, base_context, package_dict, schema, 'package_update')

except NotFound:

schema = package_plugin.create_package_schema()
data, errors = lib_plugins.plugin_validate(
package_plugin, base_context, package_dict, schema, 'package_create')

if errors:
raise ValidationError(errors)

result = self._create_or_update_package(
package_dict, harvest_object, package_dict_form='package_show')

Expand Down