mds-sync

Service to sync items from MDS to SMB Online-Sammlungen.

Introduction

This project contains all the code required to run a sync between SMB-Online-Sammlungen and MDS.
4 types of sync are implemented:

1. Incremental - Incremental update of changes determined by __lastModified timestamp of the MDS objects
2. Deletions - Removal of objects that have been put into the MDS trash-bin
3. Highlights - Resolving of MDS objects that have been flagged as highlight
4. Manual - Sync MDS objects based on ids or timestamps

Triggering a Sync

To start a sync the AdminRestController is available. It provides a /admin/sync endpoint that needs to be called via http PATCH with a fairly simple payload:

Attribute	Type	Optional	Default Value	Comment
type	enum(INCREMENTAL,HIGHLIGHTS,DELETIONS,ASSORTMENTS,MANUAL)	x	MANUAL	Specifies what should be synced
ids	string[] / number[]	x		for type MANUAL; List of MDS-Ids that should be synced, also allows range definitions ["0..n"]
module	string	x	Object	for type MANUAL; the MDS module that should be synced
modifiedFrom	timestamp	x	1970-01-01T00:00:00+00:00	for type MANUAL; starttime filter for __lastModified timestamp
modifiedTo	timestamp	x	now	for type MANUAL; endtime filter for __lastModified timestamp

Note: ids as well as modifiedFrom and modifiedTo are ignored for a non-MANUAL-typed request. For a MANUAL-typed request either the ids or the date must be given. It is not possible to combine both at the same time.

Examples:

{
    "type": "HIGHLIGHTS"
}

{
    "ids": ["122143..362234"]
}

{
    "ids": [12, "2132..2245", 3434, "98234..98241", 324432]
}

{
    "modifiedFrom": "2021-01-16T01:36:10.649335+00:00",
    "modifiedTo": "2021-01-25T01:34:48.778002+00:00"
}

Note: There is no authentication required on the endpoint since it is not exposed to the outside world. It is available only inside the k8s cluster and hence needs port-forwarding to be invoked locally.

kubectl port-forward pods/smb-mds-sync-... 8082:8082

There is an additional REST endpoint /admin/trigger/sync that is supposed to be called from Hasura. It expects an event-trigger payload with object info in the request. The implementation behind this endpoint is similar to calling the /admin/sync with a single-element in the ids array.

Example:

{
    "event": {
        "data": {
            "new": {
                "id": 122143
            }
        }
    }
}

Development

Technology Stack

• Docker
• Java/Kotlin
• Spring Boot
• Jackson
• Apollo GraphQl
• JAXB
• JUnit
• Sonar

Dependencies

• MDS Zetcom API
• SMB Online-Sammlungen API (Hasura)
• SMB Image Provider
• SMB Search-Indexer

Overall Architecture

The code is written in Java and Kotlin. There is no actual restriction on when to use which. However, the project is essentially split into two parts that are connected by a DataQueue. There is a grown convention that the producing code for the data-queue is written in Java whereas the consuming code is written Kotlin.
The producing code is the API implementation for MDS essentially using Controller, Resolver, Executer and Parser classes which fetch the objects and convert them into internal data transfer objects. These objects are then offered to the consuming part via the DataQueue. The consuming end is built by Service and Repository classes that implement the GraphQL API for Hasura to store the data into the database.

API data objects for MDS used in the producing code are generated via JAXB.
Code for the GraphQl API on the consuming side is generated by Apollo.

Environment Variables

The required env-vars are defined in application.yml.

Note: There is an additional application-test.yml that is applied during test execution and (partially) overrides what is defined in application.yml.

Package Structure

The base package of the application is de.smbonline.mdssync.

Package	Description
de.smbonline.mdssync	Application entry point
de.smbonline.mdssync.dataprocessor.controller	Service configuration (consuming side)
de.smbonline.mdssync.dataprocessor.graphql	GraphQl code (consuming side)
de.smbonline.mdssync.dataprocessor.queue	Interface between producing and consuming side
de.smbonline.mdssync.dataprocessor.repository	GraphQl and webdav repositories (consuming side)
de.smbonline.mdssync.dataprocessor.service	Service implementation (consuming side)
de.smbonline.mdssync.dto	Data transfer objects
de.smbonline.mdssync.exc	Exception classes
de.smbonline.mdssync.exec	Sync execution code (producing side)
de.smbonline.mdssync.index	Search-indexer connection (producing side)
de.smbonline.mdssync.log	Logging utils
de.smbonline.mdssync.pattern	Generic code for software patterns
de.smbonline.mdssync.properties	Wrappers for application.yml configuration
de.smbonline.mdssync.rest	REST code (producing side)
de.smbonline.mdssync.rules	Data transformation rules (producing side)
de.smbonline.mdssync.search	MDS code (producing side)
de.smbonline.mdssync.util	Shared utilities

Developer Guidelines and Code Conventions

• The code is based on @NonNullApi, everything that can have a null value is flagged as @Nullable.
• All access to member attributes is qualified with this..
• Method arguments are final.
• Every code change needs to be reflected by an increased version number in build.gradle.kts.
• We use fully-qualified imports for interfaces and classes.
• We use static wildcard imports for constants and methods from static-only utility classes.
• Developers run gradlew sonar before pushing code to Git.