detail.adoc

JobPack Standard Definition

Table of Contents

• 1. Introduction
  • 1.1. Examples
  • 1.2. Definitions
• 2. Standard
• 3. Usage
• 4. Discovery
  • 4.1. Docker Hub
  • 4.2. Docker Trusted Registry
  • 4.3. Docker Registry
• 5. Labels
• 6. Manifest

1. Introduction

JobPack is a general standard to aid in the discovery and consumption of discrete units of work contained within a Docker image. While initially developed to support the Scale data processing system with job discovery, it is designed to be readily applied to other systems as well.

JobPack compliant images must be named in a specific fashion due to the lack of label search capability on Docker Hub and Registry services. The suffix -jobpack must be used when naming images to enable discovery, prior to Hub or Registry push. This requirement will be deprecated as label search support is standardized across Docker registry services.

1.1. Examples

Dockerfile snippet containing required label for JobPack compliance:

link:examples/Dockerfile-snippet[role=include]

The com.jobpack.manifest label contents must be serialized as a string-escaped JSON object. The following is a complete example including required and optional keys:

link:examples/jobpack-full.json[role=include]

1.2. Definitions

• JavaScript Object Notation (JSON), and the terms object, name, value, array, and number, are defined in IETF RFC 4627.

• Semantic Versioning (SemVer), and the terms major, minor, and patch version are defined at http://semver.org/spec/v2.0.0.html

• The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in IETF RFC 2119.

2. Standard

sections/standard.adoc

3. Usage

A few requirements must be satisfied when constructing a system capable of executing JobPack standardized images. These are outlined below:

• Environment variable injection must be performed. May be consumed by job directly as environment variables or shell variable expansion may be leveraged in the interface.args member. The following minimum variables MUST be provided:

  • JOB_OUTPUT_DIR: Root path job must place all products that should be captured as output.

  • ALLOCATED_CPUS: Value of cpus member.

  • ALLOCATED_MEM: Value of mem member.

  • ALLOCATED_STORAGE: Value of storage member.

  • All name member values of interface.inputData elements: Indicate the absolute path of input files. Environment variable name case MUST map exactly from name value.

4. Discovery

A primary intention of this standard is for simple job discovery from public images hosted within either Docker Hub, Docker Trusted Registry or Docker Registry. There is significant fragmentation of APIs between the various Docker offerings and the following sections describe the steps that may be taken to access the labels defined by CJSD.

4.1. Docker Hub

https://hub.docker.com/v2/search/repositories/?query=-jobpack

Use ImageLayers /registry/analyze endpoint to extrapolate the Dockerfile contents and retrieve label metadata.

4.2. Docker Trusted Registry

TBD Waiting on forum post response

4.3. Docker Registry

• Use catalog to find names: http://myreg.org/v2/_catalog

• Follow with tags query per image matched: http://myreg.org/v2/{image-id}/tags/list

• Finally use manifests API to retrieve labels per tag (extract labels from history): http://myreg.org/v2/{image-id}/manifests/{tag}

5. Labels

TODO: Give details on contents and formatting of all JobPack labels

6. Manifest

TODO: Give details on manifest usage and explanation of Scale specific use.