Skip to content

Commit

Permalink
Add metadata files
Browse files Browse the repository at this point in the history
  • Loading branch information
johnoliver committed Jan 29, 2019
1 parent 9fbea83 commit fac408c
Show file tree
Hide file tree
Showing 11 changed files with 226 additions and 38 deletions.
8 changes: 4 additions & 4 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,18 +100,18 @@ Sets the default environment variables for specific platforms that are currently
set in individual jobs. This allows us to now track and version these variables.
1. New _build-farm/sign-releases.sh added for the new AdoptOpenJDK Build Farm
jenkins pipeline to code sign Adopt OpenJDK binaries (Mac and Windows for now).
1. _pipelines/build/build_base_file.groovy_ added. This co-ordinates the various
1. _pipelines/build/common/build_base_file.groovy_ added. This co-ordinates the various
pipeline builds.
1._pipelines/build/create\_job\_from\_template.groovy_ added. This dynamically
1._pipelines/build/common/create\_job\_from\_template.groovy_ added. This dynamically
creates jenkins jobs for a particular pipeline run (e.g. All jdk8u jobs).
1. _pipelines/build/openjdk\_build\_pipeline.groovy_ added. This forms the base
1. _pipelines/build/common/openjdk\_build\_pipeline.groovy_ added. This forms the base
pipeline code for each build.
1. _pipelines/build/openjdk\<version\>\_\<variant\>\_\<nightly\|release\>\_pipeline.groovy_
files added. These will eventually replace the existing individual jobs with a
Pipeline for each version and variant.
1. _pipelines/build/openjdk\<version\>\_pipeline.groovy_
files added. These define the configurations for the
_pipelines/build/create\_job\_from\_template.groovy_ to create jobs for a pipeline
_pipelines/build/common/create\_job\_from\_template.groovy_ to create jobs for a pipeline
run.

### Documentation and Misc
Expand Down
24 changes: 24 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -305,6 +305,30 @@ to your local platform and then build (Adopt) OpenJDK and place the result in
These scripts default to using AdoptOpenJDK as the OpenJDK source repo to build
from, but you can override this with the `-r` flag.

#Metadata
Along side the built assets a metadata file will be created with info about the build. This will look of the form:

```
{
"os": "linux",
"arch": "x64",
"variant": "hotspot",
"version": "jdk8u",
"tag": "jdk8u202-b08",
{
"adopt_build_number": 2,
"major": 8,
"minor": 0,
"security": 202,
"build": 8,
"version": "8u202-b08",
"semver": "8u202+8.2",
},
"binary_type": "jdk"
}
```

It is worth noting the additional tags on the semver is the adopt build number.

# Build farm

Expand Down
8 changes: 4 additions & 4 deletions docs/build.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ that was requested to be built.
### Top level pipeline code

The actions of the pipeline are defined by [openjdk10_pipeline.groovy](/pipelines/build/openjdk10_pipeline.groovy), which
calls [build_base_file.groovy](/pipelines/build/build_base_file.groovy).
calls [build_base_file.groovy](/pipelines/build/common/build_base_file.groovy).

The actions of this script are:

Expand All @@ -36,12 +36,12 @@ The actions of this script are:

#### Generated build jobs

These are generated using the template at [create_job_from_template.groovy](/pipelines/build/create_job_from_template.groovy)
This defines a pipeline job that calls [openjdk_build_pipeline.groovy](/pipelines/build/openjdk_build_pipeline.groovy), this script defines the work of what a Build Pipeline does.
These are generated using the template at [create_job_from_template.groovy](/pipelines/build/common/create_job_from_template.groovy)
This defines a pipeline job that calls [openjdk_build_pipeline.groovy](/pipelines/build/common/openjdk_build_pipeline.groovy), this script defines the work of what a Build Pipeline does.

## Build Pipelines

Generated by [create_job_from_template.groovy](/pipelines/build/create_job_from_template.groovy), and its work defined in [openjdk_build_pipeline.groovy](/pipelines/build/openjdk_build_pipeline.groovy).
Generated by [create_job_from_template.groovy](/pipelines/build/common/create_job_from_template.groovy), and its work defined in [openjdk_build_pipeline.groovy](/pipelines/build/common/openjdk_build_pipeline.groovy).

The actions of this are:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@ limitations under the License.
* 3. Push generated artifacts to github
*/



def toBuildParams(enableTests, cleanWorkspace, params) {

List buildParams = []
Expand All @@ -40,7 +42,7 @@ def toBuildParams(enableTests, cleanWorkspace, params) {
return buildParams
}

def buildConfiguration(javaToBuild, variant, configuration, releaseTag, branch, additionalConfigureArgs, additionalBuildArgs) {
def buildConfiguration(javaToBuild, variant, configuration, releaseTag, branch, additionalConfigureArgs, additionalBuildArgs, adoptBuildNumber) {

def additionalNodeLabels = formAdditionalNodeLabels(configuration, variant)

Expand Down Expand Up @@ -78,12 +80,13 @@ def buildConfiguration(javaToBuild, variant, configuration, releaseTag, branch,
def testList = getTestList(configuration, isRelease)

return [
javaVersion: javaToBuild,
arch : configuration.arch,
os : configuration.os,
variant : variant,
parameters : buildParams,
test : testList
javaVersion : javaToBuild,
arch : configuration.arch,
os : configuration.os,
variant : variant,
parameters : buildParams,
test : testList,
adoptBuildNumber: adoptBuildNumber
]
}

Expand Down Expand Up @@ -162,7 +165,7 @@ static def getConfigureArgs(configuration, additionalConfigureArgs) {
return buildParams
}

def getJobConfigurations(javaVersionToBuild, availableConfigurations, String targetConfigurations, String releaseTag, String branch, String additionalConfigureArgs, String additionalBuildArgs, String additionalFileNameTag) {
def getJobConfigurations(javaVersionToBuild, availableConfigurations, String targetConfigurations, String releaseTag, String branch, String additionalConfigureArgs, String additionalBuildArgs, String additionalFileNameTag, String adoptBuildNumber) {
def jobConfigurations = [:]

//Parse config passed to jenkins job
Expand All @@ -188,7 +191,7 @@ def getJobConfigurations(javaVersionToBuild, availableConfigurations, String tar
}
}

jobConfigurations[name] = buildConfiguration(javaVersionToBuild, variant, configuration, releaseTag, branch, additionalConfigureArgs, additionalBuildArgs)
jobConfigurations[name] = buildConfiguration(javaVersionToBuild, variant, configuration, releaseTag, branch, additionalConfigureArgs, additionalBuildArgs, adoptBuildNumber)
}
}
}
Expand Down Expand Up @@ -228,7 +231,7 @@ def createJob(jobName, jobFolder, config, enableTests, scmVars) {
params.put("GIT_URI", scmVars["GIT_URL"])
params.put("GIT_BRANCH", scmVars["GIT_BRANCH"])

create = jobDsl targets: "pipelines/build/create_job_from_template.groovy", ignoreExisting: false, additionalParameters: params
create = jobDsl targets: "pipelines/build/common/create_job_from_template.groovy", ignoreExisting: false, additionalParameters: params

return create
}
Expand Down Expand Up @@ -284,18 +287,18 @@ def doBuild(
scmVars,
String additionalBuildArgs,
String additionalFileNameTag,
String cleanWorkspaceBeforeBuild) {
String cleanWorkspaceBeforeBuild,
String adoptBuildNumber) {

if (releaseTag == null || releaseTag == "false") {
releaseTag = ""
}

def jobConfigurations = getJobConfigurations(javaVersionToBuild, availableConfigurations, targetConfigurations, releaseTag, branch, additionalConfigureArgs, additionalBuildArgs, additionalFileNameTag)
def jobConfigurations = getJobConfigurations(javaVersionToBuild, availableConfigurations, targetConfigurations, releaseTag, branch, additionalConfigureArgs, additionalBuildArgs, additionalFileNameTag, adoptBuildNumber)

if (!checkSaneConfig(releaseTag, jobConfigurations)) {
return
}

def jobs = [:]

def enableTests = Boolean.valueOf(enableTestsArg)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ pipelineJob("$buildFolder/$JOB_NAME") {
}
}
}
scriptPath('pipelines/build/openjdk_build_pipeline.groovy')
scriptPath('pipelines/build/common/openjdk_build_pipeline.groovy')
lightweight(true)
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,9 @@ limitations under the License.

@Library('openjdk-jenkins-helper@master')
import JobHelper
import NodeHelper
@Library('openjdk-jenkins-helper@master')
import JobHelper
import groovy.json.JsonOutput
import groovy.json.JsonSlurper

/**
Expand All @@ -31,16 +33,110 @@ import groovy.json.JsonSlurper
*
*/

def getJavaVersionNumber(version) {
// version should be something like "jdk8u"
def matcher = (version =~ /(\d+)/)
return Integer.parseInt(matcher[0][1])
def addOr0(map, name, matched, groupName) {
def number = matched.group(groupName)
if (number != null) {
map.put(name, number as Integer)
} else {
map.put(name, 0)
}
return map
}

def matchPre223(version) {
final pre223regex = "jdk(?<version>(?<major>[0-8]+)(u(?<update>[0-9]+))?(-b(?<build>[0-9]+))(_(?<opt>[-a-zA-Z0-9\\.]+))?)"
final matched = version =~ /${pre223regex}/

if (matched.matches()) {
result = [:]
result = addOr0(result, 'major', matched, 'major')
result.put('minor', 0)
result = addOr0(result, 'security', matched, 'update')
result = addOr0(result, 'build', matched, 'build')
if (matched.group('opt') != null) result.put('opt', matched.group('opt'))
result.put('version', matched.group('version'))

return result
} else {
return null
}
}

def match223(version) {
//Regexes based on those in http://openjdk.java.net/jeps/223
// Technically the standard supports an arbitrary number of numbers, we will support 3 for now
final vnumRegex = "(?<major>[0-9]+)(\\.(?<minor>[0-9]+))?(\\.(?<security>[0-9]+))?"
final pre = "(?<pre>[a-zA-Z0-9]+)"
final build = "(?<build>[0-9]+)"
final opt = "(?<opt>[-a-zA-Z0-9\\.]+)"

final version223Regexs = [
"(?:jdk\\-)(?<version>${vnumRegex}(\\-${pre})?\\+${build}(\\-${opt})?)",
"(?:jdk\\-)(?<version>${vnumRegex}\\-${pre}(\\-${opt})?)",
"(?:jdk\\-)(?<version>${vnumRegex}(\\+\\-${opt})?)"
]

return version223Regexs
.findResult({ regex ->
final matched223 = version =~ /${regex}/
if (matched223.matches()) {
result = [:]
result = addOr0(result, 'major', matched223, 'major')
result = addOr0(result, 'minor', matched223, 'minor')
result = addOr0(result, 'security', matched223, 'security')
if (matched223.group('pre') != null) result.put('pre', matched223.group('pre'))
result = addOr0(result, 'build', matched223, 'build')
if (matched223.group('opt') != null) result.put('opt', matched223.group('opt'))
result.put('version', matched223.group('version'))

return result
} else {
return null
}
})
}

private void formSemver(data, config) {
def semver = data.major + "." + data.minor + "." + data.security

if (data.pre) {
semver += "-" + data.pre
}

def joiner = new java.util.StringJoiner('.')
joiner.add(data.build ?: "0")
joiner.add(config.adoptBuildNumber ?: "0")
semver += "+" + joiner.toString()
return semver
}

def formVersionData(config) {

def data = [:]

if (config.adoptBuildNumber) {
data.put('adopt_build_number', config.adoptBuildNumber)
}

if (config.parameters.TAG != null) {
def pre223 = matchPre223(config.parameters.TAG)
if (pre223 != null) {
data = pre223
} else {
data = match223(config.parameters.TAG)
}
}

data.put('semver', formSemver(data, config))


return data
}


def determineTestJobName(config, testType) {

def variant
def number = getJavaVersionNumber(config.javaVersion)

if (config.variant == "openj9") {
variant = "j9"
Expand Down Expand Up @@ -144,11 +240,70 @@ def sign(config) {
}
}

def listArchives() {
return sh(
script: """find workspace/target/ | egrep '.tar.gz|.zip'""",
returnStdout: true,
returnStatus: false
)
.trim()
.split('\n')
}

def writeMetadata(config, filesCreated) {

def buildMetadata = [
os : config.os,
arch : config.arch,
variant : config.variant,
version : config.javaVersion,
tag : config.parameters.TAG,
version_data: formVersionData(config)
]

def type = "jdk";
if (file.contains("-jre")) {
type = "jre";
}

/*
example data:
{
"os": "linux",
"arch": "x64",
"variant": "hotspot",
"version": "jdk8u",
"tag": "jdk8u202-b08",
"version_data": {
"adopt_build_number": 2,
"major": 8,
"minor": 0,
"security": 202,
"build": 8,
"version": "8u202-b08"
},
"binary_type": "jdk"
}
*/
node("master") {
filesCreated.each({ file ->

data = buildMetadata.clone()
data.put("binary_type", type)

writeFile file: "${file}.json", text: JsonOutput.prettyPrint(JsonOutput.toJson(data))
})
archiveArtifacts artifacts: "workspace/target/**/*.json"
}
}

try {
def config = new JsonSlurper().parseText("${TEST_CONFIG}")
println "Executing tests: ${config}"
println "Build num: ${env.BUILD_NUMBER}"

def filesCreated = [];

def enableTests = Boolean.valueOf(ENABLE_TESTS)
def cleanWorkspace = Boolean.valueOf(CLEAN_WORKSPACE)

Expand All @@ -164,6 +319,7 @@ try {
try {
sh "./build-farm/make-adopt-build-farm.sh"
archiveArtifacts artifacts: "workspace/target/*"
filesCreated = listArchives()
} finally {
if (config.os == "aix") {
cleanWs notFailBuild: true
Expand All @@ -186,9 +342,10 @@ try {
}
}

writeMetadata(config, filesCreated)

// Sign and archive jobs if needed
sign(config)

} catch (Exception e) {
currentBuild.result = 'FAILURE'
println "Execution error: " + e.getMessage()
Expand Down
Loading

0 comments on commit fac408c

Please sign in to comment.