Skip to content

Commit

Permalink
Restore the ability to bootstrap Ivy with a custom configuration file.
Browse files Browse the repository at this point in the history 
This is required when publishing with an Ivy plugin. For example, if Ivy is
configured to publish via SVN, the fm.last.ivy.plugins.ivysvnresolver plugin
will need to be loaded into Ivy.  If this is added to the main
ivysettings.xml, Ivy bootstrap will fail.  The solution is to let Ivy
bootstrap itself as per-usual, and at publish time, substitute in a custom
ivysettings.xml file that reverences (the already bootstrapped) Ivy+plugins.

Testing Done:
Test publishes in Twitter's monorepo.

build-support/bin/ci.sh mostly works (a handful of tests fail, not related to this change).

Travis running: https://travis-ci.org/pantsbuild/pants/builds/49415107

Bugs closed: pantsbuild#1032

Reviewed at https://rbcommons.com/s/twitter/r/1709/
  • Loading branch information
@areitz
areitz committed Feb 10, 2015
1 parent dbf24dd commit affdf17
Show file tree
Hide file tree
Showing 3 changed files with 122 additions and 99 deletions.
66 changes: 17 additions & 49 deletions src/docs/publish.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -142,55 +142,23 @@ maintain extra-carefully. You can
Authenticating to the Artifact Repository
-----------------------------------------

Your artifact repository probably doesn't accept anonymous uploads; you
probably need to authenticate (prove that you are really you). Depending
on how the artifact repository set up, Pants might need to interact the
authentication system. (Or it might not. E.g., if your system uses
Kerberos, when Pants invokes artifact-upload commands, Kerberos tickets
should work automatically.)

If Pants needs to provide your username and password, you can enable
this via Pants' `.netrc` support. Pants can parse [.netrc
files](http://www.gnu.org/software/inetutils/manual/html_node/The-_002enetrc-File.html)
to get a user's username and password on an artifact repository machine.
To make this work:

- Each user needs a `~/.netrc` file with a section that looks like:

machine maven.twttr.com
login sandy
password myamazingpa$sword

- One of your `BUILD` files needs a target that represents `netrc`
auth:

netrc = netrc()

credentials(
name = 'netrc',
username=netrc.getusername,
password=netrc.getpassword)

- Your `'auth'` section for that repository should refer to that
target:

[jar-publish]
workdir: %(pants_workdir)s/publish
repos: {
'public': {
# ivysettings.xml resolver to use for publishing
# this resolver's address should match root of ivysettings resolver
'resolver': 'maven.twttr.com',
'confs': ['default', 'sources', 'docs', 'changelog'],
'auth': 'build-support:netrc',
'help': 'Configure your ~/.netrc for artifact repo access!'
},
}

If you need to implement some other kind of authentication, you might
look at [the Netrc
implementation](https://github.com/pantsbuild/pants/blob/master/src/python/pants/backend/authentication/netrc_util.py)
and the <a pantsref="bdict_credentials">`credentials`</a> target type for inspiration.
Your artifact repository probably doesn't accept anonymous uploads; you probably need to
authenticate to it (prove that you are really you). Depending on how the artifact repository is
configured, Pants might need to interact with an authentication system. (Or it might not. E.g., if
your system uses Kerberos, when Pants invokes artifact-upload commands, Kerberos tickets should
work automatically).

Your Pants administrator will handle configuring Pants to submit credentials to the artifact
repository. As a user, if Pants needs to provide your username and password, you can enable this
via Pants' `.netrc` support. Pants can parse [.netrc
files](http://www.gnu.org/software/inetutils/manual/html_node/The-_002enetrc-File.html) to get your
username and password to an artifact repository server. Here is an example file:

machine maven.example.com
login sandy
password myamazingpa$sword

Check with your Pants administrator for the proper hostname to use for the "machine" entry.

Troubleshooting
---------------
Expand Down
139 changes: 93 additions & 46 deletions src/docs/setup_repo.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -217,79 +217,126 @@ repository.
Enabling Pants Publish
----------------------

Pants can ease
[["publishing"|pants('src/docs:publish')]]: uploading versioned compiled
artifacts. There are some special things to set up to enable and
customize publishing.
Pants can ease [["publishing"|pants('src/docs:publish')]]: uploading versioned compiled artifacts.
There are some special things to set up to enable and customize publishing.

### Tell Pants about your Artifact Repository

To tell Pants which artifact repsitory to publish to,
To tell Pants which artifact repository to publish to, [[Create a
plugin|pants('src/python/pants/docs:howto_plugin')]] if you haven't already. Register it with Pants.

[[Create a plugin|pants('src/python/pants/docs:howto_plugin')]]
if you haven't already. Register it with Pants.

In the plugin, define and register a `Repository` in a `BUILD` file alias
as shown in
In the plugin, define and register at least one `Repository` object in a `BUILD` file alias as
shown in
[`src/python/internal_backend/repositories/register.py`](https://github.com/pantsbuild/pants/blob/master/src/python/internal_backend/repositories/register.py).

`BUILD` targets can use this Repository's alias as the `repo` parameter
to an <a pantsref="bdict_artifact">`artifact`</a>. For example,
`BUILD` targets can use this Repository's alias as the `repo` parameter to an <a
pantsref="bdict_artifact">`artifact`</a>. For example,
[examples/src/java/com/pants/examples/hello/greet/BUILD](https://github.com/pantsbuild/pants/blob/master/examples/src/java/com/pants/examples/hello/greet/BUILD)
refers to the `public` repository defined above. (Notice it's a Python object, not a string.)

!inc[start-at=java_library](../../examples/src/java/com/pants/examples/hello/greet/BUILD)

If you get an error that the repo name (here, `public`) isn't defined,
your plugin didn't register with Pants successfully. Make sure you
bootstrap Pants in a way that loads your `register.py`.
If you get an error that the repo name (here, `public`) isn't defined, your plugin didn't register
with Pants successfully. Make sure you bootstrap Pants in a way that loads your `register.py`.

In your config file (usually `pants.ini`), set up a `[jar-publish]`
section. In that section, create a `dict` called `repos`. It should
contain a section for each Repository:
In your config file (usually `pants.ini`), set up a `[jar-publish]` section. In that section,
create a `dict` called `repos`. It should contain a section for each `Repository` object that you
defined in your plugin:

repos: {
'public': { # must match the alias above
'resolver': 'maven.twttr.com', # must match URL above and <url> name
# in ivysettings.xml
'public': { # must match the name of the `Repository` object that you defined in your plugin.
'resolver': 'maven.example.com', # must match hostname in ~/.netrc and the <url> parameter
# in your custom ivysettings.xml.
'confs': ['default', 'sources', 'docs', 'changelog'],
# 'auth': 'build-support:netrc',
# 'help': 'Configure your ~/.netrc for artifactory access.
'auth': 'build-support:netrc', # Pants spec to a 'credentials()' object.
'help': 'Configure your ~/.netrc for maven.example.com access.'
},
'testing': { # this key must match the alias name above
'resolver': 'maven.twttr.com',
'testing': {
'resolver': 'artifactory.example.com',
'confs': ['default', 'sources', 'docs', 'changelog'],
# 'auth': 'build-support:netrc',
# 'help': 'Configure your ~/.netrc for artifactory access.
'auth': 'build-support:netrc',
'help': 'Configure your ~/.netrc for artifactory.example.com access.'
},
}

If your repository requires authentication, add a `~/.netrc` file. Here is a sample file, that
matches the `repos` specified above:

machine maven.example.com
login someuser
password password123

machine artifactory.example.com
login someuser
password someotherpassword123

And place the following in a `BUILD` file somewhere in your repository (`build-support/BUILD` is a
good place, and is used in the example above):

netrc = netrc()

credentials(
name = 'netrc',
username=netrc.getusername,
password=netrc.getpassword)

Next, tell Ivy how to publish to your repository. Add a new `ivysettings.xml` file to your repo
(for example: '`build-support/ivy/ivysettings_for_publishing.xml`'). Here is an example file to get
you started:

<?xml version="1.0"?>
<!-- pants.ini forces this settings file to be loaded by Ivy, but only at
publish time. -->

<ivysettings>
<settings defaultResolver="chain-repos"/>

<include file="${ivy.settings.dir}/ivysettings.xml"/>

<credentials host="artifactory.example.com"
realm="Artifactory Realm"
<!-- These values come from a credentials() object, which is fed by '~/.netrc'.
There must be a '~/.netrc' machine entry which matches a resolver in the
"repos" object in 'pants.ini', which also matches the 'host' in this XML
block. -->
username="${login}"
passwd="${password}"/>

<resolvers>
<chain name="chain-repos" returnFirst="true">
<_remote_resolvers name="remote-repos"/>
</chain>

<url name="artifactory.example.com" m2compatible="true">
<artifact pattern="https://artifactory.example.com/libs-releases-local/[organization]/[module]/[revision]/[module]-[revision](-[classifier]).[ext]"/>
</url>
</resolvers>
</ivysettings>

With this file in place, add a `[publish]` section to `pants.ini`, and tell pants to use
the custom Ivy settings when publishing:

ivy_settings: %(pants_supportdir)s/ivy/ivysettings_for_publishing.xml

<a pantsmark="setup_publish_restrict_branch"> </a>

### Restricting Publish to "Release Branch"

Your organization might have a notion of a special "release branch": you
want
[[artifact publishing|pants('src/docs:publish')]]
to happen on this source control
branch, which you maintain extra-carefully. You can set this branch
using the `restrict_push_branches` option of the `[jar-publish]` section
of your config file (usually `pants.ini`).
Your organization might have a notion of a special "release branch": you want [[artifact
publishing|pants('src/docs:publish')]] to happen on this source control branch, which you maintain
extra-carefully. You can set this branch using the `restrict_push_branches` option of the
`[jar-publish]` section of your config file (usually `pants.ini`).

### Task to Publish "Extra" Artifacts

Pants supports "publish plugins", which allow end-users to add
additional, arbitrary files to be published along with the primary
artifact. For example, let's say that along with publishing your jar
full of class files, you would also like to publish a companion file
that contains some metadata -- code coverage info, source git
repository, java version that created the jar, etc. By
[[developing a task|pants('src/python/pants/docs:dev_tasks')]]
in a [[plugin|pants('src/python/pants/docs:howto_plugin')]],
you give Pants a new ability.
[[Develop a Task to Publish "Extra" Artifacts|pants('src/python/pants/docs:dev_tasks_publish_extras')]]
to find out how to
develop a special Task to include "extra" data with published artifacts.
Pants supports "publish plugins", which allow end-users to add additional, arbitrary files to be
published along with the primary artifact. For example, let's say that along with publishing your
jar full of class files, you would also like to publish a companion file that contains some
metadata -- code coverage info, source git repository, java version that created the jar, etc. By
[[developing a task|pants('src/python/pants/docs:dev_tasks')]] in a
[[plugin|pants('src/python/pants/docs:howto_plugin')]], you give Pants a new ability. [[Develop a
Task to Publish "Extra" Artifacts|pants('src/python/pants/docs:dev_tasks_publish_extras')]] to find
out how to develop a special Task to include "extra" data with published artifacts.

Outside Caches
--------------
Expand Down
16 changes: 12 additions & 4 deletions src/python/pants/backend/jvm/tasks/jar_publish.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -401,6 +401,8 @@ def register_options(cls, register):
'maven coordinate [org]#[name] or target. '
'For example: --restart-at=com.twitter.common#quantity '
'Or: --restart-at=src/java/com/twitter/common/base')
register('--ivy_settings', default=None, advanced=True,
help='Specify a custom ivysettings.xml file to be used when publishing.')

@classmethod
def prepare(cls, options, round_manager):
Expand Down Expand Up @@ -937,15 +939,21 @@ def changelog(self, target, sha):
return ensure_text(self.scm.changelog(from_commit=sha,
files=target.sources_relative_to_buildroot()))

def fetch_ivysettings(self, ivy):
if self.get_options().ivy_settings:
return self.get_options().ivy_settings
elif ivy.ivy_settings is None:
raise TaskError('An ivysettings.xml with writeable resolvers is required for publishing, '
'but none was configured.')
else:
return ivy.ivy_settings

def generate_ivysettings(self, ivy, publishedjars, publish_local=None):
if ivy.ivy_settings is None:
raise TaskError('A custom ivysettings.xml with writeable resolvers is required for '
'publishing, but none was configured.')
template_relpath = os.path.join('templates', 'jar_publish', 'ivysettings.mustache')
template = pkgutil.get_data(__name__, template_relpath)
with safe_open(os.path.join(self.workdir, 'ivysettings.xml'), 'w') as wrapper:
generator = Generator(template,
ivysettings=ivy.ivy_settings,
ivysettings=self.fetch_ivysettings(ivy),
dir=self.workdir,
cachedir=self.cachedir,
published=[TemplateData(org=jar.org, name=jar.name)
Expand Down

0 comments on commit affdf17

Please sign in to comment.