Unicode error when creating charm from Ansible template

[szeestraten]

Checklist

• Confirmed this is an issue with charm-tools, not charmstore-client
• Provide versions of tools used
• Described the feature or ways to replicate the issue

What version am I running?

I ran the following command: charm version and got the following ouput:

charm 2.1.1-0ubuntu1
charm-tools 2.1.2

I am using:

Distributor ID: Ubuntu
Description:    Ubuntu 16.04 LTS
Release:    16.04
Codename:   xenial

Issue/Feature

Creating a new charm from the Ansible template results in a UnicodeDecodeError in the Cheetah template engine.

I expect/expected the following

I expected to create a new charm from the Ansible template by running the following:

charm create mycharm -t ansible

What I got

I got a traceback with a UnicodeDecodeError and no charm.

INFO: Generating charm for mycharm in ./mycharm
INFO: No mycharm in apt cache; creating an empty charm instead.

Traceback (most recent call last):
  File "/usr/bin/charm-create", line 9, in <module>
    load_entry_point('charm-tools==2.1.2', 'console_scripts', 'charm-create')()
  File "/usr/lib/python2.7/dist-packages/charmtools/create.py", line 98, in main
    generator.create_charm()
  File "/usr/lib/python2.7/dist-packages/charmtools/generators/generator.py", line 74, in create_charm
    self.plugin.create_charm(user_config, tempdir)
  File "/usr/lib/python2.7/dist-packages/charmtools/templates/ansible/template.py", line 47, in create_charm
    self._template_file(config, path.join(root, outfile))
  File "/usr/lib/python2.7/dist-packages/charmtools/templates/ansible/template.py", line 64, in _template_file
    t = Template(file=outfile, searchList=(config))
  File "/usr/lib/python2.7/dist-packages/Cheetah/Template.py", line 1259, in __init__
    self._compile(source, file, compilerSettings=compilerSettings)
  File "/usr/lib/python2.7/dist-packages/Cheetah/Template.py", line 1553, in _compile
    keepRefToGeneratedCode=True)
  File "/usr/lib/python2.7/dist-packages/Cheetah/Template.py", line 740, in compile
    settings=(compilerSettings or {}))
  File "/usr/lib/python2.7/dist-packages/Cheetah/Compiler.py", line 1580, in __init__
    source = unicode(source)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf3 in position 1: ordinal not in range(128)

I managed to reproduce the issue on a clean install.

What I think is wrong

I added a print statement in charmtools/templates/ansible/template.py to print out the outfile variable to see which files are running through Cheetah and it looks like it is going through the following .pyc files:

• /unit_tests/test_hooks.pyc
• /scripts/charm_helpers_sync.pyc
• /hooks/hooks.pyc

So I changed the skip_parsing list in charmtools/templates/ansible/template.py to:

skip_parsing = ['README.ex', 'Makefile', 'test_hooks.pyc', 'charm_helpers_sync.pyc', 'hooks.pyc']

which successfully works around the issue.

Unfortunately I'm not sure what the underlying cause is in regards to why it is trying to parse the bytecode files.

[marcoceppi]

Thanks for the bug! it's odd that pyc files are being delivered at all in this. While you fix definitely address this, we'll look into getting a RCA done and getting a patch out before end of week.

[lazypower]

@szeestraten Great catch on the bug. There's also been additional straw man work to deliver an ansible template base layer in reactive. I gave a talk over this at the charmer summit:

And the resulting code was pushed here: https://github.com/chuckbutler/ansible-base.

I urge you to use this as your starting point so you're starting out in the reactive paradigm, vs classic hook-based charming.

If you'd like to talk more about this I'm happy to setup some time to coordinate and dissect how the ansible base layer was put together.

@marcoceppi I filed #228 and will look into updating our Ansible template

[szeestraten]

Thanks guys!

@chuckbutler The talk and code are very much appreciated. I'll start there and get in touch if needed.