Skip to content

Commit

Permalink
[opt-viewer] Don't Decode HTML bytes for Python 2
Browse files Browse the repository at this point in the history
Summary:
D36624 added some python3 compatibility. But that fix has a problem..

With python2 (which is specified by `#!/usr/bin/env python2.7`), if the env variables do not specify the UTF8,
and the source file is UTF8 (contains non-ASCII symbols), then the `.decode('utf-8')` causes the following exception:
```
Reading YAML files...
Rendering HTML files...
        8 of 41Traceback (most recent call last):
  File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 277, in <module>
    print_progress)
  File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 213, in generate_report
    should_print_progress)
  File "/build/llvm/tools/opt-viewer/optpmap.py", line 45, in pmap
    result = map(_wrapped_func, func_and_args, *args, **kwargs)
  File "/build/llvm/tools/opt-viewer/optpmap.py", line 25, in _wrapped_func
    return func(argument)
  File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 174, in _render_file
    SourceFileRenderer(source_dir, output_dir, filename).render(remarks)
  File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 125, in render
    self.render_source_lines(self.source_stream, line_remarks)
  File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 79, in render_source_lines
    </tr>'''.format(**locals()), file=self.stream)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf4' in position 47: ordinal not in range(128)
```
This is similar to https://bugs.llvm.org/show_bug.cgi?id=33548, which was fixed by https://reviews.llvm.org/D37661

Unlike that fix, here, *removing* `.decode('utf-8')` actually fixes it.

Since i assume that the original fix is needed, i simply made
that fix conditional, since for python2 it actually breaks things.

Reviewers: modocache, anemet

Reviewed By: anemet

Subscribers: fhahn, llvm-commits

Differential Revision: https://reviews.llvm.org/D38289

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@315350 91177308-0d34-0410-b5e6-96231b3b80d8
  • Loading branch information
LebedevRI committed Oct 10, 2017
1 parent d4ae77e commit cbdd389
Showing 1 changed file with 6 additions and 1 deletion.
7 changes: 6 additions & 1 deletion tools/opt-viewer/opt-viewer.py
Original file line number Diff line number Diff line change
@@ -10,6 +10,7 @@
import os.path
import re
import shutil
import sys

from pygments import highlight
from pygments.lexers.c_cpp import CppLexer
@@ -62,7 +63,11 @@ def render_source_lines(self, stream, line_remarks):
html_highlighted = highlight(
file_text,
self.cpp_lexer,
self.html_formatter).decode('utf-8')
self.html_formatter)

# On Python 3, pygments.highlight() returns a bytes object, not a str.
if sys.version_info >= (3, 0):
html_highlighted = html_highlighted.decode('utf-8')

# Take off the header and footer, these must be
# reapplied line-wise, within the page structure

0 comments on commit cbdd389

Please sign in to comment.