Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parsing (seemingly) normal file results in error #162

Open
dusktreader opened this issue May 30, 2018 · 1 comment
Open

Parsing (seemingly) normal file results in error #162

dusktreader opened this issue May 30, 2018 · 1 comment
Labels

Comments

@dusktreader
Copy link

When parsing this exact file:

"""Update MECC title parameters

Revision ID: c7f7c8ec8317
Revises: a804962ee102
Create Date: 2018-01-29 08:42:17.275163

"""

# revision identifiers, used by Alembic.
revision = 'c7f7c8ec8317'
down_revision = '155e6be59a0c'

from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects import postgresql

def upgrade():
    op.execute("update matcher_interfaces set noise_threshold = 0.2 where name = 'fuzzy_match'")
    op.execute("update mecc_targets set batch_size = 40000 where name = 'MATCHED_TO/title/title'") 

def downgrade():
    op.execute("update matcher_interfaces set noise_threshold = 0.9 where name = 'fuzzy_match'")
    op.execute("update mecc_targets set batch_size = 10000 where name = 'MATCHED_TO/title/title'") 

an IndexError is thrown:

In [1]: from redbaron import RedBaron

In [2]:

In [2]: with open('../cem-data/etc/alembic/versions/20180129-c7f7c8ec8317-update_mecc_title_parameters.py') as f:
   ...:     c=f.read()
   ...:

In [3]: RedBaron(c)
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-3-8c06eebe9290> in <module>()
----> 1 RedBaron(c)

~/.pyenv/versions/3.6.5/envs/ambix3.6/lib/python3.6/site-packages/redbaron/redbaron.py in __init__(self, source_code)
     34
     35         if isinstance(source_code, string_instance):
---> 36             self.node_list = base_nodes.NodeList.from_fst(baron.parse(source_code), parent=self, on_attribute="root")
     37             self.middle_separator = nodes.DotNode({"type": "endl", "formatting": [], "value": "\n", "indent": ""})
     38

~/.pyenv/versions/3.6.5/envs/ambix3.6/lib/python3.6/site-packages/baron/baron.py in parse(source_code, print_function)
     47
     48     if print_function is None:
---> 49         tokens = tokenize(source_code, False)
     50         print_function = has_print_function(tokens)
     51         if print_function:

~/.pyenv/versions/3.6.5/envs/ambix3.6/lib/python3.6/site-packages/baron/baron.py in tokenize(pouet, print_function)
     68
     69 def tokenize(pouet, print_function=False):
---> 70     return mark_indentation(inner_group(space_group(_tokenize(group(split(pouet)), print_function))))

~/.pyenv/versions/3.6.5/envs/ambix3.6/lib/python3.6/site-packages/baron/indentation_marker.py in mark_indentation(sequence)
     22
     23 def mark_indentation(sequence):
---> 24     return list(mark_indentation_generator(sequence))
     25
     26

~/.pyenv/versions/3.6.5/envs/ambix3.6/lib/python3.6/site-packages/baron/indentation_marker.py in mark_indentation_generator(sequence)
     83         # if we were in an indented situation and that the next line has a lower indentation
     84         if indentations and current[0] == "ENDL":
---> 85             the_indentation_level_changed = get_space(current) is None or get_space(current) != indentations[-1]
     86             if the_indentation_level_changed and iterator.show_next()[0] not in ("ENDL", "COMMENT"):
     87                 new_indent = get_space(current) if len(current) == 4 else ""

~/.pyenv/versions/3.6.5/envs/ambix3.6/lib/python3.6/site-packages/baron/indentation_marker.py in get_space(node)
     36     maybe not the best behavior but it seems to work for now.
     37     """
---> 38     if len(node) < 3 or len(node[3]) == 0:
     39         return None
     40     return transform_tabs_to_spaces(node[3][0][1])

IndexError: tuple index out of range
@dusktreader
Copy link
Author

It seems to be caused by trailing spaces that were left on lines 19 and 23 of the file. Removing those causes the file to parse fine

@Psycojoker Psycojoker added the bug label Aug 4, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants