Skip to content

Commit

Permalink
Merge pull request flairNLP#1353 from jjjamie/patch-1
Browse files Browse the repository at this point in the history
Make load_big_file work with read-only file
  • Loading branch information
alanakbik authored Jan 14, 2020
2 parents bbd39cc + 46b2082 commit da01dff
Showing 1 changed file with 3 additions and 3 deletions.
6 changes: 3 additions & 3 deletions flair/file_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,16 +22,16 @@
logger = logging.getLogger("flair")


def load_big_file(f):
def load_big_file(f: str) -> mmap.mmap:
"""
Workaround for loading a big pickle file. Files over 2GB cause pickle errors on certin Mac and Windows distributions.
:param f:
:return:
"""
logger.info(f"loading file {f}")
with open(f, "r+b") as f_in:
with open(f, "rb") as f_in:
# mmap seems to be much more memory efficient
bf = mmap.mmap(f_in.fileno(), 0)
bf = mmap.mmap(f_in.fileno(), 0, access=mmap.ACCESS_READ)
f_in.close()
return bf

Expand Down

0 comments on commit da01dff

Please sign in to comment.