Use os.fsdecode() and os.fsencode() for paths. Do not decode on python 2

README.md

@@ -2,7 +2,7 @@
 
 `inotify_simple` is a simple Python wrapper around
 [inotify](http://man7.org/linux/man-pages/man7/inotify.7.html).
-No fancy bells and whistles, just a literal wrapper with ctypes. Only 96
+No fancy bells and whistles, just a literal wrapper with ctypes. Only 108
 lines of code!
 
 `inotify_init()` is wrapped as a class that does little more than hold the
@@ -11,8 +11,8 @@ available data from the file descriptor and returns events as `namedtuple`
 objects after unpacking them with the `struct` module. `inotify_add_watch()`
 and `inotify_rm_watch()` are wrapped with no changes at all, taking and
 returning watch descriptor integers that calling code is expected to keep
-track of itself, just as one would use inotify from C. Works with Python 2 or
-3.
+track of itself, just as one would use inotify from C. Works with Python 2.7
+or Python >= 3.2.
 
 [View on PyPI](http://pypi.python.org/pypi/inotify_simple) |
 [Fork me on github](https://github.com/chrisjbillington/inotify_simple) |
@@ -24,13 +24,13 @@ track of itself, just as one would use inotify from C. Works with Python 2 or
 to install `inotify_simple`, run:
 
 ```
-$ pip install inotify_simple
+$ pip3 install inotify_simple
 ```
 
 or to install from source:
 
 ```
-$ python setup.py install
+$ python3 setup.py install
 ```
 
 Note:  If on Python < 3.4, you'll need the backported [enum34

doc/index.rst

@@ -12,7 +12,7 @@ inotify_simple |release|
 
 ``inotify_simple`` is a simple Python wrapper around
 `inotify <http://man7.org/linux/man-pages/man7/inotify.7.html>`_.
-No fancy bells and whistles, just a literal wrapper with ctypes. Only 96
+No fancy bells and whistles, just a literal wrapper with ctypes. Only 108
 lines of code!
 
 ``inotify_init()`` is wrapped as a class that does little more than hold the
@@ -22,7 +22,7 @@ available data from the file descriptor and returns events as a list of
 ``inotify_add_watch()`` and ``inotify_rm_watch()`` are wrapped with no changes
 at all, taking and returning watch descriptor integers that calling code is
 expected to keep track of itself, just as one would use inotify from C. Works
-with Python 2 or 3.
+with Python 2.7 or Python >= 3.2.
 
 `View on PyPI <http://pypi.python.org/pypi/inotify_simple>`_
 | `Fork me on GitHub <https://github.com/chrisjbillington/inotify_simple>`_
@@ -36,13 +36,13 @@ to install ``inotify_simple``, run:
 
 .. code-block:: bash
 
-    $ pip install inotify_simple
+    $ pip3 install inotify_simple
 
 or to install from source:
 
 .. code-block:: bash
 
-    $ python setup.py install
+    $ python3 setup.py install
 
 .. note::
     If on Python < 3.4, you'll need the backported `enum34 module.
@@ -130,12 +130,32 @@ Example usage
 Note that the flags, since they are defined with an ``enum.IntEnum``, print as
 what they are called rather than their integer values. However they are still
 just integers and so can be bitwise-ANDed and ORed etc with masks etc. The
-``flags.from_mask()`` method bitwise-ANDs a mask with all possible flags and
-returns a list of matches. This is for convenience and useful for debugging
-which events are coming through, but performance critical code should
-generally bitwise-AND masks with flags of interest itself so as to not do
-unnecessary checks.
+:func:`~inotify_simple.flags.from_mask` method bitwise-ANDs a mask with all
+possible flags and returns a list of matches. This is for convenience and
+useful for debugging which events are coming through, but performance critical
+code should generally bitwise-AND masks with flags of interest itself so as to
+not do unnecessary checks.
 
+.. note::
+    On Python 2, you should use ``inotify_simple`` with bytestrings (``str``).
+    You *can* pass ``unicode`` strings to
+    :func:`~inotify_simple.INotify.add_watch` if you like, and if you do they
+    will be encoded with the filesystem encoding before being passed to the
+    underlying C API. However, filesystems do not enforce that filepaths must
+    actually use the declared filesystem encoding, and so some filepaths may
+    not even be valid UTF8 or whatever your filesystem encoding is. Whilst the
+    encoding that :func:`~inotify_simple.INotify.add_watch` does will work
+    fine, your own code's decoding prior to that may break for some filepaths.
+    If you want your code to work with all filepaths, resist the temptation to
+    guess the encoding, and keep your paths as bytestrings. In Python 2,
+    :attr:`~inotify_simple.Event` namedtuples do not decode the 'name' field -
+    it is left as a bytestring.
+
+    Python 3.2 solved this problem with ``os.fsdecode()``, ``os.fsdecode()``,
+    which use the ``surrogateescape`` error handler, allowing incorrectly
+    encoded filepaths to survive the round trip of decoding and encoding
+    unchanged. So when using ``inotify_simple`` with Python 3, use ``str``
+    filepaths.
 
 ----------------
 Module reference
@@ -161,6 +181,6 @@ Full source code
 
 Free to copy and paste into your project subject to the simplified BSD
 license. Presented here for ease of verifying that this wrapper is as sensible
-as it claims to be.
+as it claims to be (comments stripped - see source on github to see comments).
 
 .. literalinclude:: fullsource.py

inotify_simple/inotify_simple.py

@@ -10,23 +10,37 @@
 from termios import FIONREAD
 from fcntl import ioctl
 
+
 if sys.version_info.major < 3:
-    #In 32-bit Python < 3 the inotify constants don't fit in an IntEnum and will
-    #cause an OverflowError. Overwiting the IntEnum with a LongEnum fixes this
-    #problem.
-    class LongEnum(long, enum.Enum):
-        """Enum where members are also (and must be) longs"""
-    EnumType = LongEnum
+    # For Python 2, we work with bytestrings. If the user passes in a unicode
+    # string, it will be encoded with the filesystem encoding before use:
+    _fsencoding = sys.getfilesystemencoding()
+    _fsencode = lambda s: s.encode(_fsencoding)
+    # And we will not decode bytestrings in inotify events, we will simply
+    # give the user bytestrings back:
+    _fsdecode = lambda s: s
+    # In 32-bit Python < 3 the inotify constants don't fit in an IntEnum and
+    # will cause an OverflowError. Overwiting the IntEnum with a LongEnum
+    # fixes this problem.
+    class LongEnum(long, enum.Enum): pass
+    _EnumType = LongEnum
+
 else:
-    EnumType = enum.IntEnum
-
+    # For Python 3, we work with (unicode) strings. We use os.fsencode and
+    # os.fsdecode, which are used by standard-library functions that return
+    # paths, and are able to round-trip possibly incorrectly encoded
+    # filepaths:
+    _fsencode = os.fsencode
+    _fsdecode = os.fsdecode
+    _EnumType = enum.IntEnum
+
 __all__ = ['flags', 'masks', 'parse_events', 'INotify', 'Event']
 
 _libc = ctypes.cdll.LoadLibrary('libc.so.6')
 _libc.__errno_location.restype = ctypes.POINTER(ctypes.c_int)
 
 def _libc_call(function, *args):
-    """Wrapper to which raises errors and retries on EINTR."""
+    """Wrapper which raises errors and retries on EINTR."""
     while True:
         rc = function(*args)
         if rc == -1:
@@ -57,14 +71,19 @@ def add_watch(self, path, mask):
         descriptor or raises an OSError on failure.
 
         Args:
-            path (str): The path to watch
+            path (str or bytes) / (python2: unicode or str): The path to watch.
+                If str in python3 or unicode in python2, will be encoded with
+                the filesystem encoding before being passed to
+                inotify_add_watch().
 
             mask (int): The mask of events to watch for. Can be constructed by
                 bitwise-ORing :class:`~inotify_simple.flags` together.
 
         Returns:
             int: watch descriptor"""
-        return _libc_call(_libc.inotify_add_watch, self.fd, path.encode('utf8'), mask)
+        if not isinstance(path, bytes):
+            path = _fsencode(path)
+        return _libc_call(_libc.inotify_add_watch, self.fd, path, mask)
 
     def rm_watch(self, wd):
         """Wrapper around ``inotify_rm_watch()``. Raises OSError on failure.
@@ -119,10 +138,11 @@ def __exit__(self, exc_type, exc_value, traceback):
 
 
 #: A ``namedtuple`` (wd, mask, cookie, name) for an inotify event.
-#: ``nemdtuple`` objects are very lightweight to instantiate and access, whilst
+#: ``namedtuple`` objects are very lightweight to instantiate and access, whilst
 #: being human readable when printed, which is useful for debugging and
 #: logging. For best performance, note that element access by index is about
-#: four times faster than by name.
+#: four times faster than by name. Note: in Python 2, name is a bytestring,
+#: not a unicode string. In Python 3 it is a string decoded with ``os.fsdecode()``.
 Event = collections.namedtuple('Event', ['wd', 'mask', 'cookie', 'name'])
 
 _EVENT_STRUCT_FORMAT = 'iIII'
@@ -146,13 +166,13 @@ def parse_events(data):
     while offset < buffer_size:
         wd, mask, cookie, namesize = struct.unpack_from(_EVENT_STRUCT_FORMAT, data, offset)
         offset += _EVENT_STRUCT_SIZE
-        name = ctypes.c_buffer(data[offset:offset + namesize], namesize).value.decode('utf8')
+        name = _fsdecode(ctypes.c_buffer(data[offset:offset + namesize], namesize).value)
         offset += namesize
         events.append(Event(wd, mask, cookie, name))
     return events
 
 
-class flags(EnumType):
+class flags(_EnumType):
     """Inotify flags as defined in ``inotify.h`` but with ``IN_`` prefix
     omitted. Includes a convenience method for extracting flags from a mask.
     """
@@ -186,7 +206,7 @@ def from_mask(cls, mask):
         return [flag for flag in cls.__members__.values() if flag & mask]
 
 
-class masks(EnumType):
+class masks(_EnumType):
     """Convenience masks as defined in ``inotify.h`` but with ``IN_`` prefix
     omitted."""
     #: helper event mask equal to ``flags.CLOSE_WRITE | flags.CLOSE_NOWRITE``

setup.py

@@ -8,7 +8,7 @@
 import os
 from distutils.core import setup
 
-__version__ = '1.0.4'
+__version__ = '1.1.0'
 
 DESCRIPTION = \
     """A simple wrapper around inotify. No fancy bells and whistles, just a