Skip to content

Commit

Permalink
Fix caching in python3 (apache#806)
Browse files Browse the repository at this point in the history
* caravel: fix visualization cache for python3

python3 wants bytes and not strings:

2016-07-22 10:36:09,474:INFO:root:Caching for the next 28800 seconds
2016-07-22 10:36:09,475:WARNING:root:Could not cache key 1eeb45f32960f0df0ad99a125bdaf199
2016-07-22 10:36:09,475:ERROR:root:'str' does not support the buffer interface
Traceback (most recent call last):
  File "/home/rm/caraveltest/venv/lib/python3.4/site-packages/caravel/viz.py", line 306, in get_json
    zlib.compress(self.json_dumps(payload)),
TypeError: 'str' does not support the buffer interface

Tested with memcached and pylibmc client library.

* docs: add note about using a proper memcached client library
  • Loading branch information
xrmx authored and mistercrunch committed Jul 22, 2016
1 parent 187149c commit 8b694dd
Show file tree
Hide file tree
Showing 2 changed files with 13 additions and 5 deletions.
12 changes: 9 additions & 3 deletions caravel/viz.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
from flask_babel import lazy_gettext as _
from markdown import markdown
import simplejson as json
from six import string_types
from six import string_types, PY3
from werkzeug.datastructures import ImmutableMultiDict, MultiDict
from werkzeug.urls import Href
from dateutil import relativedelta as rdelta
Expand Down Expand Up @@ -279,7 +279,10 @@ def get_json(self):
if payload:
is_cached = True
try:
payload = json.loads(zlib.decompress(payload))
cached_data = zlib.decompress(payload)
if PY3:
cached_data = cached_data.decode('utf-8')
payload = json.loads(cached_data)
except Exception as e:
logging.error("Error reading cache")
payload = None
Expand All @@ -302,9 +305,12 @@ def get_json(self):
logging.info("Caching for the next {} seconds".format(
cache_timeout))
try:
data = self.json_dumps(payload)
if PY3:
data = bytes(data, 'utf-8')
cache.set(
cache_key,
zlib.compress(self.json_dumps(payload)),
zlib.compress(data),
timeout=cache_timeout)
except Exception as e:
# cache.set call can fail if the backend is down or if
Expand Down
6 changes: 4 additions & 2 deletions docs/installation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -177,8 +177,10 @@ caching purpose. Configuring your caching backend is as easy as providing
a ``CACHE_CONFIG``, constant in your ``caravel_config.py`` that
complies with the Flask-Cache specifications.

Flask-Cache supports multiple caching backends (Redis, Memcache,
SimpleCache (in-memory), or the local filesystem).
Flask-Cache supports multiple caching backends (Redis, Memcached,
SimpleCache (in-memory), or the local filesystem). If you are going to use
Memcached please use the pylibmc client library as python-memcached does
not handle storing binary data correctly.

For setting your timeouts, this is done in the Caravel metadata and goes
up the "timeout searchpath", from your slice configuration, to your
Expand Down

0 comments on commit 8b694dd

Please sign in to comment.