Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix transfer of large data #43

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
fix transfer of large data
Data larger than the max integer value would fail to transfer. changed
size of message read from the IPC protocol from an int to an unsigned
int. Also changed to load large data in chunks since stream.read() does
not support values larger than max int.
  • Loading branch information
Derek Wisong committed Oct 1, 2016
commit 9a0725a6d2f769477ed3724f4e0595f1cab3a0b5
33 changes: 31 additions & 2 deletions qpython/qreader.py
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ def read_header(self, source = None):
# skip 1 byte
self._buffer.skip()

message_size = self._buffer.get_int()
message_size = self._buffer.get_uint()
return QMessage(None, message_type, message_size, message_compressed)


Expand Down Expand Up @@ -384,8 +384,28 @@ def _read_bytes(self, length):

if length == 0:
return b''
else:
if length <= sys.maxint:
data = self._stream.read(length)
else:
try:
from cStringIO import StringIO
except:
from StringIO import StringIO

# for large messages, read from the stream in chunks
remaining = length
buff = StringIO()

while remaining > 0:
chunk = self._stream.read(min(remaining, 2048))

if chunk:
remaining = remaining - len(chunk)
buff.write(chunk)
else:
break

data = buff.getvalue()

if len(data) == 0:
raise QReaderException('Error while reading data')
Expand Down Expand Up @@ -491,6 +511,15 @@ def get_byte(self):
return self.get('b')


def get_uint(self):
'''
Gets a single unsigned 32-bit integer from the buffer.

:returns: single unsigned integer
'''
return self.get('I')


def get_int(self):
'''
Gets a single 32-bit integer from the buffer.
Expand Down