Mercurial > kallithea

annotate rhodecode/lib/vcs/subprocessio.py @ 3895:e39fb661998b beta

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
small docstring fixes
author Marcin Kuzminski <marcin@python-works.com>
date Wed, 22 May 2013 22:46:51 +0200
parents b84c83b651de
children ffd45b185016
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
1 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
2 Module provides a class allowing to wrap communication over subprocess.Popen
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
3 input, output, error streams into a meaningfull, non-blocking, concurrent
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
4 stream processor exposing the output data as an iterator fitting to be a
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
5 return value passed by a WSGI applicaiton to a WSGI server per PEP 3333.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
6
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
7 Copyright (c) 2011 Daniel Dotsenko <dotsa@hotmail.com>
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
8
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
9 This file is part of git_http_backend.py Project.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
10
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
11 git_http_backend.py Project is free software: you can redistribute it and/or
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
12 modify it under the terms of the GNU Lesser General Public License as
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
13 published by the Free Software Foundation, either version 2.1 of the License,
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
14 or (at your option) any later version.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
15
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
16 git_http_backend.py Project is distributed in the hope that it will be useful,
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
17 but WITHOUT ANY WARRANTY; without even the implied warranty of
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
18 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
19 GNU Lesser General Public License for more details.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
20
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
21 You should have received a copy of the GNU Lesser General Public License
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
22 along with git_http_backend.py Project.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
23 If not, see <http://www.gnu.org/licenses/>.
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
24 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
25 import os
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
26 import subprocess
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
27 from rhodecode.lib.vcs.utils.compat import deque, Event, Thread, _bytes, _bytearray
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
28
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
29
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
30 class StreamFeeder(Thread):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
31 """
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
32 Normal writing into pipe-like is blocking once the buffer is filled.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
33 This thread allows a thread to seep data from a file-like into a pipe
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
34 without blocking the main thread.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
35 We close inpipe once the end of the source stream is reached.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
36 """
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
37 def __init__(self, source):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
38 super(StreamFeeder, self).__init__()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
39 self.daemon = True
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
40 filelike = False
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
41 self.bytes = _bytes()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
42 if type(source) in (type(''), _bytes, _bytearray): # string-like
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
43 self.bytes = _bytes(source)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
44 else: # can be either file pointer or file-like
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
45 if type(source) in (int, long): # file pointer it is
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
46 ## converting file descriptor (int) stdin into file-like
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
47 try:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
48 source = os.fdopen(source, 'rb', 16384)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
49 except Exception:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
50 pass
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
51 # let's see if source is file-like by now
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
52 try:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
53 filelike = source.read
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
54 except Exception:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
55 pass
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
56 if not filelike and not self.bytes:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
57 raise TypeError("StreamFeeder's source object must be a readable "
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
58 "file-like, a file descriptor, or a string-like.")
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
59 self.source = source
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
60 self.readiface, self.writeiface = os.pipe()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
61
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
62 def run(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
63 t = self.writeiface
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
64 if self.bytes:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
65 os.write(t, self.bytes)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
66 else:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
67 s = self.source
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
68 b = s.read(4096)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
69 while b:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
70 os.write(t, b)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
71 b = s.read(4096)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
72 os.close(t)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
73
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
74 @property
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
75 def output(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
76 return self.readiface
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
77
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
78
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
79 class InputStreamChunker(Thread):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
80 def __init__(self, source, target, buffer_size, chunk_size):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
81
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
82 super(InputStreamChunker, self).__init__()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
83
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
84 self.daemon = True # die die die.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
85
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
86 self.source = source
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
87 self.target = target
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
88 self.chunk_count_max = int(buffer_size / chunk_size) + 1
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
89 self.chunk_size = chunk_size
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
90
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
91 self.data_added = Event()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
92 self.data_added.clear()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
93
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
94 self.keep_reading = Event()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
95 self.keep_reading.set()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
96
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
97 self.EOF = Event()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
98 self.EOF.clear()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
99
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
100 self.go = Event()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
101 self.go.set()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
102
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
103 def stop(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
104 self.go.clear()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
105 self.EOF.set()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
106 try:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
107 # this is not proper, but is done to force the reader thread let
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
108 # go of the input because, if successful, .close() will send EOF
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
109 # down the pipe.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
110 self.source.close()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
111 except:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
112 pass
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
113
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
114 def run(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
115 s = self.source
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
116 t = self.target
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
117 cs = self.chunk_size
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
118 ccm = self.chunk_count_max
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
119 kr = self.keep_reading
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
120 da = self.data_added
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
121 go = self.go
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
122
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
123 try:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
124 b = s.read(cs)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
125 except ValueError:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
126 b = ''
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
127
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
128 while b and go.is_set():
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
129 if len(t) > ccm:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
130 kr.clear()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
131 kr.wait(2)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
132 # # this only works on 2.7.x and up
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
133 # if not kr.wait(10):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
134 # raise Exception("Timed out while waiting for input to be read.")
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
135 # instead we'll use this
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
136 if len(t) > ccm + 3:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
137 raise IOError("Timed out while waiting for input from subprocess.")
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
138 t.append(b)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
139 da.set()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
140 b = s.read(cs)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
141 self.EOF.set()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
142 da.set() # for cases when done but there was no input.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
143
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
144
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
145 class BufferedGenerator():
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
146 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
147 Class behaves as a non-blocking, buffered pipe reader.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
148 Reads chunks of data (through a thread)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
149 from a blocking pipe, and attaches these to an array (Deque) of chunks.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
150 Reading is halted in the thread when max chunks is internally buffered.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
151 The .next() may operate in blocking or non-blocking fashion by yielding
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
152 '' if no data is ready
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
153 to be sent or by not returning until there is some data to send
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
154 When we get EOF from underlying source pipe we raise the marker to raise
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
155 StopIteration after the last chunk of data is yielded.
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
156 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
157
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
158 def __init__(self, source, buffer_size=65536, chunk_size=4096,
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
159 starting_values=[], bottomless=False):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
160
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
161 if bottomless:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
162 maxlen = int(buffer_size / chunk_size)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
163 else:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
164 maxlen = None
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
165
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
166 self.data = deque(starting_values, maxlen)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
167
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
168 self.worker = InputStreamChunker(source, self.data, buffer_size,
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
169 chunk_size)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
170 if starting_values:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
171 self.worker.data_added.set()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
172 self.worker.start()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
173
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
174 ####################
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
175 # Generator's methods
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
176 ####################
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
177
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
178 def __iter__(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
179 return self
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
180
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
181 def next(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
182 while not len(self.data) and not self.worker.EOF.is_set():
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
183 self.worker.data_added.clear()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
184 self.worker.data_added.wait(0.2)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
185 if len(self.data):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
186 self.worker.keep_reading.set()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
187 return _bytes(self.data.popleft())
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
188 elif self.worker.EOF.is_set():
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
189 raise StopIteration
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
190
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
191 def throw(self, type, value=None, traceback=None):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
192 if not self.worker.EOF.is_set():
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
193 raise type(value)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
194
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
195 def start(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
196 self.worker.start()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
197
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
198 def stop(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
199 self.worker.stop()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
200
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
201 def close(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
202 try:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
203 self.worker.stop()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
204 self.throw(GeneratorExit)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
205 except (GeneratorExit, StopIteration):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
206 pass
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
207
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
208 def __del__(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
209 self.close()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
210
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
211 ####################
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
212 # Threaded reader's infrastructure.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
213 ####################
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
214 @property
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
215 def input(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
216 return self.worker.w
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
217
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
218 @property
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
219 def data_added_event(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
220 return self.worker.data_added
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
221
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
222 @property
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
223 def data_added(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
224 return self.worker.data_added.is_set()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
225
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
226 @property
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
227 def reading_paused(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
228 return not self.worker.keep_reading.is_set()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
229
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
230 @property
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
231 def done_reading_event(self):
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
232 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
233 Done_reding does not mean that the iterator's buffer is empty.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
234 Iterator might have done reading from underlying source, but the read
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
235 chunks might still be available for serving through .next() method.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
236
3895
e39fb661998b small docstring fixes
Marcin Kuzminski <marcin@python-works.com>
parents: 3889
diff changeset
237 :returns: An Event class instance.
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
238 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
239 return self.worker.EOF
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
240
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
241 @property
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
242 def done_reading(self):
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
243 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
244 Done_reding does not mean that the iterator's buffer is empty.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
245 Iterator might have done reading from underlying source, but the read
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
246 chunks might still be available for serving through .next() method.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
247
3895
e39fb661998b small docstring fixes
Marcin Kuzminski <marcin@python-works.com>
parents: 3889
diff changeset
248 :returns: An Bool value.
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
249 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
250 return self.worker.EOF.is_set()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
251
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
252 @property
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
253 def length(self):
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
254 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
255 returns int.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
256
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
257 This is the lenght of the que of chunks, not the length of
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
258 the combined contents in those chunks.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
259
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
260 __len__() cannot be meaningfully implemented because this
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
261 reader is just flying throuh a bottomless pit content and
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
262 can only know the lenght of what it already saw.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
263
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
264 If __len__() on WSGI server per PEP 3333 returns a value,
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
265 the responce's length will be set to that. In order not to
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
266 confuse WSGI PEP3333 servers, we will not implement __len__
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
267 at all.
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
268 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
269 return len(self.data)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
270
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
271 def prepend(self, x):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
272 self.data.appendleft(x)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
273
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
274 def append(self, x):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
275 self.data.append(x)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
276
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
277 def extend(self, o):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
278 self.data.extend(o)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
279
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
280 def __getitem__(self, i):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
281 return self.data[i]
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
282
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
283
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
284 class SubprocessIOChunker(object):
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
285 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
286 Processor class wrapping handling of subprocess IO.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
287
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
288 In a way, this is a "communicate()" replacement with a twist.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
289
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
290 - We are multithreaded. Writing in and reading out, err are all sep threads.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
291 - We support concurrent (in and out) stream processing.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
292 - The output is not a stream. It's a queue of read string (bytes, not unicode)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
293 chunks. The object behaves as an iterable. You can "for chunk in obj:" us.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
294 - We are non-blocking in more respects than communicate()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
295 (reading from subprocess out pauses when internal buffer is full, but
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
296 does not block the parent calling code. On the flip side, reading from
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
297 slow-yielding subprocess may block the iteration until data shows up. This
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
298 does not block the parallel inpipe reading occurring parallel thread.)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
299
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
300 The purpose of the object is to allow us to wrap subprocess interactions into
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
301 and interable that can be passed to a WSGI server as the application's return
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
302 value. Because of stream-processing-ability, WSGI does not have to read ALL
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
303 of the subprocess's output and buffer it, before handing it to WSGI server for
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
304 HTTP response. Instead, the class initializer reads just a bit of the stream
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
305 to figure out if error ocurred or likely to occur and if not, just hands the
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
306 further iteration over subprocess output to the server for completion of HTTP
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
307 response.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
308
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
309 The real or perceived subprocess error is trapped and raised as one of
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
310 EnvironmentError family of exceptions
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
311
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
312 Example usage:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
313 # try:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
314 # answer = SubprocessIOChunker(
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
315 # cmd,
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
316 # input,
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
317 # buffer_size = 65536,
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
318 # chunk_size = 4096
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
319 # )
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
320 # except (EnvironmentError) as e:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
321 # print str(e)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
322 # raise e
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
323 #
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
324 # return answer
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
325
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
326
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
327 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
328 def __init__(self, cmd, inputstream=None, buffer_size=65536,
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
329 chunk_size=4096, starting_values=[], **kwargs):
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
330 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
331 Initializes SubprocessIOChunker
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
332
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
333 :param cmd: A Subprocess.Popen style "cmd". Can be string or array of strings
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
334 :param inputstream: (Default: None) A file-like, string, or file pointer.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
335 :param buffer_size: (Default: 65536) A size of total buffer per stream in bytes.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
336 :param chunk_size: (Default: 4096) A max size of a chunk. Actual chunk may be smaller.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
337 :param starting_values: (Default: []) An array of strings to put in front of output que.
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
338 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
339
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
340 if inputstream:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
341 input_streamer = StreamFeeder(inputstream)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
342 input_streamer.start()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
343 inputstream = input_streamer.output
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
344
3830
08d439bfbd8c fixed handling shell argument in subprocess calls, it always was hardcoded even when passed properly in arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 3797
diff changeset
345 _shell = kwargs.get('shell', True)
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
346 if isinstance(cmd, (list, tuple)):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
347 cmd = ' '.join(cmd)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
348
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
349 kwargs['shell'] = _shell
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
350 _p = subprocess.Popen(cmd,
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
351 bufsize=-1,
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
352 stdin=inputstream,
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
353 stdout=subprocess.PIPE,
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
354 stderr=subprocess.PIPE,
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
355 **kwargs
3830
08d439bfbd8c fixed handling shell argument in subprocess calls, it always was hardcoded even when passed properly in arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 3797
diff changeset
356 )
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
357
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
358 bg_out = BufferedGenerator(_p.stdout, buffer_size, chunk_size, starting_values)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
359 bg_err = BufferedGenerator(_p.stderr, 16000, 1, bottomless=True)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
360
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
361 while not bg_out.done_reading and not bg_out.reading_paused and not bg_err.length:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
362 # doing this until we reach either end of file, or end of buffer.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
363 bg_out.data_added_event.wait(1)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
364 bg_out.data_added_event.clear()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
365
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
366 # at this point it's still ambiguous if we are done reading or just full buffer.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
367 # Either way, if error (returned by ended process, or implied based on
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
368 # presence of stuff in stderr output) we error out.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
369 # Else, we are happy.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
370 _returncode = _p.poll()
3889
b84c83b651de replace equality comparision to None
Marcin Kuzminski <marcin@python-works.com>
parents: 3886
diff changeset
371 if _returncode or (_returncode is None and bg_err.length):
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
372 try:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
373 _p.terminate()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
374 except:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
375 pass
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
376 bg_out.stop()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
377 bg_err.stop()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
378 err = '%s' % ''.join(bg_err)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
379 if err:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
380 raise EnvironmentError("Subprocess exited due to an error:\n" + err)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
381 raise EnvironmentError("Subprocess exited with non 0 ret code:%s" % _returncode)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
382
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
383 self.process = _p
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
384 self.output = bg_out
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
385 self.error = bg_err
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
386
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
387 def __iter__(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
388 return self
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
389
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
390 def next(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
391 if self.process.poll():
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
392 err = '%s' % ''.join(self.error)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
393 raise EnvironmentError("Subprocess exited due to an error:\n" + err)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
394 return self.output.next()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
395
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
396 def throw(self, type, value=None, traceback=None):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
397 if self.output.length or not self.output.done_reading:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
398 raise type(value)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
399
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
400 def close(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
401 try:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
402 self.process.terminate()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
403 except:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
404 pass
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
405 try:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
406 self.output.close()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
407 except:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
408 pass
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
409 try:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
410 self.error.close()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
411 except:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
412 pass
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
413
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
414 def __del__(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
415 self.close()