Mercurial > kallithea
annotate rhodecode/lib/vcs/subprocessio.py @ 3997:156cb1cdd7ad
Added handling of copied files diff parsing
author | Marcin Kuzminski <marcin@python-works.com> |
---|---|
date | Sat, 15 Jun 2013 20:47:41 +0200 |
parents | e39fb661998b |
children | ffd45b185016 |
rev | line source |
---|---|
3886
a1696507b3ad
use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents:
3830
diff
changeset
|
1 """ |
3797
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
2 Module provides a class allowing to wrap communication over subprocess.Popen |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
3 input, output, error streams into a meaningfull, non-blocking, concurrent |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
4 stream processor exposing the output data as an iterator fitting to be a |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
5 return value passed by a WSGI applicaiton to a WSGI server per PEP 3333. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
6 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
7 Copyright (c) 2011 Daniel Dotsenko <dotsa@hotmail.com> |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
8 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
9 This file is part of git_http_backend.py Project. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
10 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
11 git_http_backend.py Project is free software: you can redistribute it and/or |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
12 modify it under the terms of the GNU Lesser General Public License as |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
13 published by the Free Software Foundation, either version 2.1 of the License, |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
14 or (at your option) any later version. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
15 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
16 git_http_backend.py Project is distributed in the hope that it will be useful, |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
17 but WITHOUT ANY WARRANTY; without even the implied warranty of |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
18 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
19 GNU Lesser General Public License for more details. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
20 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
21 You should have received a copy of the GNU Lesser General Public License |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
22 along with git_http_backend.py Project. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
23 If not, see <http://www.gnu.org/licenses/>. |
3886
a1696507b3ad
use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents:
3830
diff
changeset
|
24 """ |
3797
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
25 import os |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
26 import subprocess |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
27 from rhodecode.lib.vcs.utils.compat import deque, Event, Thread, _bytes, _bytearray |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
28 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
29 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
30 class StreamFeeder(Thread): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
31 """ |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
32 Normal writing into pipe-like is blocking once the buffer is filled. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
33 This thread allows a thread to seep data from a file-like into a pipe |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
34 without blocking the main thread. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
35 We close inpipe once the end of the source stream is reached. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
36 """ |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
37 def __init__(self, source): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
38 super(StreamFeeder, self).__init__() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
39 self.daemon = True |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
40 filelike = False |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
41 self.bytes = _bytes() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
42 if type(source) in (type(''), _bytes, _bytearray): # string-like |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
43 self.bytes = _bytes(source) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
44 else: # can be either file pointer or file-like |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
45 if type(source) in (int, long): # file pointer it is |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
46 ## converting file descriptor (int) stdin into file-like |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
47 try: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
48 source = os.fdopen(source, 'rb', 16384) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
49 except Exception: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
50 pass |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
51 # let's see if source is file-like by now |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
52 try: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
53 filelike = source.read |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
54 except Exception: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
55 pass |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
56 if not filelike and not self.bytes: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
57 raise TypeError("StreamFeeder's source object must be a readable " |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
58 "file-like, a file descriptor, or a string-like.") |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
59 self.source = source |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
60 self.readiface, self.writeiface = os.pipe() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
61 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
62 def run(self): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
63 t = self.writeiface |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
64 if self.bytes: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
65 os.write(t, self.bytes) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
66 else: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
67 s = self.source |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
68 b = s.read(4096) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
69 while b: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
70 os.write(t, b) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
71 b = s.read(4096) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
72 os.close(t) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
73 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
74 @property |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
75 def output(self): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
76 return self.readiface |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
77 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
78 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
79 class InputStreamChunker(Thread): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
80 def __init__(self, source, target, buffer_size, chunk_size): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
81 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
82 super(InputStreamChunker, self).__init__() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
83 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
84 self.daemon = True # die die die. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
85 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
86 self.source = source |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
87 self.target = target |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
88 self.chunk_count_max = int(buffer_size / chunk_size) + 1 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
89 self.chunk_size = chunk_size |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
90 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
91 self.data_added = Event() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
92 self.data_added.clear() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
93 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
94 self.keep_reading = Event() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
95 self.keep_reading.set() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
96 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
97 self.EOF = Event() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
98 self.EOF.clear() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
99 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
100 self.go = Event() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
101 self.go.set() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
102 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
103 def stop(self): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
104 self.go.clear() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
105 self.EOF.set() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
106 try: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
107 # this is not proper, but is done to force the reader thread let |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
108 # go of the input because, if successful, .close() will send EOF |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
109 # down the pipe. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
110 self.source.close() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
111 except: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
112 pass |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
113 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
114 def run(self): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
115 s = self.source |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
116 t = self.target |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
117 cs = self.chunk_size |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
118 ccm = self.chunk_count_max |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
119 kr = self.keep_reading |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
120 da = self.data_added |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
121 go = self.go |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
122 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
123 try: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
124 b = s.read(cs) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
125 except ValueError: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
126 b = '' |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
127 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
128 while b and go.is_set(): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
129 if len(t) > ccm: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
130 kr.clear() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
131 kr.wait(2) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
132 # # this only works on 2.7.x and up |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
133 # if not kr.wait(10): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
134 # raise Exception("Timed out while waiting for input to be read.") |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
135 # instead we'll use this |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
136 if len(t) > ccm + 3: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
137 raise IOError("Timed out while waiting for input from subprocess.") |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
138 t.append(b) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
139 da.set() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
140 b = s.read(cs) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
141 self.EOF.set() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
142 da.set() # for cases when done but there was no input. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
143 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
144 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
145 class BufferedGenerator(): |
3886
a1696507b3ad
use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents:
3830
diff
changeset
|
146 """ |
3797
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
147 Class behaves as a non-blocking, buffered pipe reader. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
148 Reads chunks of data (through a thread) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
149 from a blocking pipe, and attaches these to an array (Deque) of chunks. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
150 Reading is halted in the thread when max chunks is internally buffered. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
151 The .next() may operate in blocking or non-blocking fashion by yielding |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
152 '' if no data is ready |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
153 to be sent or by not returning until there is some data to send |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
154 When we get EOF from underlying source pipe we raise the marker to raise |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
155 StopIteration after the last chunk of data is yielded. |
3886
a1696507b3ad
use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents:
3830
diff
changeset
|
156 """ |
3797
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
157 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
158 def __init__(self, source, buffer_size=65536, chunk_size=4096, |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
159 starting_values=[], bottomless=False): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
160 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
161 if bottomless: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
162 maxlen = int(buffer_size / chunk_size) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
163 else: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
164 maxlen = None |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
165 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
166 self.data = deque(starting_values, maxlen) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
167 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
168 self.worker = InputStreamChunker(source, self.data, buffer_size, |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
169 chunk_size) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
170 if starting_values: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
171 self.worker.data_added.set() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
172 self.worker.start() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
173 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
174 #################### |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
175 # Generator's methods |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
176 #################### |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
177 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
178 def __iter__(self): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
179 return self |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
180 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
181 def next(self): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
182 while not len(self.data) and not self.worker.EOF.is_set(): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
183 self.worker.data_added.clear() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
184 self.worker.data_added.wait(0.2) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
185 if len(self.data): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
186 self.worker.keep_reading.set() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
187 return _bytes(self.data.popleft()) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
188 elif self.worker.EOF.is_set(): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
189 raise StopIteration |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
190 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
191 def throw(self, type, value=None, traceback=None): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
192 if not self.worker.EOF.is_set(): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
193 raise type(value) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
194 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
195 def start(self): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
196 self.worker.start() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
197 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
198 def stop(self): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
199 self.worker.stop() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
200 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
201 def close(self): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
202 try: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
203 self.worker.stop() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
204 self.throw(GeneratorExit) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
205 except (GeneratorExit, StopIteration): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
206 pass |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
207 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
208 def __del__(self): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
209 self.close() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
210 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
211 #################### |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
212 # Threaded reader's infrastructure. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
213 #################### |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
214 @property |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
215 def input(self): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
216 return self.worker.w |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
217 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
218 @property |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
219 def data_added_event(self): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
220 return self.worker.data_added |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
221 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
222 @property |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
223 def data_added(self): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
224 return self.worker.data_added.is_set() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
225 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
226 @property |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
227 def reading_paused(self): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
228 return not self.worker.keep_reading.is_set() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
229 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
230 @property |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
231 def done_reading_event(self): |
3886
a1696507b3ad
use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents:
3830
diff
changeset
|
232 """ |
3797
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
233 Done_reding does not mean that the iterator's buffer is empty. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
234 Iterator might have done reading from underlying source, but the read |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
235 chunks might still be available for serving through .next() method. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
236 |
3895
e39fb661998b
small docstring fixes
Marcin Kuzminski <marcin@python-works.com>
parents:
3889
diff
changeset
|
237 :returns: An Event class instance. |
3886
a1696507b3ad
use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents:
3830
diff
changeset
|
238 """ |
3797
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
239 return self.worker.EOF |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
240 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
241 @property |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
242 def done_reading(self): |
3886
a1696507b3ad
use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents:
3830
diff
changeset
|
243 """ |
3797
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
244 Done_reding does not mean that the iterator's buffer is empty. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
245 Iterator might have done reading from underlying source, but the read |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
246 chunks might still be available for serving through .next() method. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
247 |
3895
e39fb661998b
small docstring fixes
Marcin Kuzminski <marcin@python-works.com>
parents:
3889
diff
changeset
|
248 :returns: An Bool value. |
3886
a1696507b3ad
use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents:
3830
diff
changeset
|
249 """ |
3797
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
250 return self.worker.EOF.is_set() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
251 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
252 @property |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
253 def length(self): |
3886
a1696507b3ad
use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents:
3830
diff
changeset
|
254 """ |
3797
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
255 returns int. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
256 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
257 This is the lenght of the que of chunks, not the length of |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
258 the combined contents in those chunks. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
259 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
260 __len__() cannot be meaningfully implemented because this |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
261 reader is just flying throuh a bottomless pit content and |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
262 can only know the lenght of what it already saw. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
263 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
264 If __len__() on WSGI server per PEP 3333 returns a value, |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
265 the responce's length will be set to that. In order not to |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
266 confuse WSGI PEP3333 servers, we will not implement __len__ |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
267 at all. |
3886
a1696507b3ad
use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents:
3830
diff
changeset
|
268 """ |
3797
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
269 return len(self.data) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
270 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
271 def prepend(self, x): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
272 self.data.appendleft(x) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
273 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
274 def append(self, x): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
275 self.data.append(x) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
276 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
277 def extend(self, o): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
278 self.data.extend(o) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
279 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
280 def __getitem__(self, i): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
281 return self.data[i] |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
282 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
283 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
284 class SubprocessIOChunker(object): |
3886
a1696507b3ad
use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents:
3830
diff
changeset
|
285 """ |
3797
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
286 Processor class wrapping handling of subprocess IO. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
287 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
288 In a way, this is a "communicate()" replacement with a twist. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
289 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
290 - We are multithreaded. Writing in and reading out, err are all sep threads. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
291 - We support concurrent (in and out) stream processing. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
292 - The output is not a stream. It's a queue of read string (bytes, not unicode) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
293 chunks. The object behaves as an iterable. You can "for chunk in obj:" us. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
294 - We are non-blocking in more respects than communicate() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
295 (reading from subprocess out pauses when internal buffer is full, but |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
296 does not block the parent calling code. On the flip side, reading from |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
297 slow-yielding subprocess may block the iteration until data shows up. This |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
298 does not block the parallel inpipe reading occurring parallel thread.) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
299 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
300 The purpose of the object is to allow us to wrap subprocess interactions into |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
301 and interable that can be passed to a WSGI server as the application's return |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
302 value. Because of stream-processing-ability, WSGI does not have to read ALL |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
303 of the subprocess's output and buffer it, before handing it to WSGI server for |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
304 HTTP response. Instead, the class initializer reads just a bit of the stream |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
305 to figure out if error ocurred or likely to occur and if not, just hands the |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
306 further iteration over subprocess output to the server for completion of HTTP |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
307 response. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
308 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
309 The real or perceived subprocess error is trapped and raised as one of |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
310 EnvironmentError family of exceptions |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
311 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
312 Example usage: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
313 # try: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
314 # answer = SubprocessIOChunker( |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
315 # cmd, |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
316 # input, |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
317 # buffer_size = 65536, |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
318 # chunk_size = 4096 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
319 # ) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
320 # except (EnvironmentError) as e: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
321 # print str(e) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
322 # raise e |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
323 # |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
324 # return answer |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
325 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
326 |
3886
a1696507b3ad
use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents:
3830
diff
changeset
|
327 """ |
3797
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
328 def __init__(self, cmd, inputstream=None, buffer_size=65536, |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
329 chunk_size=4096, starting_values=[], **kwargs): |
3886
a1696507b3ad
use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents:
3830
diff
changeset
|
330 """ |
3797
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
331 Initializes SubprocessIOChunker |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
332 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
333 :param cmd: A Subprocess.Popen style "cmd". Can be string or array of strings |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
334 :param inputstream: (Default: None) A file-like, string, or file pointer. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
335 :param buffer_size: (Default: 65536) A size of total buffer per stream in bytes. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
336 :param chunk_size: (Default: 4096) A max size of a chunk. Actual chunk may be smaller. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
337 :param starting_values: (Default: []) An array of strings to put in front of output que. |
3886
a1696507b3ad
use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents:
3830
diff
changeset
|
338 """ |
3797
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
339 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
340 if inputstream: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
341 input_streamer = StreamFeeder(inputstream) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
342 input_streamer.start() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
343 inputstream = input_streamer.output |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
344 |
3830
08d439bfbd8c
fixed handling shell argument in subprocess calls, it always was hardcoded even when passed properly in arguments
Marcin Kuzminski <marcin@python-works.com>
parents:
3797
diff
changeset
|
345 _shell = kwargs.get('shell', True) |
3797
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
346 if isinstance(cmd, (list, tuple)): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
347 cmd = ' '.join(cmd) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
348 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
349 kwargs['shell'] = _shell |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
350 _p = subprocess.Popen(cmd, |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
351 bufsize=-1, |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
352 stdin=inputstream, |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
353 stdout=subprocess.PIPE, |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
354 stderr=subprocess.PIPE, |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
355 **kwargs |
3830
08d439bfbd8c
fixed handling shell argument in subprocess calls, it always was hardcoded even when passed properly in arguments
Marcin Kuzminski <marcin@python-works.com>
parents:
3797
diff
changeset
|
356 ) |
3797
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
357 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
358 bg_out = BufferedGenerator(_p.stdout, buffer_size, chunk_size, starting_values) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
359 bg_err = BufferedGenerator(_p.stderr, 16000, 1, bottomless=True) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
360 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
361 while not bg_out.done_reading and not bg_out.reading_paused and not bg_err.length: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
362 # doing this until we reach either end of file, or end of buffer. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
363 bg_out.data_added_event.wait(1) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
364 bg_out.data_added_event.clear() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
365 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
366 # at this point it's still ambiguous if we are done reading or just full buffer. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
367 # Either way, if error (returned by ended process, or implied based on |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
368 # presence of stuff in stderr output) we error out. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
369 # Else, we are happy. |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
370 _returncode = _p.poll() |
3889
b84c83b651de
replace equality comparision to None
Marcin Kuzminski <marcin@python-works.com>
parents:
3886
diff
changeset
|
371 if _returncode or (_returncode is None and bg_err.length): |
3797
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
372 try: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
373 _p.terminate() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
374 except: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
375 pass |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
376 bg_out.stop() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
377 bg_err.stop() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
378 err = '%s' % ''.join(bg_err) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
379 if err: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
380 raise EnvironmentError("Subprocess exited due to an error:\n" + err) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
381 raise EnvironmentError("Subprocess exited with non 0 ret code:%s" % _returncode) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
382 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
383 self.process = _p |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
384 self.output = bg_out |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
385 self.error = bg_err |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
386 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
387 def __iter__(self): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
388 return self |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
389 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
390 def next(self): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
391 if self.process.poll(): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
392 err = '%s' % ''.join(self.error) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
393 raise EnvironmentError("Subprocess exited due to an error:\n" + err) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
394 return self.output.next() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
395 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
396 def throw(self, type, value=None, traceback=None): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
397 if self.output.length or not self.output.done_reading: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
398 raise type(value) |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
399 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
400 def close(self): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
401 try: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
402 self.process.terminate() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
403 except: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
404 pass |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
405 try: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
406 self.output.close() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
407 except: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
408 pass |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
409 try: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
410 self.error.close() |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
411 except: |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
412 pass |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
413 |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
414 def __del__(self): |
d7488551578e
synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
415 self.close() |