Mercurial > kallithea

annotate rhodecode/lib/vcs/subprocessio.py @ 4116:ffd45b185016 rhodecode-2.2.5-gpl

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Imported some of the GPLv3'd changes from RhodeCode v2.2.5. This imports changes between changesets 21af6c4eab3d and 6177597791c2 in RhodeCode's original repository, including only changes to Python files and HTML. RhodeCode clearly licensed its changes to these files under GPLv3 in their /LICENSE file, which states the following: The Python code and integrated HTML are licensed under the GPLv3 license. (See: https://code.rhodecode.com/rhodecode/files/v2.2.5/LICENSE or http://web.archive.org/web/20140512193334/https://code.rhodecode.com/rhodecode/files/f3b123159901f15426d18e3dc395e8369f70ebe0/LICENSE for an online copy of that LICENSE file) Conservancy reviewed these changes and confirmed that they can be licensed as a whole to the Kallithea project under GPLv3-only. While some of the contents committed herein are clearly licensed GPLv3-or-later, on the whole we must assume the are GPLv3-only, since the statement above from RhodeCode indicates that they intend GPLv3-only as their license, per GPLv3ยง14 and other relevant sections of GPLv3.
author Bradley M. Kuhn <bkuhn@sfconservancy.org>
date Wed, 02 Jul 2014 19:03:13 -0400
parents e39fb661998b
children 7e5f8c12a3fc
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
1 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
2 Module provides a class allowing to wrap communication over subprocess.Popen
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
3 input, output, error streams into a meaningfull, non-blocking, concurrent
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
4 stream processor exposing the output data as an iterator fitting to be a
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
5 return value passed by a WSGI applicaiton to a WSGI server per PEP 3333.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
6
4116
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3895
diff changeset
7 Copyright (c) 2011 Daniel Dotsenko <dotsa[at]hotmail.com>
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
8
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
9 This file is part of git_http_backend.py Project.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
10
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
11 git_http_backend.py Project is free software: you can redistribute it and/or
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
12 modify it under the terms of the GNU Lesser General Public License as
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
13 published by the Free Software Foundation, either version 2.1 of the License,
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
14 or (at your option) any later version.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
15
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
16 git_http_backend.py Project is distributed in the hope that it will be useful,
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
17 but WITHOUT ANY WARRANTY; without even the implied warranty of
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
18 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
19 GNU Lesser General Public License for more details.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
20
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
21 You should have received a copy of the GNU Lesser General Public License
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
22 along with git_http_backend.py Project.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
23 If not, see <http://www.gnu.org/licenses/>.
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
24 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
25 import os
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
26 import subprocess
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
27 from rhodecode.lib.vcs.utils.compat import deque, Event, Thread, _bytes, _bytearray
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
28
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
29
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
30 class StreamFeeder(Thread):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
31 """
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
32 Normal writing into pipe-like is blocking once the buffer is filled.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
33 This thread allows a thread to seep data from a file-like into a pipe
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
34 without blocking the main thread.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
35 We close inpipe once the end of the source stream is reached.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
36 """
4116
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3895
diff changeset
37
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
38 def __init__(self, source):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
39 super(StreamFeeder, self).__init__()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
40 self.daemon = True
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
41 filelike = False
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
42 self.bytes = _bytes()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
43 if type(source) in (type(''), _bytes, _bytearray): # string-like
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
44 self.bytes = _bytes(source)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
45 else: # can be either file pointer or file-like
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
46 if type(source) in (int, long): # file pointer it is
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
47 ## converting file descriptor (int) stdin into file-like
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
48 try:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
49 source = os.fdopen(source, 'rb', 16384)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
50 except Exception:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
51 pass
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
52 # let's see if source is file-like by now
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
53 try:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
54 filelike = source.read
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
55 except Exception:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
56 pass
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
57 if not filelike and not self.bytes:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
58 raise TypeError("StreamFeeder's source object must be a readable "
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
59 "file-like, a file descriptor, or a string-like.")
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
60 self.source = source
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
61 self.readiface, self.writeiface = os.pipe()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
62
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
63 def run(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
64 t = self.writeiface
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
65 if self.bytes:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
66 os.write(t, self.bytes)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
67 else:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
68 s = self.source
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
69 b = s.read(4096)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
70 while b:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
71 os.write(t, b)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
72 b = s.read(4096)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
73 os.close(t)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
74
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
75 @property
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
76 def output(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
77 return self.readiface
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
78
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
79
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
80 class InputStreamChunker(Thread):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
81 def __init__(self, source, target, buffer_size, chunk_size):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
82
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
83 super(InputStreamChunker, self).__init__()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
84
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
85 self.daemon = True # die die die.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
86
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
87 self.source = source
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
88 self.target = target
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
89 self.chunk_count_max = int(buffer_size / chunk_size) + 1
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
90 self.chunk_size = chunk_size
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
91
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
92 self.data_added = Event()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
93 self.data_added.clear()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
94
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
95 self.keep_reading = Event()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
96 self.keep_reading.set()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
97
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
98 self.EOF = Event()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
99 self.EOF.clear()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
100
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
101 self.go = Event()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
102 self.go.set()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
103
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
104 def stop(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
105 self.go.clear()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
106 self.EOF.set()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
107 try:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
108 # this is not proper, but is done to force the reader thread let
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
109 # go of the input because, if successful, .close() will send EOF
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
110 # down the pipe.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
111 self.source.close()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
112 except:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
113 pass
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
114
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
115 def run(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
116 s = self.source
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
117 t = self.target
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
118 cs = self.chunk_size
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
119 ccm = self.chunk_count_max
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
120 kr = self.keep_reading
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
121 da = self.data_added
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
122 go = self.go
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
123
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
124 try:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
125 b = s.read(cs)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
126 except ValueError:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
127 b = ''
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
128
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
129 while b and go.is_set():
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
130 if len(t) > ccm:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
131 kr.clear()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
132 kr.wait(2)
4116
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3895
diff changeset
133 # # this only works on 2.7.x and up
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3895
diff changeset
134 # if not kr.wait(10):
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3895
diff changeset
135 # raise Exception("Timed out while waiting for input to be read.")
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
136 # instead we'll use this
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
137 if len(t) > ccm + 3:
4116
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3895
diff changeset
138 raise IOError(
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3895
diff changeset
139 "Timed out while waiting for input from subprocess.")
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
140 t.append(b)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
141 da.set()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
142 b = s.read(cs)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
143 self.EOF.set()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
144 da.set() # for cases when done but there was no input.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
145
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
146
4116
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3895
diff changeset
147 class BufferedGenerator(object):
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
148 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
149 Class behaves as a non-blocking, buffered pipe reader.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
150 Reads chunks of data (through a thread)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
151 from a blocking pipe, and attaches these to an array (Deque) of chunks.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
152 Reading is halted in the thread when max chunks is internally buffered.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
153 The .next() may operate in blocking or non-blocking fashion by yielding
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
154 '' if no data is ready
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
155 to be sent or by not returning until there is some data to send
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
156 When we get EOF from underlying source pipe we raise the marker to raise
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
157 StopIteration after the last chunk of data is yielded.
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
158 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
159
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
160 def __init__(self, source, buffer_size=65536, chunk_size=4096,
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
161 starting_values=[], bottomless=False):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
162
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
163 if bottomless:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
164 maxlen = int(buffer_size / chunk_size)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
165 else:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
166 maxlen = None
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
167
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
168 self.data = deque(starting_values, maxlen)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
169 self.worker = InputStreamChunker(source, self.data, buffer_size,
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
170 chunk_size)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
171 if starting_values:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
172 self.worker.data_added.set()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
173 self.worker.start()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
174
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
175 ####################
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
176 # Generator's methods
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
177 ####################
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
178
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
179 def __iter__(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
180 return self
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
181
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
182 def next(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
183 while not len(self.data) and not self.worker.EOF.is_set():
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
184 self.worker.data_added.clear()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
185 self.worker.data_added.wait(0.2)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
186 if len(self.data):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
187 self.worker.keep_reading.set()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
188 return _bytes(self.data.popleft())
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
189 elif self.worker.EOF.is_set():
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
190 raise StopIteration
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
191
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
192 def throw(self, type, value=None, traceback=None):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
193 if not self.worker.EOF.is_set():
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
194 raise type(value)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
195
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
196 def start(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
197 self.worker.start()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
198
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
199 def stop(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
200 self.worker.stop()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
201
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
202 def close(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
203 try:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
204 self.worker.stop()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
205 self.throw(GeneratorExit)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
206 except (GeneratorExit, StopIteration):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
207 pass
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
208
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
209 def __del__(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
210 self.close()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
211
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
212 ####################
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
213 # Threaded reader's infrastructure.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
214 ####################
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
215 @property
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
216 def input(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
217 return self.worker.w
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
218
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
219 @property
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
220 def data_added_event(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
221 return self.worker.data_added
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
222
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
223 @property
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
224 def data_added(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
225 return self.worker.data_added.is_set()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
226
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
227 @property
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
228 def reading_paused(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
229 return not self.worker.keep_reading.is_set()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
230
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
231 @property
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
232 def done_reading_event(self):
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
233 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
234 Done_reding does not mean that the iterator's buffer is empty.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
235 Iterator might have done reading from underlying source, but the read
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
236 chunks might still be available for serving through .next() method.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
237
3895
e39fb661998b small docstring fixes
Marcin Kuzminski <marcin@python-works.com>
parents: 3889
diff changeset
238 :returns: An Event class instance.
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
239 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
240 return self.worker.EOF
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
241
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
242 @property
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
243 def done_reading(self):
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
244 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
245 Done_reding does not mean that the iterator's buffer is empty.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
246 Iterator might have done reading from underlying source, but the read
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
247 chunks might still be available for serving through .next() method.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
248
3895
e39fb661998b small docstring fixes
Marcin Kuzminski <marcin@python-works.com>
parents: 3889
diff changeset
249 :returns: An Bool value.
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
250 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
251 return self.worker.EOF.is_set()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
252
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
253 @property
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
254 def length(self):
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
255 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
256 returns int.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
257
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
258 This is the lenght of the que of chunks, not the length of
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
259 the combined contents in those chunks.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
260
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
261 __len__() cannot be meaningfully implemented because this
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
262 reader is just flying throuh a bottomless pit content and
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
263 can only know the lenght of what it already saw.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
264
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
265 If __len__() on WSGI server per PEP 3333 returns a value,
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
266 the responce's length will be set to that. In order not to
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
267 confuse WSGI PEP3333 servers, we will not implement __len__
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
268 at all.
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
269 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
270 return len(self.data)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
271
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
272 def prepend(self, x):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
273 self.data.appendleft(x)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
274
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
275 def append(self, x):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
276 self.data.append(x)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
277
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
278 def extend(self, o):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
279 self.data.extend(o)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
280
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
281 def __getitem__(self, i):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
282 return self.data[i]
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
283
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
284
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
285 class SubprocessIOChunker(object):
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
286 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
287 Processor class wrapping handling of subprocess IO.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
288
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
289 In a way, this is a "communicate()" replacement with a twist.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
290
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
291 - We are multithreaded. Writing in and reading out, err are all sep threads.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
292 - We support concurrent (in and out) stream processing.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
293 - The output is not a stream. It's a queue of read string (bytes, not unicode)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
294 chunks. The object behaves as an iterable. You can "for chunk in obj:" us.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
295 - We are non-blocking in more respects than communicate()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
296 (reading from subprocess out pauses when internal buffer is full, but
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
297 does not block the parent calling code. On the flip side, reading from
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
298 slow-yielding subprocess may block the iteration until data shows up. This
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
299 does not block the parallel inpipe reading occurring parallel thread.)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
300
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
301 The purpose of the object is to allow us to wrap subprocess interactions into
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
302 and interable that can be passed to a WSGI server as the application's return
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
303 value. Because of stream-processing-ability, WSGI does not have to read ALL
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
304 of the subprocess's output and buffer it, before handing it to WSGI server for
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
305 HTTP response. Instead, the class initializer reads just a bit of the stream
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
306 to figure out if error ocurred or likely to occur and if not, just hands the
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
307 further iteration over subprocess output to the server for completion of HTTP
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
308 response.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
309
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
310 The real or perceived subprocess error is trapped and raised as one of
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
311 EnvironmentError family of exceptions
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
312
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
313 Example usage:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
314 # try:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
315 # answer = SubprocessIOChunker(
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
316 # cmd,
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
317 # input,
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
318 # buffer_size = 65536,
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
319 # chunk_size = 4096
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
320 # )
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
321 # except (EnvironmentError) as e:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
322 # print str(e)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
323 # raise e
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
324 #
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
325 # return answer
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
326
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
327
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
328 """
4116
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3895
diff changeset
329
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
330 def __init__(self, cmd, inputstream=None, buffer_size=65536,
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
331 chunk_size=4096, starting_values=[], **kwargs):
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
332 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
333 Initializes SubprocessIOChunker
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
334
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
335 :param cmd: A Subprocess.Popen style "cmd". Can be string or array of strings
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
336 :param inputstream: (Default: None) A file-like, string, or file pointer.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
337 :param buffer_size: (Default: 65536) A size of total buffer per stream in bytes.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
338 :param chunk_size: (Default: 4096) A max size of a chunk. Actual chunk may be smaller.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
339 :param starting_values: (Default: []) An array of strings to put in front of output que.
3886
a1696507b3ad use consisten double quote docstring formatting
Marcin Kuzminski <marcin@python-works.com>
parents: 3830
diff changeset
340 """
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
341
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
342 if inputstream:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
343 input_streamer = StreamFeeder(inputstream)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
344 input_streamer.start()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
345 inputstream = input_streamer.output
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
346
3830
08d439bfbd8c fixed handling shell argument in subprocess calls, it always was hardcoded even when passed properly in arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 3797
diff changeset
347 _shell = kwargs.get('shell', True)
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
348 if isinstance(cmd, (list, tuple)):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
349 cmd = ' '.join(cmd)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
350
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
351 kwargs['shell'] = _shell
4116
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3895
diff changeset
352 _p = subprocess.Popen(cmd, bufsize=-1,
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3895
diff changeset
353 stdin=inputstream,
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3895
diff changeset
354 stdout=subprocess.PIPE,
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3895
diff changeset
355 stderr=subprocess.PIPE,
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3895
diff changeset
356 **kwargs)
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
357
4116
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3895
diff changeset
358 bg_out = BufferedGenerator(_p.stdout, buffer_size, chunk_size,
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3895
diff changeset
359 starting_values)
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
360 bg_err = BufferedGenerator(_p.stderr, 16000, 1, bottomless=True)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
361
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
362 while not bg_out.done_reading and not bg_out.reading_paused and not bg_err.length:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
363 # doing this until we reach either end of file, or end of buffer.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
364 bg_out.data_added_event.wait(1)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
365 bg_out.data_added_event.clear()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
366
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
367 # at this point it's still ambiguous if we are done reading or just full buffer.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
368 # Either way, if error (returned by ended process, or implied based on
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
369 # presence of stuff in stderr output) we error out.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
370 # Else, we are happy.
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
371 _returncode = _p.poll()
3889
b84c83b651de replace equality comparision to None
Marcin Kuzminski <marcin@python-works.com>
parents: 3886
diff changeset
372 if _returncode or (_returncode is None and bg_err.length):
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
373 try:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
374 _p.terminate()
4116
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3895
diff changeset
375 except Exception:
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
376 pass
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
377 bg_out.stop()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
378 bg_err.stop()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
379 err = '%s' % ''.join(bg_err)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
380 if err:
4116
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3895
diff changeset
381 raise EnvironmentError(
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3895
diff changeset
382 "Subprocess exited due to an error:\n" + err)
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3895
diff changeset
383 raise EnvironmentError(
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 3895
diff changeset
384 "Subprocess exited with non 0 ret code:%s" % _returncode)
3797
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
385
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
386 self.process = _p
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
387 self.output = bg_out
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
388 self.error = bg_err
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
389
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
390 def __iter__(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
391 return self
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
392
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
393 def next(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
394 if self.process.poll():
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
395 err = '%s' % ''.join(self.error)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
396 raise EnvironmentError("Subprocess exited due to an error:\n" + err)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
397 return self.output.next()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
398
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
399 def throw(self, type, value=None, traceback=None):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
400 if self.output.length or not self.output.done_reading:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
401 raise type(value)
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
402
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
403 def close(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
404 try:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
405 self.process.terminate()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
406 except:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
407 pass
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
408 try:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
409 self.output.close()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
410 except:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
411 pass
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
412 try:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
413 self.error.close()
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
414 except:
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
415 pass
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
416
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
417 def __del__(self):
d7488551578e synced vcs with upstream
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
418 self.close()