annotate scripts/logformat.py @ 8455:d727e81e0097 stable

vcs: fix cloning remote repository with HTTP authentication (Issue #379) Using a remote clone URI of http://user:pass@host/... triggered an exception: ... E File ".../kallithea/lib/utils.py", line 256, in is_valid_repo_uri E GitRepository._check_url(url) E File ".../kallithea/lib/vcs/backends/git/repository.py", line 183, in _check_url E passmgr.add_password(*authinfo) E File "/usr/lib/python3.7/urllib/request.py", line 848, in add_password E self.reduce_uri(u, default_port) for u in uri) E File "/usr/lib/python3.7/urllib/request.py", line 848, in <genexpr> E self.reduce_uri(u, default_port) for u in uri) E File "/usr/lib/python3.7/urllib/request.py", line 875, in reduce_uri E host, port = splitport(authority) E File "/usr/lib/python3.7/urllib/parse.py", line 1022, in splitport E match = _portprog.fullmatch(host) E TypeError: cannot use a string pattern on a bytes-like object The authinfo tuple is obtained via mercurial.util.url, which unfortunately returns a tuple of bytes whereas urllib expects strings. It seems that mercurial internally has some more hacking around urllib as urllibcompat.py, which we don't use. Therefore, transform the bytes into strings before passing authinfo to urllib. As the realm can be None, we need to check it specifically otherwise safe_str would return a string 'None'. A basic test that catches the mentioned problem is added, even though it does not actually test that cloning with auth info will actually work (it only tests that it fails cleanly if the URI is not reachable). Additionally, one use of 'test_uri' in hg/repository.py still needed to be transformed from bytes to string. For git this was already ok.
author Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
date Wed, 22 Jul 2020 21:55:57 +0200
parents 4b68fbe195b6
children 0a84ef075575
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
8173
aa6f17a53b49 py3: switch to use Python 3 interpreter, temporarily leaving many things very broken until they have been migrated/fixed in a reviewable way
Mads Kiilerich <mads@kiilerich.com>
parents: 7844
diff changeset
1 #!/usr/bin/env python3
5599
8bc8366a6874 cleanup: introduce scripts/logformat.py for cleanup of string formatting of logging statements
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
2
8bc8366a6874 cleanup: introduce scripts/logformat.py for cleanup of string formatting of logging statements
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
3 import re
8bc8366a6874 cleanup: introduce scripts/logformat.py for cleanup of string formatting of logging statements
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
4 import sys
8bc8366a6874 cleanup: introduce scripts/logformat.py for cleanup of string formatting of logging statements
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
5
7811
0a277465fddf scripts: initial run of import cleanup using isort
Mads Kiilerich <mads@kiilerich.com>
parents: 7666
diff changeset
6
5599
8bc8366a6874 cleanup: introduce scripts/logformat.py for cleanup of string formatting of logging statements
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
7 logre = r'''
8bc8366a6874 cleanup: introduce scripts/logformat.py for cleanup of string formatting of logging statements
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
8 (log\.(?:error|info|warning|debug)
8bc8366a6874 cleanup: introduce scripts/logformat.py for cleanup of string formatting of logging statements
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
9 [(][ \n]*
8bc8366a6874 cleanup: introduce scripts/logformat.py for cleanup of string formatting of logging statements
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
10 )
8bc8366a6874 cleanup: introduce scripts/logformat.py for cleanup of string formatting of logging statements
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
11 %s
8bc8366a6874 cleanup: introduce scripts/logformat.py for cleanup of string formatting of logging statements
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
12 (
8bc8366a6874 cleanup: introduce scripts/logformat.py for cleanup of string formatting of logging statements
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
13 [ \n]*[)]
8bc8366a6874 cleanup: introduce scripts/logformat.py for cleanup of string formatting of logging statements
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
14 )
8bc8366a6874 cleanup: introduce scripts/logformat.py for cleanup of string formatting of logging statements
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
15 '''
7666
4473f1094d3d scripts: clean up and run the old scripts/logformat.py script
Mads Kiilerich <mads@kiilerich.com>
parents: 6860
diff changeset
16
4473f1094d3d scripts: clean up and run the old scripts/logformat.py script
Mads Kiilerich <mads@kiilerich.com>
parents: 6860
diff changeset
17
5599
8bc8366a6874 cleanup: introduce scripts/logformat.py for cleanup of string formatting of logging statements
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
18 res = [
8bc8366a6874 cleanup: introduce scripts/logformat.py for cleanup of string formatting of logging statements
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
19 # handle % () - keeping spaces around the old %
7820
63b548dd5ef3 flake8: fix E227 missing whitespace around bitwise or shift operator
Mads Kiilerich <mads@kiilerich.com>
parents: 7811
diff changeset
20 (re.compile(logre % r'''("[^"]*"|'[^']*') ([\n ]*) % ([\n ]*) \( ( (?:[^()]|\n)* (?: \( (?:[^()]|\n)* \) (?:[^()]|\n)* )* ) \) ''', flags=re.MULTILINE | re.VERBOSE), r'\1\2,\3\4\5\6'),
5599
8bc8366a6874 cleanup: introduce scripts/logformat.py for cleanup of string formatting of logging statements
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
21 # handle % without () - keeping spaces around the old %
7820
63b548dd5ef3 flake8: fix E227 missing whitespace around bitwise or shift operator
Mads Kiilerich <mads@kiilerich.com>
parents: 7811
diff changeset
22 (re.compile(logre % r'''("[^"]*"|'[^']*') ([\n ]*) % ([\n ]*) ( (?:[^()]|\n)* (?: \( (?:[^()]|\n)* \) (?:[^()]|\n)* )* ) ''', flags=re.MULTILINE | re.VERBOSE), r'\1\2,\3\4\5\6'),
5599
8bc8366a6874 cleanup: introduce scripts/logformat.py for cleanup of string formatting of logging statements
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
23 # remove extra space if it is on next line
7820
63b548dd5ef3 flake8: fix E227 missing whitespace around bitwise or shift operator
Mads Kiilerich <mads@kiilerich.com>
parents: 7811
diff changeset
24 (re.compile(logre % r'''("[^"]*"|'[^']*') , (\n [ ]) ([ ][\n ]*) ( (?:[^()]|\n)* (?: \( (?:[^()]|\n)* \) (?:[^()]|\n)* )* ) ''', flags=re.MULTILINE | re.VERBOSE), r'\1\2,\3\4\5\6'),
5599
8bc8366a6874 cleanup: introduce scripts/logformat.py for cleanup of string formatting of logging statements
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
25 # remove extra space if it is on same line
7820
63b548dd5ef3 flake8: fix E227 missing whitespace around bitwise or shift operator
Mads Kiilerich <mads@kiilerich.com>
parents: 7811
diff changeset
26 (re.compile(logre % r'''("[^"]*"|'[^']*') , [ ]+ () ( [\n ]+) ( (?:[^()]|\n)* (?: \( (?:[^()]|\n)* \) (?:[^()]|\n)* )* ) ''', flags=re.MULTILINE | re.VERBOSE), r'\1\2,\3\4\5\6'),
5599
8bc8366a6874 cleanup: introduce scripts/logformat.py for cleanup of string formatting of logging statements
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
27 # remove trailing , and space
7820
63b548dd5ef3 flake8: fix E227 missing whitespace around bitwise or shift operator
Mads Kiilerich <mads@kiilerich.com>
parents: 7811
diff changeset
28 (re.compile(logre % r'''("[^"]*"|'[^']*') , () ( [\n ]*) ( (?:[^()]|\n)* (?: \( (?:[^()]|\n)* \) (?:[^()]|\n)* )* [^(), \n] ) [ ,]*''', flags=re.MULTILINE | re.VERBOSE), r'\1\2,\3\4\5\6'),
5599
8bc8366a6874 cleanup: introduce scripts/logformat.py for cleanup of string formatting of logging statements
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
29 ]
8bc8366a6874 cleanup: introduce scripts/logformat.py for cleanup of string formatting of logging statements
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
30
7666
4473f1094d3d scripts: clean up and run the old scripts/logformat.py script
Mads Kiilerich <mads@kiilerich.com>
parents: 6860
diff changeset
31
4473f1094d3d scripts: clean up and run the old scripts/logformat.py script
Mads Kiilerich <mads@kiilerich.com>
parents: 6860
diff changeset
32 def rewrite(f):
6860
665dfa112f2c py3: replace "file" with "open"
Lars Kruse <devel@sumpfralle.de>
parents: 5599
diff changeset
33 s = open(f).read()
5599
8bc8366a6874 cleanup: introduce scripts/logformat.py for cleanup of string formatting of logging statements
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
34 for r, t in res:
8bc8366a6874 cleanup: introduce scripts/logformat.py for cleanup of string formatting of logging statements
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
35 s = r.sub(t, s)
6860
665dfa112f2c py3: replace "file" with "open"
Lars Kruse <devel@sumpfralle.de>
parents: 5599
diff changeset
36 open(f, 'w').write(s)
7666
4473f1094d3d scripts: clean up and run the old scripts/logformat.py script
Mads Kiilerich <mads@kiilerich.com>
parents: 6860
diff changeset
37
4473f1094d3d scripts: clean up and run the old scripts/logformat.py script
Mads Kiilerich <mads@kiilerich.com>
parents: 6860
diff changeset
38
4473f1094d3d scripts: clean up and run the old scripts/logformat.py script
Mads Kiilerich <mads@kiilerich.com>
parents: 6860
diff changeset
39 if __name__ == '__main__':
4473f1094d3d scripts: clean up and run the old scripts/logformat.py script
Mads Kiilerich <mads@kiilerich.com>
parents: 6860
diff changeset
40 if len(sys.argv) < 2:
7844
a8e6bb9ee9ea future: use Python print function
Mads Kiilerich <mads@kiilerich.com>
parents: 7820
diff changeset
41 print('Cleanup of superfluous % formatting of log statements.')
a8e6bb9ee9ea future: use Python print function
Mads Kiilerich <mads@kiilerich.com>
parents: 7820
diff changeset
42 print('Usage:')
a8e6bb9ee9ea future: use Python print function
Mads Kiilerich <mads@kiilerich.com>
parents: 7820
diff changeset
43 print(''' hg revert `hg loc '*.py'|grep -v logformat.py` && scripts/logformat.py `hg loc '*.py'` && hg diff''')
7666
4473f1094d3d scripts: clean up and run the old scripts/logformat.py script
Mads Kiilerich <mads@kiilerich.com>
parents: 6860
diff changeset
44 raise SystemExit(1)
4473f1094d3d scripts: clean up and run the old scripts/logformat.py script
Mads Kiilerich <mads@kiilerich.com>
parents: 6860
diff changeset
45
4473f1094d3d scripts: clean up and run the old scripts/logformat.py script
Mads Kiilerich <mads@kiilerich.com>
parents: 6860
diff changeset
46 for f in sys.argv[1:]:
4473f1094d3d scripts: clean up and run the old scripts/logformat.py script
Mads Kiilerich <mads@kiilerich.com>
parents: 6860
diff changeset
47 rewrite(f)