annotate scripts/generate-ini.py @ 8455:d727e81e0097 stable

vcs: fix cloning remote repository with HTTP authentication (Issue #379) Using a remote clone URI of http://user:pass@host/... triggered an exception: ... E File ".../kallithea/lib/utils.py", line 256, in is_valid_repo_uri E GitRepository._check_url(url) E File ".../kallithea/lib/vcs/backends/git/repository.py", line 183, in _check_url E passmgr.add_password(*authinfo) E File "/usr/lib/python3.7/urllib/request.py", line 848, in add_password E self.reduce_uri(u, default_port) for u in uri) E File "/usr/lib/python3.7/urllib/request.py", line 848, in <genexpr> E self.reduce_uri(u, default_port) for u in uri) E File "/usr/lib/python3.7/urllib/request.py", line 875, in reduce_uri E host, port = splitport(authority) E File "/usr/lib/python3.7/urllib/parse.py", line 1022, in splitport E match = _portprog.fullmatch(host) E TypeError: cannot use a string pattern on a bytes-like object The authinfo tuple is obtained via mercurial.util.url, which unfortunately returns a tuple of bytes whereas urllib expects strings. It seems that mercurial internally has some more hacking around urllib as urllibcompat.py, which we don't use. Therefore, transform the bytes into strings before passing authinfo to urllib. As the realm can be None, we need to check it specifically otherwise safe_str would return a string 'None'. A basic test that catches the mentioned problem is added, even though it does not actually test that cloning with auth info will actually work (it only tests that it fails cleanly if the URI is not reachable). Additionally, one use of 'test_uri' in hg/repository.py still needed to be transformed from bytes to string. For git this was already ok.
author Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
date Wed, 22 Jul 2020 21:55:57 +0200
parents 6eb1f66ac23f
children 495dea7c2a13
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
8173
aa6f17a53b49 py3: switch to use Python 3 interpreter, temporarily leaving many things very broken until they have been migrated/fixed in a reviewable way
Mads Kiilerich <mads@kiilerich.com>
parents: 7844
diff changeset
1 #!/usr/bin/env python3
5536
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
2 """
7390
451b3f9d814e docs: update i18n doc after TG migration changed lang to i18n.lang and test.ini is generated
Mads Kiilerich <mads@kiilerich.com>
parents: 7260
diff changeset
3 Based on kallithea/lib/paster_commands/template.ini.mako, generate development.ini
5536
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
4 """
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
5
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
6 import re
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
7
6884
e3cce237d77c ini: extract .ini handling from scripts/generate-ini.py to kallithea/lib/inifile.py
Mads Kiilerich <mads@kiilerich.com>
parents: 6883
diff changeset
8 from kallithea.lib import inifile
e3cce237d77c ini: extract .ini handling from scripts/generate-ini.py to kallithea/lib/inifile.py
Mads Kiilerich <mads@kiilerich.com>
parents: 6883
diff changeset
9
7811
0a277465fddf scripts: initial run of import cleanup using isort
Mads Kiilerich <mads@kiilerich.com>
parents: 7768
diff changeset
10
5536
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
11 # files to be generated from the mako template
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
12 ini_files = [
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
13 ('development.ini',
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
14 {
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
15 '[server:main]': {
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
16 'host': '0.0.0.0',
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
17 },
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
18 '[app:main]': {
6573
fc6b1b0e1096 ini: set 'debug' only in one place
Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
parents: 6555
diff changeset
19 'debug': 'true',
5536
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
20 'app_instance_uuid': 'development-not-secret',
7738
bbf7be28a11e config: rename .ini settings to avoid beaker-session deprecation warnings
Mads Kiilerich <mads@kiilerich.com>
parents: 7390
diff changeset
21 'session.secret': 'development-not-secret',
5536
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
22 },
7768
609d52bbf917 ini: create separate log handlers for color and null, and add comments hinting how they can be used
Mads Kiilerich <mads@kiilerich.com>
parents: 7738
diff changeset
23 '[logger_root]': {
609d52bbf917 ini: create separate log handlers for color and null, and add comments hinting how they can be used
Mads Kiilerich <mads@kiilerich.com>
parents: 7738
diff changeset
24 'handlers': 'console_color',
5536
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
25 },
7193
150173a027ee ini: set default log level to WARN
domruf <dominikruf@gmail.com>
parents: 6895
diff changeset
26 '[logger_routes]': {
150173a027ee ini: set default log level to WARN
domruf <dominikruf@gmail.com>
parents: 6895
diff changeset
27 'level': 'DEBUG',
150173a027ee ini: set default log level to WARN
domruf <dominikruf@gmail.com>
parents: 6895
diff changeset
28 },
150173a027ee ini: set default log level to WARN
domruf <dominikruf@gmail.com>
parents: 6895
diff changeset
29 '[logger_beaker]': {
150173a027ee ini: set default log level to WARN
domruf <dominikruf@gmail.com>
parents: 6895
diff changeset
30 'level': 'DEBUG',
150173a027ee ini: set default log level to WARN
domruf <dominikruf@gmail.com>
parents: 6895
diff changeset
31 },
150173a027ee ini: set default log level to WARN
domruf <dominikruf@gmail.com>
parents: 6895
diff changeset
32 '[logger_templates]': {
150173a027ee ini: set default log level to WARN
domruf <dominikruf@gmail.com>
parents: 6895
diff changeset
33 'level': 'INFO',
150173a027ee ini: set default log level to WARN
domruf <dominikruf@gmail.com>
parents: 6895
diff changeset
34 },
150173a027ee ini: set default log level to WARN
domruf <dominikruf@gmail.com>
parents: 6895
diff changeset
35 '[logger_kallithea]': {
150173a027ee ini: set default log level to WARN
domruf <dominikruf@gmail.com>
parents: 6895
diff changeset
36 'level': 'DEBUG',
150173a027ee ini: set default log level to WARN
domruf <dominikruf@gmail.com>
parents: 6895
diff changeset
37 },
150173a027ee ini: set default log level to WARN
domruf <dominikruf@gmail.com>
parents: 6895
diff changeset
38 '[logger_tg]': {
150173a027ee ini: set default log level to WARN
domruf <dominikruf@gmail.com>
parents: 6895
diff changeset
39 'level': 'DEBUG',
150173a027ee ini: set default log level to WARN
domruf <dominikruf@gmail.com>
parents: 6895
diff changeset
40 },
150173a027ee ini: set default log level to WARN
domruf <dominikruf@gmail.com>
parents: 6895
diff changeset
41 '[logger_gearbox]': {
150173a027ee ini: set default log level to WARN
domruf <dominikruf@gmail.com>
parents: 6895
diff changeset
42 'level': 'DEBUG',
150173a027ee ini: set default log level to WARN
domruf <dominikruf@gmail.com>
parents: 6895
diff changeset
43 },
150173a027ee ini: set default log level to WARN
domruf <dominikruf@gmail.com>
parents: 6895
diff changeset
44 '[logger_whoosh_indexer]': {
150173a027ee ini: set default log level to WARN
domruf <dominikruf@gmail.com>
parents: 6895
diff changeset
45 'level': 'DEBUG',
150173a027ee ini: set default log level to WARN
domruf <dominikruf@gmail.com>
parents: 6895
diff changeset
46 },
5536
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
47 },
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
48 ),
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
49 ]
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
50
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
51
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
52 def main():
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
53 # make sure all mako lines starting with '#' (the '##' comments) are marked up as <text>
6894
94f6b23e52d0 ini: move high level functionality and defaults to inifiles library
Mads Kiilerich <mads@kiilerich.com>
parents: 6893
diff changeset
54 makofile = inifile.template_file
7844
a8e6bb9ee9ea future: use Python print function
Mads Kiilerich <mads@kiilerich.com>
parents: 7811
diff changeset
55 print('reading:', makofile)
6860
665dfa112f2c py3: replace "file" with "open"
Lars Kruse <devel@sumpfralle.de>
parents: 6857
diff changeset
56 mako_org = open(makofile).read()
5536
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
57 mako_no_text_markup = re.sub(r'</?%text>', '', mako_org)
8428
6eb1f66ac23f ini: tweak mako comment markup - prepare for variable expansion in ## lines by only "quoting" ##
Mads Kiilerich <mads@kiilerich.com>
parents: 8354
diff changeset
58 mako_marked_up = re.sub(r'\n##(.*)', r'\n<%text>##</%text>\1', mako_no_text_markup, flags=re.MULTILINE)
5536
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
59 if mako_marked_up != mako_org:
7844
a8e6bb9ee9ea future: use Python print function
Mads Kiilerich <mads@kiilerich.com>
parents: 7811
diff changeset
60 print('writing:', makofile)
6860
665dfa112f2c py3: replace "file" with "open"
Lars Kruse <devel@sumpfralle.de>
parents: 6857
diff changeset
61 open(makofile, 'w').write(mako_marked_up)
5536
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
62
8354
ef9fd1434270 ini: be more consistent about how # is used in the config template
Mads Kiilerich <mads@kiilerich.com>
parents: 8225
diff changeset
63 lines = re.findall(r'\n(# [^ ].*)', mako_marked_up)
ef9fd1434270 ini: be more consistent about how # is used in the config template
Mads Kiilerich <mads@kiilerich.com>
parents: 8225
diff changeset
64 if lines:
ef9fd1434270 ini: be more consistent about how # is used in the config template
Mads Kiilerich <mads@kiilerich.com>
parents: 8225
diff changeset
65 print('ERROR: the template .ini file convention is to use "## Foo Bar" for text comments and "#foo = bar" for disabled settings')
ef9fd1434270 ini: be more consistent about how # is used in the config template
Mads Kiilerich <mads@kiilerich.com>
parents: 8225
diff changeset
66 for line in lines:
ef9fd1434270 ini: be more consistent about how # is used in the config template
Mads Kiilerich <mads@kiilerich.com>
parents: 8225
diff changeset
67 print(line)
ef9fd1434270 ini: be more consistent about how # is used in the config template
Mads Kiilerich <mads@kiilerich.com>
parents: 8225
diff changeset
68 raise SystemExit(1)
ef9fd1434270 ini: be more consistent about how # is used in the config template
Mads Kiilerich <mads@kiilerich.com>
parents: 8225
diff changeset
69
5536
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
70 # create ini files
6893
d06039dc4ca2 ini: drop insertion of header comments in generated ini files
Mads Kiilerich <mads@kiilerich.com>
parents: 6891
diff changeset
71 for fn, settings in ini_files:
7844
a8e6bb9ee9ea future: use Python print function
Mads Kiilerich <mads@kiilerich.com>
parents: 7811
diff changeset
72 print('updating:', fn)
6894
94f6b23e52d0 ini: move high level functionality and defaults to inifiles library
Mads Kiilerich <mads@kiilerich.com>
parents: 6893
diff changeset
73 inifile.create(fn, None, settings)
94f6b23e52d0 ini: move high level functionality and defaults to inifiles library
Mads Kiilerich <mads@kiilerich.com>
parents: 6893
diff changeset
74
5536
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
75
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
76 if __name__ == '__main__':
06d5c043e989 ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
77 main()