Mercurial > kallithea
annotate scripts/docs-headings.py @ 8588:210e76d69b62
mysql: bump charset to to 'utf8mb4' to get full UTF-8 support
We used to use 'utf8', but in MySQL, this is just an alias for 'utf8mb3' which
isn't full unicode. 'utf8mb4' has less surprises (especially if used with the
'utf8mb4_unicode_ci' collation).
MySQL character sets for server, database, tables, and connection are set
independently. Until now, we have specified 'utf8' when creating tables to
overrule the database charset and in the default MySQL connection URL.
For existing databases, it is recommended to change encoding (and collation) by
altering the each of the tables inside it as described on
https://stackoverflow.com/questions/6115612/how-to-convert-an-entire-mysql-database-characterset-and-collation-to-utf-8 .
author | Mads Kiilerich <mads@kiilerich.com> |
---|---|
date | Mon, 15 Jun 2020 16:57:20 +0200 |
parents | 4b68fbe195b6 |
children | 0a84ef075575 |
rev | line source |
---|---|
8173
aa6f17a53b49
py3: switch to use Python 3 interpreter, temporarily leaving many things very broken until they have been migrated/fixed in a reviewable way
Mads Kiilerich <mads@kiilerich.com>
parents:
7844
diff
changeset
|
1 #!/usr/bin/env python3 |
5537
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
2 |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
3 """ |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
4 Consistent formatting of rst section titles |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
5 """ |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
6 |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
7 import re |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
8 import subprocess |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
9 |
7811
0a277465fddf
scripts: initial run of import cleanup using isort
Mads Kiilerich <mads@kiilerich.com>
parents:
7499
diff
changeset
|
10 |
5537
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
11 spaces = [ |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
12 (0, 1), # we assume this is a over-and-underlined header |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
13 (2, 1), |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
14 (1, 1), |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
15 (1, 0), |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
16 (1, 0), |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
17 ] |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
18 |
5575
ed2fb6e84a02
docs: use consistent style for section titles
Mads Kiilerich <madski@unity3d.com>
parents:
5537
diff
changeset
|
19 # http://sphinx-doc.org/rest.html : |
ed2fb6e84a02
docs: use consistent style for section titles
Mads Kiilerich <madski@unity3d.com>
parents:
5537
diff
changeset
|
20 # for the Python documentation, this convention is used which you may follow: |
ed2fb6e84a02
docs: use consistent style for section titles
Mads Kiilerich <madski@unity3d.com>
parents:
5537
diff
changeset
|
21 # # with overline, for parts |
ed2fb6e84a02
docs: use consistent style for section titles
Mads Kiilerich <madski@unity3d.com>
parents:
5537
diff
changeset
|
22 # * with overline, for chapters |
ed2fb6e84a02
docs: use consistent style for section titles
Mads Kiilerich <madski@unity3d.com>
parents:
5537
diff
changeset
|
23 # =, for sections |
ed2fb6e84a02
docs: use consistent style for section titles
Mads Kiilerich <madski@unity3d.com>
parents:
5537
diff
changeset
|
24 # -, for subsections |
ed2fb6e84a02
docs: use consistent style for section titles
Mads Kiilerich <madski@unity3d.com>
parents:
5537
diff
changeset
|
25 # ^, for subsubsections |
ed2fb6e84a02
docs: use consistent style for section titles
Mads Kiilerich <madski@unity3d.com>
parents:
5537
diff
changeset
|
26 # ", for paragraphs |
ed2fb6e84a02
docs: use consistent style for section titles
Mads Kiilerich <madski@unity3d.com>
parents:
5537
diff
changeset
|
27 pystyles = ['#', '*', '=', '-', '^', '"'] |
ed2fb6e84a02
docs: use consistent style for section titles
Mads Kiilerich <madski@unity3d.com>
parents:
5537
diff
changeset
|
28 |
5537
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
29 # match on a header line underlined with one of the valid characters |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
30 headermatch = re.compile(r'''\n*(.+)\n([][!"#$%&'()*+,./:;<=>?@\\^_`{|}~-])\2{2,}\n+''', flags=re.MULTILINE) |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
31 |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
32 |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
33 def main(): |
7499
a188803df37e
scripts: docs-headings: improve performance by grouping 'hg diff' invocations
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
6860
diff
changeset
|
34 filenames = subprocess.check_output(['hg', 'loc', 'set:**.rst+kallithea/i18n/how_to']).splitlines() |
a188803df37e
scripts: docs-headings: improve performance by grouping 'hg diff' invocations
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
6860
diff
changeset
|
35 for fn in filenames: |
8209
01aca0a4f876
py3: officially support Python 3
Mads Kiilerich <mads@kiilerich.com>
parents:
8173
diff
changeset
|
36 fn = fn.decode() |
7844
a8e6bb9ee9ea
future: use Python print function
Mads Kiilerich <mads@kiilerich.com>
parents:
7811
diff
changeset
|
37 print('processing %s' % fn) |
6860
665dfa112f2c
py3: replace "file" with "open"
Lars Kruse <devel@sumpfralle.de>
parents:
5575
diff
changeset
|
38 s = open(fn).read() |
5537
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
39 |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
40 # find levels and their styles |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
41 lastpos = 0 |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
42 styles = [] |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
43 for markup in headermatch.findall(s): |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
44 style = markup[1] |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
45 if style in styles: |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
46 stylepos = styles.index(style) |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
47 if stylepos > lastpos + 1: |
7844
a8e6bb9ee9ea
future: use Python print function
Mads Kiilerich <mads@kiilerich.com>
parents:
7811
diff
changeset
|
48 print('bad style %r with level %s - was at %s' % (style, stylepos, lastpos)) |
5537
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
49 else: |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
50 stylepos = len(styles) |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
51 if stylepos > lastpos + 1: |
7844
a8e6bb9ee9ea
future: use Python print function
Mads Kiilerich <mads@kiilerich.com>
parents:
7811
diff
changeset
|
52 print('bad new style %r - expected %r' % (style, styles[lastpos + 1])) |
5537
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
53 else: |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
54 styles.append(style) |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
55 lastpos = stylepos |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
56 |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
57 # remove superfluous spacing (may however be restored by header spacing) |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
58 s = re.sub(r'''(\n\n)\n*''', r'\1', s, flags=re.MULTILINE) |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
59 |
5575
ed2fb6e84a02
docs: use consistent style for section titles
Mads Kiilerich <madski@unity3d.com>
parents:
5537
diff
changeset
|
60 if styles: |
ed2fb6e84a02
docs: use consistent style for section titles
Mads Kiilerich <madski@unity3d.com>
parents:
5537
diff
changeset
|
61 newstyles = pystyles[pystyles.index(styles[0]):] |
ed2fb6e84a02
docs: use consistent style for section titles
Mads Kiilerich <madski@unity3d.com>
parents:
5537
diff
changeset
|
62 |
ed2fb6e84a02
docs: use consistent style for section titles
Mads Kiilerich <madski@unity3d.com>
parents:
5537
diff
changeset
|
63 def subf(m): |
ed2fb6e84a02
docs: use consistent style for section titles
Mads Kiilerich <madski@unity3d.com>
parents:
5537
diff
changeset
|
64 title, style = m.groups() |
ed2fb6e84a02
docs: use consistent style for section titles
Mads Kiilerich <madski@unity3d.com>
parents:
5537
diff
changeset
|
65 level = styles.index(style) |
ed2fb6e84a02
docs: use consistent style for section titles
Mads Kiilerich <madski@unity3d.com>
parents:
5537
diff
changeset
|
66 before, after = spaces[level] |
ed2fb6e84a02
docs: use consistent style for section titles
Mads Kiilerich <madski@unity3d.com>
parents:
5537
diff
changeset
|
67 newstyle = newstyles[level] |
ed2fb6e84a02
docs: use consistent style for section titles
Mads Kiilerich <madski@unity3d.com>
parents:
5537
diff
changeset
|
68 return '\n' * (before + 1) + title + '\n' + newstyle * len(title) + '\n' * (after + 1) |
ed2fb6e84a02
docs: use consistent style for section titles
Mads Kiilerich <madski@unity3d.com>
parents:
5537
diff
changeset
|
69 s = headermatch.sub(subf, s) |
5537
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
70 |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
71 # remove superfluous spacing when headers are adjacent |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
72 s = re.sub(r'''(\n.+\n([][!"#$%&'()*+,./:;<=>?@\\^_`{|}~-])\2{2,}\n\n\n)\n*''', r'\1', s, flags=re.MULTILINE) |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
73 # fix trailing space and spacing before link sections |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
74 s = s.strip() + '\n' |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
75 s = re.sub(r'''\n+((?:\.\. _[^\n]*\n)+)$''', r'\n\n\n\1', s) |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
76 |
6860
665dfa112f2c
py3: replace "file" with "open"
Lars Kruse <devel@sumpfralle.de>
parents:
5575
diff
changeset
|
77 open(fn, 'w').write(s) |
7499
a188803df37e
scripts: docs-headings: improve performance by grouping 'hg diff' invocations
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
6860
diff
changeset
|
78 |
7844
a8e6bb9ee9ea
future: use Python print function
Mads Kiilerich <mads@kiilerich.com>
parents:
7811
diff
changeset
|
79 print(subprocess.check_output(['hg', 'diff'] + filenames)) |
5537
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
80 |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
81 if __name__ == '__main__': |
f38b50f8a6a6
scripts: introduce scripts/docs-headings.py for reformatting rst section titles in docs
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
82 main() |