Mercurial > kallithea

annotate scripts/i18n_utils.py @ 8365:d5e77cc4a4f2 i18n

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Merge divergent i18n changes that were caused by broken weblate repo access
author Mads Kiilerich <mads@kiilerich.com>
date Mon, 04 May 2020 18:59:36 +0200
parents 19735bc60455
children 6bde1c0a04d4
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
8313
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
1 # This program is free software: you can redistribute it and/or modify
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
2 # it under the terms of the GNU General Public License as published by
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
3 # the Free Software Foundation, either version 3 of the License, or
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
4 # (at your option) any later version.
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
5 #
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
6 # This program is distributed in the hope that it will be useful,
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
7 # but WITHOUT ANY WARRANTY; without even the implied warranty of
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
8 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
9 # GNU General Public License for more details.
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
10 #
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
11 # You should have received a copy of the GNU General Public License
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
12 # along with this program. If not, see <http://www.gnu.org/licenses/>.
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
13
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
14 from __future__ import print_function
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
15
8314
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
16 import os
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
17 import re
8315
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
18 import shutil
8313
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
19 import subprocess
8315
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
20 import tempfile
8313
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
21
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
22
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
23 do_debug = False # set from scripts/i18n --debug
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
24
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
25 def debug(*args, **kwargs):
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
26 if do_debug:
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
27 print(*args, **kwargs)
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
28
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
29 def runcmd(cmd, *args, **kwargs):
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
30 debug('... Executing command: %s' % ' '.join(cmd))
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
31 subprocess.check_call(cmd, *args, **kwargs)
8314
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
32
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
33 header_comment_strip_re = re.compile(r'''
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
34 ^
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
35 [#][ ]Translations[ ]template[ ]for[ ]Kallithea[.] \n
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
36 |
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
37 ^
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
38 [#][ ]FIRST[ ]AUTHOR[ ]<EMAIL@ADDRESS>,[ ]\d+[.] \n
8333
e505395c3a03 i18n: better stripping of header comment for new translations
Mads Kiilerich <mads@kiilerich.com>
parents: 8318
diff changeset
39 (?:[#] \n)?
e505395c3a03 i18n: better stripping of header comment for new translations
Mads Kiilerich <mads@kiilerich.com>
parents: 8318
diff changeset
40 |
e505395c3a03 i18n: better stripping of header comment for new translations
Mads Kiilerich <mads@kiilerich.com>
parents: 8318
diff changeset
41 ^
e505395c3a03 i18n: better stripping of header comment for new translations
Mads Kiilerich <mads@kiilerich.com>
parents: 8318
diff changeset
42 (?:[#] \n)?
8314
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
43 [#],[ ]fuzzy \n
8334
19735bc60455 i18n: also strip '# #, fuzzy' from header comment - it might appear when verifying branches are in sync
Mads Kiilerich <mads@kiilerich.com>
parents: 8333
diff changeset
44 |
19735bc60455 i18n: also strip '# #, fuzzy' from header comment - it might appear when verifying branches are in sync
Mads Kiilerich <mads@kiilerich.com>
parents: 8333
diff changeset
45 ^
19735bc60455 i18n: also strip '# #, fuzzy' from header comment - it might appear when verifying branches are in sync
Mads Kiilerich <mads@kiilerich.com>
parents: 8333
diff changeset
46 [#][ ][#],[ ]fuzzy \n
8314
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
47 ''',
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
48 re.MULTILINE|re.VERBOSE)
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
49
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
50 header_normalize_re = re.compile(r'''
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
51 ^ "
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
52 (POT-Creation-Date|PO-Revision-Date|Last-Translator|Language-Team|X-Generator|Generated-By|Project-Id-Version):
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
53 [ ][^\\]*\\n
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
54 " \n
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
55 ''',
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
56 re.MULTILINE|re.IGNORECASE|re.VERBOSE)
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
57
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
58 def _normalize_po(raw_content):
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
59 r"""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
60 >>> print(_normalize_po(r'''
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
61 ... # header comment
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
62 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
63 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
64 ... # comment before header
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
65 ... msgid ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
66 ... msgstr "yada"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
67 ... "POT-Creation-Date: 2019-05-04 21:13+0200\n"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
68 ... "MIME-Version: "
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
69 ... "1.0\n"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
70 ... "Last-Translator: Jabba"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
71 ... "the Hutt\n"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
72 ... "X-Generator: Weblate 1.2.3\n"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
73 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
74 ... # comment, but not in header
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
75 ... msgid "None"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
76 ... msgstr "Ingen"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
77 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
78 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
79 ... line 2
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
80 ... # third comment
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
81 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
82 ... msgid "Special"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
83 ... msgstr ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
84 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
85 ... msgid "Specialist"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
86 ... # odd comment
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
87 ... msgstr ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
88 ... "Expert"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
89 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
90 ... # crazy fuzzy auto translation by msgmerge, using foo for bar
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
91 ... #, fuzzy
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
92 ... #| msgid "some foo string"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
93 ... msgid "some bar string."
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
94 ... msgstr "translation of foo string"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
95 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
96 ... msgid "%d minute"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
97 ... msgid_plural "%d minutes"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
98 ... msgstr[0] "minut"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
99 ... msgstr[1] "minutter"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
100 ... msgstr[2] ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
101 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
102 ... msgid "%d year"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
103 ... msgid_plural "%d years"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
104 ... msgstr[0] ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
105 ... msgstr[1] ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
106 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
107 ... # last comment
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
108 ... ''') + '^^^')
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
109 # header comment
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
110 <BLANKLINE>
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
111 <BLANKLINE>
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
112 # comment before header
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
113 <BLANKLINE>
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
114 msgid ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
115 msgstr "yada"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
116 "MIME-Version: "
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
117 "1.0\n"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
118 <BLANKLINE>
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
119 msgid "None"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
120 msgstr "Ingen"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
121 <BLANKLINE>
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
122 line 2
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
123 <BLANKLINE>
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
124 msgid "Specialist"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
125 msgstr ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
126 "Expert"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
127 <BLANKLINE>
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
128 msgid "%d minute"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
129 msgid_plural "%d minutes"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
130 msgstr[0] "minut"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
131 msgstr[1] "minutter"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
132 msgstr[2] ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
133 ^^^
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
134 """
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
135 header_start = raw_content.find('\nmsgid ""\n') + 1
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
136 header_end = raw_content.find('\n\n', header_start) + 1 or len(raw_content)
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
137 chunks = [
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
138 header_comment_strip_re.sub('', raw_content[0:header_start])
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
139 .strip(),
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
140 '',
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
141 header_normalize_re.sub('', raw_content[header_start:header_end])
8318
30e137b4ff18 scripts/i18n: also normalize casing of UTF-8 in Content-Type
Mads Kiilerich <mads@kiilerich.com>
parents: 8317
diff changeset
142 .replace(
30e137b4ff18 scripts/i18n: also normalize casing of UTF-8 in Content-Type
Mads Kiilerich <mads@kiilerich.com>
parents: 8317
diff changeset
143 r'"Content-Type: text/plain; charset=utf-8\n"',
30e137b4ff18 scripts/i18n: also normalize casing of UTF-8 in Content-Type
Mads Kiilerich <mads@kiilerich.com>
parents: 8317
diff changeset
144 r'"Content-Type: text/plain; charset=UTF-8\n"') # maintain msgmerge casing
8314
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
145 .strip(),
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
146 ''] # preserve normalized header
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
147 # all chunks are separated by empty line
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
148 for raw_chunk in raw_content[header_end:].split('\n\n'):
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
149 if '\n#, fuzzy' in raw_chunk: # might be like "#, fuzzy, python-format"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
150 continue # drop crazy auto translation that is worse than useless
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
151 # strip all comment lines from chunk
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
152 chunk_lines = [
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
153 line
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
154 for line in raw_chunk.splitlines()
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
155 if line
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
156 and not line.startswith('#')
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
157 ]
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
158 if not chunk_lines:
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
159 continue
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
160 # check lines starting from first msgstr, skip chunk if no translation lines
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
161 msgstr_i = [i for i, line in enumerate(chunk_lines) if line.startswith('msgstr')]
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
162 if (
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
163 chunk_lines[0].startswith('msgid') and
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
164 msgstr_i and
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
165 all(line.endswith(' ""') for line in chunk_lines[msgstr_i[0]:])
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
166 ): # skip translation chunks that doesn't have any actual translations
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
167 continue
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
168 chunks.append('\n'.join(chunk_lines) + '\n')
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
169 return '\n'.join(chunks)
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
170
8317
5553ecc962e0 scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents: 8315
diff changeset
171 def _normalize_po_file(po_file, merge_pot_file=None, strip=False):
5553ecc962e0 scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents: 8315
diff changeset
172 if merge_pot_file:
5553ecc962e0 scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents: 8315
diff changeset
173 runcmd(['msgmerge', '--width=76', '--backup=none', '--previous',
5553ecc962e0 scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents: 8315
diff changeset
174 '--update', po_file, '-q', merge_pot_file])
8314
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
175 if strip:
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
176 po_tmp = po_file + '.tmp'
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
177 with open(po_file, 'r') as src, open(po_tmp, 'w') as dest:
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
178 raw_content = src.read()
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
179 normalized_content = _normalize_po(raw_content)
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
180 dest.write(normalized_content)
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
181 os.rename(po_tmp, po_file)
8315
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
182
8317
5553ecc962e0 scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents: 8315
diff changeset
183 def _normalized_diff(file1, file2, merge_pot_file=None, strip=False):
8315
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
184 # Create temporary copies of both files
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
185 temp1 = tempfile.NamedTemporaryFile(prefix=os.path.basename(file1))
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
186 temp2 = tempfile.NamedTemporaryFile(prefix=os.path.basename(file2))
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
187 debug('normalized_diff: %s -> %s / %s -> %s' % (file1, temp1.name, file2, temp2.name))
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
188 shutil.copyfile(file1, temp1.name)
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
189 shutil.copyfile(file2, temp2.name)
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
190 # Normalize them in place
8317
5553ecc962e0 scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents: 8315
diff changeset
191 _normalize_po_file(temp1.name, merge_pot_file=merge_pot_file, strip=strip)
5553ecc962e0 scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents: 8315
diff changeset
192 _normalize_po_file(temp2.name, merge_pot_file=merge_pot_file, strip=strip)
8315
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
193 # Now compare
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
194 try:
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
195 runcmd(['diff', '-u', temp1.name, temp2.name])
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
196 except subprocess.CalledProcessError as e:
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
197 return e.returncode