Mercurial > kallithea
annotate scripts/i18n_utils.py @ 8751:ad239692ea95
mail: fix duplicate "From" headers
Problem introduced in 9a0c41175e66: When iterating the headers dict and setting
"msg[key] = value", it wasn't replacing the header but performing add_header so
we sometimes ended up with two From headers.
It is also a general problem that while the headers dict only can contain each
key once, it can contain entries that only differ in casing and thus will fold
to the same message header, making it possible to end up adding duplicate
headers.
"msg.replace_header(key, value)" is not a simple solution to the problem: it
will raise KeyError if no such previous key exists.
Now, make the problem more clear by explicitly using add_header.
Avoid the duplication problem by deleting the key (no matter which casing)
before invoking add_header. Delete promises that "No exception is raised if the
named field isn’t present in the headers".
author | Mads Kiilerich <mads@kiilerich.com> |
---|---|
date | Wed, 04 Nov 2020 00:35:21 +0100 |
parents | 6bde1c0a04d4 |
children | 0c9c91ac3873 |
rev | line source |
---|---|
8313
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
1 # This program is free software: you can redistribute it and/or modify |
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
2 # it under the terms of the GNU General Public License as published by |
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
3 # the Free Software Foundation, either version 3 of the License, or |
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
4 # (at your option) any later version. |
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
5 # |
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
6 # This program is distributed in the hope that it will be useful, |
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
7 # but WITHOUT ANY WARRANTY; without even the implied warranty of |
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
8 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
9 # GNU General Public License for more details. |
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
10 # |
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
11 # You should have received a copy of the GNU General Public License |
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
12 # along with this program. If not, see <http://www.gnu.org/licenses/>. |
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
13 |
8314
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
14 import os |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
15 import re |
8315
93dabafa567e
scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8314
diff
changeset
|
16 import shutil |
8313
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
17 import subprocess |
8315
93dabafa567e
scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8314
diff
changeset
|
18 import tempfile |
8313
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
19 |
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
20 |
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
21 do_debug = False # set from scripts/i18n --debug |
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
22 |
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
23 def debug(*args, **kwargs): |
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
24 if do_debug: |
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
25 print(*args, **kwargs) |
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
26 |
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
27 def runcmd(cmd, *args, **kwargs): |
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
28 debug('... Executing command: %s' % ' '.join(cmd)) |
4bc712f1ec93
scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff
changeset
|
29 subprocess.check_call(cmd, *args, **kwargs) |
8314
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
30 |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
31 header_comment_strip_re = re.compile(r''' |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
32 ^ |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
33 [#][ ]Translations[ ]template[ ]for[ ]Kallithea[.] \n |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
34 | |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
35 ^ |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
36 [#][ ]FIRST[ ]AUTHOR[ ]<EMAIL@ADDRESS>,[ ]\d+[.] \n |
8333
e505395c3a03
i18n: better stripping of header comment for new translations
Mads Kiilerich <mads@kiilerich.com>
parents:
8318
diff
changeset
|
37 (?:[#] \n)? |
e505395c3a03
i18n: better stripping of header comment for new translations
Mads Kiilerich <mads@kiilerich.com>
parents:
8318
diff
changeset
|
38 | |
e505395c3a03
i18n: better stripping of header comment for new translations
Mads Kiilerich <mads@kiilerich.com>
parents:
8318
diff
changeset
|
39 ^ |
e505395c3a03
i18n: better stripping of header comment for new translations
Mads Kiilerich <mads@kiilerich.com>
parents:
8318
diff
changeset
|
40 (?:[#] \n)? |
8314
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
41 [#],[ ]fuzzy \n |
8334
19735bc60455
i18n: also strip '# #, fuzzy' from header comment - it might appear when verifying branches are in sync
Mads Kiilerich <mads@kiilerich.com>
parents:
8333
diff
changeset
|
42 | |
19735bc60455
i18n: also strip '# #, fuzzy' from header comment - it might appear when verifying branches are in sync
Mads Kiilerich <mads@kiilerich.com>
parents:
8333
diff
changeset
|
43 ^ |
19735bc60455
i18n: also strip '# #, fuzzy' from header comment - it might appear when verifying branches are in sync
Mads Kiilerich <mads@kiilerich.com>
parents:
8333
diff
changeset
|
44 [#][ ][#],[ ]fuzzy \n |
8314
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
45 ''', |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
46 re.MULTILINE|re.VERBOSE) |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
47 |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
48 header_normalize_re = re.compile(r''' |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
49 ^ " |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
50 (POT-Creation-Date|PO-Revision-Date|Last-Translator|Language-Team|X-Generator|Generated-By|Project-Id-Version): |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
51 [ ][^\\]*\\n |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
52 " \n |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
53 ''', |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
54 re.MULTILINE|re.IGNORECASE|re.VERBOSE) |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
55 |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
56 def _normalize_po(raw_content): |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
57 r""" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
58 >>> print(_normalize_po(r''' |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
59 ... # header comment |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
60 ... |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
61 ... |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
62 ... # comment before header |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
63 ... msgid "" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
64 ... msgstr "yada" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
65 ... "POT-Creation-Date: 2019-05-04 21:13+0200\n" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
66 ... "MIME-Version: " |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
67 ... "1.0\n" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
68 ... "Last-Translator: Jabba" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
69 ... "the Hutt\n" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
70 ... "X-Generator: Weblate 1.2.3\n" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
71 ... |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
72 ... # comment, but not in header |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
73 ... msgid "None" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
74 ... msgstr "Ingen" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
75 ... |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
76 ... |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
77 ... line 2 |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
78 ... # third comment |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
79 ... |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
80 ... msgid "Special" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
81 ... msgstr "" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
82 ... |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
83 ... msgid "Specialist" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
84 ... # odd comment |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
85 ... msgstr "" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
86 ... "Expert" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
87 ... |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
88 ... # crazy fuzzy auto translation by msgmerge, using foo for bar |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
89 ... #, fuzzy |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
90 ... #| msgid "some foo string" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
91 ... msgid "some bar string." |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
92 ... msgstr "translation of foo string" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
93 ... |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
94 ... msgid "%d minute" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
95 ... msgid_plural "%d minutes" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
96 ... msgstr[0] "minut" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
97 ... msgstr[1] "minutter" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
98 ... msgstr[2] "" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
99 ... |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
100 ... msgid "%d year" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
101 ... msgid_plural "%d years" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
102 ... msgstr[0] "" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
103 ... msgstr[1] "" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
104 ... |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
105 ... # last comment |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
106 ... ''') + '^^^') |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
107 # header comment |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
108 <BLANKLINE> |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
109 <BLANKLINE> |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
110 # comment before header |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
111 <BLANKLINE> |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
112 msgid "" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
113 msgstr "yada" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
114 "MIME-Version: " |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
115 "1.0\n" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
116 <BLANKLINE> |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
117 msgid "None" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
118 msgstr "Ingen" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
119 <BLANKLINE> |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
120 line 2 |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
121 <BLANKLINE> |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
122 msgid "Specialist" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
123 msgstr "" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
124 "Expert" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
125 <BLANKLINE> |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
126 msgid "%d minute" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
127 msgid_plural "%d minutes" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
128 msgstr[0] "minut" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
129 msgstr[1] "minutter" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
130 msgstr[2] "" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
131 ^^^ |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
132 """ |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
133 header_start = raw_content.find('\nmsgid ""\n') + 1 |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
134 header_end = raw_content.find('\n\n', header_start) + 1 or len(raw_content) |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
135 chunks = [ |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
136 header_comment_strip_re.sub('', raw_content[0:header_start]) |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
137 .strip(), |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
138 '', |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
139 header_normalize_re.sub('', raw_content[header_start:header_end]) |
8318
30e137b4ff18
scripts/i18n: also normalize casing of UTF-8 in Content-Type
Mads Kiilerich <mads@kiilerich.com>
parents:
8317
diff
changeset
|
140 .replace( |
30e137b4ff18
scripts/i18n: also normalize casing of UTF-8 in Content-Type
Mads Kiilerich <mads@kiilerich.com>
parents:
8317
diff
changeset
|
141 r'"Content-Type: text/plain; charset=utf-8\n"', |
30e137b4ff18
scripts/i18n: also normalize casing of UTF-8 in Content-Type
Mads Kiilerich <mads@kiilerich.com>
parents:
8317
diff
changeset
|
142 r'"Content-Type: text/plain; charset=UTF-8\n"') # maintain msgmerge casing |
8314
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
143 .strip(), |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
144 ''] # preserve normalized header |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
145 # all chunks are separated by empty line |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
146 for raw_chunk in raw_content[header_end:].split('\n\n'): |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
147 if '\n#, fuzzy' in raw_chunk: # might be like "#, fuzzy, python-format" |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
148 continue # drop crazy auto translation that is worse than useless |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
149 # strip all comment lines from chunk |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
150 chunk_lines = [ |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
151 line |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
152 for line in raw_chunk.splitlines() |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
153 if line |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
154 and not line.startswith('#') |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
155 ] |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
156 if not chunk_lines: |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
157 continue |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
158 # check lines starting from first msgstr, skip chunk if no translation lines |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
159 msgstr_i = [i for i, line in enumerate(chunk_lines) if line.startswith('msgstr')] |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
160 if ( |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
161 chunk_lines[0].startswith('msgid') and |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
162 msgstr_i and |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
163 all(line.endswith(' ""') for line in chunk_lines[msgstr_i[0]:]) |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
164 ): # skip translation chunks that doesn't have any actual translations |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
165 continue |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
166 chunks.append('\n'.join(chunk_lines) + '\n') |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
167 return '\n'.join(chunks) |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
168 |
8317
5553ecc962e0
scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents:
8315
diff
changeset
|
169 def _normalize_po_file(po_file, merge_pot_file=None, strip=False): |
5553ecc962e0
scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents:
8315
diff
changeset
|
170 if merge_pot_file: |
5553ecc962e0
scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents:
8315
diff
changeset
|
171 runcmd(['msgmerge', '--width=76', '--backup=none', '--previous', |
5553ecc962e0
scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents:
8315
diff
changeset
|
172 '--update', po_file, '-q', merge_pot_file]) |
8314
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
173 if strip: |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
174 po_tmp = po_file + '.tmp' |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
175 with open(po_file, 'r') as src, open(po_tmp, 'w') as dest: |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
176 raw_content = src.read() |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
177 normalized_content = _normalize_po(raw_content) |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
178 dest.write(normalized_content) |
ae9d205f4407
scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8313
diff
changeset
|
179 os.rename(po_tmp, po_file) |
8315
93dabafa567e
scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8314
diff
changeset
|
180 |
8317
5553ecc962e0
scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents:
8315
diff
changeset
|
181 def _normalized_diff(file1, file2, merge_pot_file=None, strip=False): |
8315
93dabafa567e
scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8314
diff
changeset
|
182 # Create temporary copies of both files |
93dabafa567e
scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8314
diff
changeset
|
183 temp1 = tempfile.NamedTemporaryFile(prefix=os.path.basename(file1)) |
93dabafa567e
scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8314
diff
changeset
|
184 temp2 = tempfile.NamedTemporaryFile(prefix=os.path.basename(file2)) |
93dabafa567e
scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8314
diff
changeset
|
185 debug('normalized_diff: %s -> %s / %s -> %s' % (file1, temp1.name, file2, temp2.name)) |
93dabafa567e
scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8314
diff
changeset
|
186 shutil.copyfile(file1, temp1.name) |
93dabafa567e
scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8314
diff
changeset
|
187 shutil.copyfile(file2, temp2.name) |
93dabafa567e
scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8314
diff
changeset
|
188 # Normalize them in place |
8317
5553ecc962e0
scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents:
8315
diff
changeset
|
189 _normalize_po_file(temp1.name, merge_pot_file=merge_pot_file, strip=strip) |
5553ecc962e0
scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents:
8315
diff
changeset
|
190 _normalize_po_file(temp2.name, merge_pot_file=merge_pot_file, strip=strip) |
8315
93dabafa567e
scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8314
diff
changeset
|
191 # Now compare |
93dabafa567e
scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8314
diff
changeset
|
192 try: |
93dabafa567e
scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8314
diff
changeset
|
193 runcmd(['diff', '-u', temp1.name, temp2.name]) |
93dabafa567e
scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8314
diff
changeset
|
194 except subprocess.CalledProcessError as e: |
93dabafa567e
scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
8314
diff
changeset
|
195 return e.returncode |