annotate scripts/i18n_utils.py @ 8751:ad239692ea95

mail: fix duplicate "From" headers Problem introduced in 9a0c41175e66: When iterating the headers dict and setting "msg[key] = value", it wasn't replacing the header but performing add_header so we sometimes ended up with two From headers. It is also a general problem that while the headers dict only can contain each key once, it can contain entries that only differ in casing and thus will fold to the same message header, making it possible to end up adding duplicate headers. "msg.replace_header(key, value)" is not a simple solution to the problem: it will raise KeyError if no such previous key exists. Now, make the problem more clear by explicitly using add_header. Avoid the duplication problem by deleting the key (no matter which casing) before invoking add_header. Delete promises that "No exception is raised if the named field isn’t present in the headers".
author Mads Kiilerich <mads@kiilerich.com>
date Wed, 04 Nov 2020 00:35:21 +0100
parents 6bde1c0a04d4
children 0c9c91ac3873
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
8313
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
1 # This program is free software: you can redistribute it and/or modify
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
2 # it under the terms of the GNU General Public License as published by
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
3 # the Free Software Foundation, either version 3 of the License, or
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
4 # (at your option) any later version.
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
5 #
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
6 # This program is distributed in the hope that it will be useful,
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
7 # but WITHOUT ANY WARRANTY; without even the implied warranty of
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
8 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
9 # GNU General Public License for more details.
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
10 #
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
11 # You should have received a copy of the GNU General Public License
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
12 # along with this program. If not, see <http://www.gnu.org/licenses/>.
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
13
8314
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
14 import os
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
15 import re
8315
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
16 import shutil
8313
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
17 import subprocess
8315
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
18 import tempfile
8313
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
19
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
20
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
21 do_debug = False # set from scripts/i18n --debug
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
22
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
23 def debug(*args, **kwargs):
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
24 if do_debug:
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
25 print(*args, **kwargs)
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
26
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
27 def runcmd(cmd, *args, **kwargs):
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
28 debug('... Executing command: %s' % ' '.join(cmd))
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
29 subprocess.check_call(cmd, *args, **kwargs)
8314
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
30
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
31 header_comment_strip_re = re.compile(r'''
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
32 ^
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
33 [#][ ]Translations[ ]template[ ]for[ ]Kallithea[.] \n
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
34 |
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
35 ^
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
36 [#][ ]FIRST[ ]AUTHOR[ ]<EMAIL@ADDRESS>,[ ]\d+[.] \n
8333
e505395c3a03 i18n: better stripping of header comment for new translations
Mads Kiilerich <mads@kiilerich.com>
parents: 8318
diff changeset
37 (?:[#] \n)?
e505395c3a03 i18n: better stripping of header comment for new translations
Mads Kiilerich <mads@kiilerich.com>
parents: 8318
diff changeset
38 |
e505395c3a03 i18n: better stripping of header comment for new translations
Mads Kiilerich <mads@kiilerich.com>
parents: 8318
diff changeset
39 ^
e505395c3a03 i18n: better stripping of header comment for new translations
Mads Kiilerich <mads@kiilerich.com>
parents: 8318
diff changeset
40 (?:[#] \n)?
8314
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
41 [#],[ ]fuzzy \n
8334
19735bc60455 i18n: also strip '# #, fuzzy' from header comment - it might appear when verifying branches are in sync
Mads Kiilerich <mads@kiilerich.com>
parents: 8333
diff changeset
42 |
19735bc60455 i18n: also strip '# #, fuzzy' from header comment - it might appear when verifying branches are in sync
Mads Kiilerich <mads@kiilerich.com>
parents: 8333
diff changeset
43 ^
19735bc60455 i18n: also strip '# #, fuzzy' from header comment - it might appear when verifying branches are in sync
Mads Kiilerich <mads@kiilerich.com>
parents: 8333
diff changeset
44 [#][ ][#],[ ]fuzzy \n
8314
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
45 ''',
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
46 re.MULTILINE|re.VERBOSE)
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
47
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
48 header_normalize_re = re.compile(r'''
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
49 ^ "
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
50 (POT-Creation-Date|PO-Revision-Date|Last-Translator|Language-Team|X-Generator|Generated-By|Project-Id-Version):
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
51 [ ][^\\]*\\n
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
52 " \n
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
53 ''',
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
54 re.MULTILINE|re.IGNORECASE|re.VERBOSE)
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
55
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
56 def _normalize_po(raw_content):
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
57 r"""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
58 >>> print(_normalize_po(r'''
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
59 ... # header comment
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
60 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
61 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
62 ... # comment before header
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
63 ... msgid ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
64 ... msgstr "yada"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
65 ... "POT-Creation-Date: 2019-05-04 21:13+0200\n"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
66 ... "MIME-Version: "
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
67 ... "1.0\n"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
68 ... "Last-Translator: Jabba"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
69 ... "the Hutt\n"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
70 ... "X-Generator: Weblate 1.2.3\n"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
71 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
72 ... # comment, but not in header
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
73 ... msgid "None"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
74 ... msgstr "Ingen"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
75 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
76 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
77 ... line 2
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
78 ... # third comment
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
79 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
80 ... msgid "Special"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
81 ... msgstr ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
82 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
83 ... msgid "Specialist"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
84 ... # odd comment
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
85 ... msgstr ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
86 ... "Expert"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
87 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
88 ... # crazy fuzzy auto translation by msgmerge, using foo for bar
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
89 ... #, fuzzy
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
90 ... #| msgid "some foo string"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
91 ... msgid "some bar string."
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
92 ... msgstr "translation of foo string"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
93 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
94 ... msgid "%d minute"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
95 ... msgid_plural "%d minutes"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
96 ... msgstr[0] "minut"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
97 ... msgstr[1] "minutter"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
98 ... msgstr[2] ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
99 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
100 ... msgid "%d year"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
101 ... msgid_plural "%d years"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
102 ... msgstr[0] ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
103 ... msgstr[1] ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
104 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
105 ... # last comment
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
106 ... ''') + '^^^')
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
107 # header comment
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
108 <BLANKLINE>
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
109 <BLANKLINE>
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
110 # comment before header
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
111 <BLANKLINE>
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
112 msgid ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
113 msgstr "yada"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
114 "MIME-Version: "
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
115 "1.0\n"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
116 <BLANKLINE>
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
117 msgid "None"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
118 msgstr "Ingen"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
119 <BLANKLINE>
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
120 line 2
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
121 <BLANKLINE>
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
122 msgid "Specialist"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
123 msgstr ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
124 "Expert"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
125 <BLANKLINE>
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
126 msgid "%d minute"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
127 msgid_plural "%d minutes"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
128 msgstr[0] "minut"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
129 msgstr[1] "minutter"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
130 msgstr[2] ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
131 ^^^
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
132 """
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
133 header_start = raw_content.find('\nmsgid ""\n') + 1
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
134 header_end = raw_content.find('\n\n', header_start) + 1 or len(raw_content)
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
135 chunks = [
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
136 header_comment_strip_re.sub('', raw_content[0:header_start])
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
137 .strip(),
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
138 '',
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
139 header_normalize_re.sub('', raw_content[header_start:header_end])
8318
30e137b4ff18 scripts/i18n: also normalize casing of UTF-8 in Content-Type
Mads Kiilerich <mads@kiilerich.com>
parents: 8317
diff changeset
140 .replace(
30e137b4ff18 scripts/i18n: also normalize casing of UTF-8 in Content-Type
Mads Kiilerich <mads@kiilerich.com>
parents: 8317
diff changeset
141 r'"Content-Type: text/plain; charset=utf-8\n"',
30e137b4ff18 scripts/i18n: also normalize casing of UTF-8 in Content-Type
Mads Kiilerich <mads@kiilerich.com>
parents: 8317
diff changeset
142 r'"Content-Type: text/plain; charset=UTF-8\n"') # maintain msgmerge casing
8314
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
143 .strip(),
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
144 ''] # preserve normalized header
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
145 # all chunks are separated by empty line
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
146 for raw_chunk in raw_content[header_end:].split('\n\n'):
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
147 if '\n#, fuzzy' in raw_chunk: # might be like "#, fuzzy, python-format"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
148 continue # drop crazy auto translation that is worse than useless
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
149 # strip all comment lines from chunk
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
150 chunk_lines = [
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
151 line
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
152 for line in raw_chunk.splitlines()
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
153 if line
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
154 and not line.startswith('#')
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
155 ]
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
156 if not chunk_lines:
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
157 continue
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
158 # check lines starting from first msgstr, skip chunk if no translation lines
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
159 msgstr_i = [i for i, line in enumerate(chunk_lines) if line.startswith('msgstr')]
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
160 if (
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
161 chunk_lines[0].startswith('msgid') and
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
162 msgstr_i and
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
163 all(line.endswith(' ""') for line in chunk_lines[msgstr_i[0]:])
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
164 ): # skip translation chunks that doesn't have any actual translations
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
165 continue
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
166 chunks.append('\n'.join(chunk_lines) + '\n')
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
167 return '\n'.join(chunks)
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
168
8317
5553ecc962e0 scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents: 8315
diff changeset
169 def _normalize_po_file(po_file, merge_pot_file=None, strip=False):
5553ecc962e0 scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents: 8315
diff changeset
170 if merge_pot_file:
5553ecc962e0 scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents: 8315
diff changeset
171 runcmd(['msgmerge', '--width=76', '--backup=none', '--previous',
5553ecc962e0 scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents: 8315
diff changeset
172 '--update', po_file, '-q', merge_pot_file])
8314
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
173 if strip:
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
174 po_tmp = po_file + '.tmp'
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
175 with open(po_file, 'r') as src, open(po_tmp, 'w') as dest:
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
176 raw_content = src.read()
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
177 normalized_content = _normalize_po(raw_content)
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
178 dest.write(normalized_content)
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
179 os.rename(po_tmp, po_file)
8315
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
180
8317
5553ecc962e0 scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents: 8315
diff changeset
181 def _normalized_diff(file1, file2, merge_pot_file=None, strip=False):
8315
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
182 # Create temporary copies of both files
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
183 temp1 = tempfile.NamedTemporaryFile(prefix=os.path.basename(file1))
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
184 temp2 = tempfile.NamedTemporaryFile(prefix=os.path.basename(file2))
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
185 debug('normalized_diff: %s -> %s / %s -> %s' % (file1, temp1.name, file2, temp2.name))
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
186 shutil.copyfile(file1, temp1.name)
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
187 shutil.copyfile(file2, temp2.name)
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
188 # Normalize them in place
8317
5553ecc962e0 scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents: 8315
diff changeset
189 _normalize_po_file(temp1.name, merge_pot_file=merge_pot_file, strip=strip)
5553ecc962e0 scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents: 8315
diff changeset
190 _normalize_po_file(temp2.name, merge_pot_file=merge_pot_file, strip=strip)
8315
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
191 # Now compare
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
192 try:
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
193 runcmd(['diff', '-u', temp1.name, temp2.name])
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
194 except subprocess.CalledProcessError as e:
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
195 return e.returncode