Mercurial > kallithea

annotate scripts/i18n_utils.py @ 8333:e505395c3a03

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
i18n: better stripping of header comment for new translations Also strip any header occurrence of: # FIRST AUTHOR <EMAIL@ADDRESS>, 2019. # while preserving stripping of: #, fuzzy and leading empty comment lines.
author Mads Kiilerich <mads@kiilerich.com>
date Sun, 02 Feb 2020 21:29:18 +0100
parents 30e137b4ff18
children 19735bc60455
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
8313
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
1 # This program is free software: you can redistribute it and/or modify
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
2 # it under the terms of the GNU General Public License as published by
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
3 # the Free Software Foundation, either version 3 of the License, or
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
4 # (at your option) any later version.
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
5 #
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
6 # This program is distributed in the hope that it will be useful,
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
7 # but WITHOUT ANY WARRANTY; without even the implied warranty of
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
8 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
9 # GNU General Public License for more details.
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
10 #
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
11 # You should have received a copy of the GNU General Public License
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
12 # along with this program. If not, see <http://www.gnu.org/licenses/>.
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
13
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
14 from __future__ import print_function
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
15
8314
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
16 import os
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
17 import re
8315
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
18 import shutil
8313
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
19 import subprocess
8315
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
20 import tempfile
8313
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
21
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
22
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
23 do_debug = False # set from scripts/i18n --debug
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
24
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
25 def debug(*args, **kwargs):
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
26 if do_debug:
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
27 print(*args, **kwargs)
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
28
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
29 def runcmd(cmd, *args, **kwargs):
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
30 debug('... Executing command: %s' % ' '.join(cmd))
4bc712f1ec93 scripts/i18n: introduce new i18n maintenance script
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents:
diff changeset
31 subprocess.check_call(cmd, *args, **kwargs)
8314
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
32
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
33 header_comment_strip_re = re.compile(r'''
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
34 ^
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
35 [#][ ]Translations[ ]template[ ]for[ ]Kallithea[.] \n
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
36 |
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
37 ^
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
38 [#][ ]FIRST[ ]AUTHOR[ ]<EMAIL@ADDRESS>,[ ]\d+[.] \n
8333
e505395c3a03 i18n: better stripping of header comment for new translations
Mads Kiilerich <mads@kiilerich.com>
parents: 8318
diff changeset
39 (?:[#] \n)?
e505395c3a03 i18n: better stripping of header comment for new translations
Mads Kiilerich <mads@kiilerich.com>
parents: 8318
diff changeset
40 |
e505395c3a03 i18n: better stripping of header comment for new translations
Mads Kiilerich <mads@kiilerich.com>
parents: 8318
diff changeset
41 ^
e505395c3a03 i18n: better stripping of header comment for new translations
Mads Kiilerich <mads@kiilerich.com>
parents: 8318
diff changeset
42 (?:[#] \n)?
8314
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
43 [#],[ ]fuzzy \n
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
44 ''',
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
45 re.MULTILINE|re.VERBOSE)
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
46
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
47 header_normalize_re = re.compile(r'''
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
48 ^ "
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
49 (POT-Creation-Date|PO-Revision-Date|Last-Translator|Language-Team|X-Generator|Generated-By|Project-Id-Version):
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
50 [ ][^\\]*\\n
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
51 " \n
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
52 ''',
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
53 re.MULTILINE|re.IGNORECASE|re.VERBOSE)
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
54
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
55 def _normalize_po(raw_content):
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
56 r"""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
57 >>> print(_normalize_po(r'''
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
58 ... # header comment
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
59 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
60 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
61 ... # comment before header
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
62 ... msgid ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
63 ... msgstr "yada"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
64 ... "POT-Creation-Date: 2019-05-04 21:13+0200\n"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
65 ... "MIME-Version: "
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
66 ... "1.0\n"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
67 ... "Last-Translator: Jabba"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
68 ... "the Hutt\n"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
69 ... "X-Generator: Weblate 1.2.3\n"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
70 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
71 ... # comment, but not in header
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
72 ... msgid "None"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
73 ... msgstr "Ingen"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
74 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
75 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
76 ... line 2
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
77 ... # third comment
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
78 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
79 ... msgid "Special"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
80 ... msgstr ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
81 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
82 ... msgid "Specialist"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
83 ... # odd comment
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
84 ... msgstr ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
85 ... "Expert"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
86 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
87 ... # crazy fuzzy auto translation by msgmerge, using foo for bar
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
88 ... #, fuzzy
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
89 ... #| msgid "some foo string"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
90 ... msgid "some bar string."
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
91 ... msgstr "translation of foo string"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
92 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
93 ... msgid "%d minute"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
94 ... msgid_plural "%d minutes"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
95 ... msgstr[0] "minut"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
96 ... msgstr[1] "minutter"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
97 ... msgstr[2] ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
98 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
99 ... msgid "%d year"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
100 ... msgid_plural "%d years"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
101 ... msgstr[0] ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
102 ... msgstr[1] ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
103 ...
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
104 ... # last comment
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
105 ... ''') + '^^^')
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
106 # header comment
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
107 <BLANKLINE>
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
108 <BLANKLINE>
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
109 # comment before header
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
110 <BLANKLINE>
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
111 msgid ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
112 msgstr "yada"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
113 "MIME-Version: "
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
114 "1.0\n"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
115 <BLANKLINE>
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
116 msgid "None"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
117 msgstr "Ingen"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
118 <BLANKLINE>
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
119 line 2
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
120 <BLANKLINE>
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
121 msgid "Specialist"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
122 msgstr ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
123 "Expert"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
124 <BLANKLINE>
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
125 msgid "%d minute"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
126 msgid_plural "%d minutes"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
127 msgstr[0] "minut"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
128 msgstr[1] "minutter"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
129 msgstr[2] ""
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
130 ^^^
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
131 """
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
132 header_start = raw_content.find('\nmsgid ""\n') + 1
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
133 header_end = raw_content.find('\n\n', header_start) + 1 or len(raw_content)
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
134 chunks = [
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
135 header_comment_strip_re.sub('', raw_content[0:header_start])
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
136 .strip(),
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
137 '',
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
138 header_normalize_re.sub('', raw_content[header_start:header_end])
8318
30e137b4ff18 scripts/i18n: also normalize casing of UTF-8 in Content-Type
Mads Kiilerich <mads@kiilerich.com>
parents: 8317
diff changeset
139 .replace(
30e137b4ff18 scripts/i18n: also normalize casing of UTF-8 in Content-Type
Mads Kiilerich <mads@kiilerich.com>
parents: 8317
diff changeset
140 r'"Content-Type: text/plain; charset=utf-8\n"',
30e137b4ff18 scripts/i18n: also normalize casing of UTF-8 in Content-Type
Mads Kiilerich <mads@kiilerich.com>
parents: 8317
diff changeset
141 r'"Content-Type: text/plain; charset=UTF-8\n"') # maintain msgmerge casing
8314
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
142 .strip(),
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
143 ''] # preserve normalized header
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
144 # all chunks are separated by empty line
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
145 for raw_chunk in raw_content[header_end:].split('\n\n'):
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
146 if '\n#, fuzzy' in raw_chunk: # might be like "#, fuzzy, python-format"
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
147 continue # drop crazy auto translation that is worse than useless
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
148 # strip all comment lines from chunk
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
149 chunk_lines = [
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
150 line
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
151 for line in raw_chunk.splitlines()
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
152 if line
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
153 and not line.startswith('#')
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
154 ]
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
155 if not chunk_lines:
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
156 continue
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
157 # check lines starting from first msgstr, skip chunk if no translation lines
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
158 msgstr_i = [i for i, line in enumerate(chunk_lines) if line.startswith('msgstr')]
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
159 if (
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
160 chunk_lines[0].startswith('msgid') and
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
161 msgstr_i and
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
162 all(line.endswith(' ""') for line in chunk_lines[msgstr_i[0]:])
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
163 ): # skip translation chunks that doesn't have any actual translations
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
164 continue
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
165 chunks.append('\n'.join(chunk_lines) + '\n')
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
166 return '\n'.join(chunks)
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
167
8317
5553ecc962e0 scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents: 8315
diff changeset
168 def _normalize_po_file(po_file, merge_pot_file=None, strip=False):
5553ecc962e0 scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents: 8315
diff changeset
169 if merge_pot_file:
5553ecc962e0 scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents: 8315
diff changeset
170 runcmd(['msgmerge', '--width=76', '--backup=none', '--previous',
5553ecc962e0 scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents: 8315
diff changeset
171 '--update', po_file, '-q', merge_pot_file])
8314
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
172 if strip:
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
173 po_tmp = po_file + '.tmp'
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
174 with open(po_file, 'r') as src, open(po_tmp, 'w') as dest:
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
175 raw_content = src.read()
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
176 normalized_content = _normalize_po(raw_content)
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
177 dest.write(normalized_content)
ae9d205f4407 scripts/i18n: add command 'normalize-po-files'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8313
diff changeset
178 os.rename(po_tmp, po_file)
8315
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
179
8317
5553ecc962e0 scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents: 8315
diff changeset
180 def _normalized_diff(file1, file2, merge_pot_file=None, strip=False):
8315
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
181 # Create temporary copies of both files
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
182 temp1 = tempfile.NamedTemporaryFile(prefix=os.path.basename(file1))
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
183 temp2 = tempfile.NamedTemporaryFile(prefix=os.path.basename(file2))
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
184 debug('normalized_diff: %s -> %s / %s -> %s' % (file1, temp1.name, file2, temp2.name))
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
185 shutil.copyfile(file1, temp1.name)
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
186 shutil.copyfile(file2, temp2.name)
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
187 # Normalize them in place
8317
5553ecc962e0 scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents: 8315
diff changeset
188 _normalize_po_file(temp1.name, merge_pot_file=merge_pot_file, strip=strip)
5553ecc962e0 scripts/i18n: introduce --merge-pot-file to control normalization
Mads Kiilerich <mads@kiilerich.com>
parents: 8315
diff changeset
189 _normalize_po_file(temp2.name, merge_pot_file=merge_pot_file, strip=strip)
8315
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
190 # Now compare
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
191 try:
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
192 runcmd(['diff', '-u', temp1.name, temp2.name])
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
193 except subprocess.CalledProcessError as e:
93dabafa567e scripts/i18n: add command 'normalized-diff'
Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
parents: 8314
diff changeset
194 return e.returncode