Mercurial > kallithea

annotate rhodecode/bin/rhodecode_backup.py @ 4147:1c8f818787b3 rhodecode-2.2.5-gpl

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
old style: show the full link box on summary page - no overlap or truncation
author Mads Kiilerich <madski@unity3d.com>
date Wed, 02 Jul 2014 19:03:23 -0400
parents ffd45b185016
children 7e5f8c12a3fc
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1307
c1516b35f91d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
1 # -*- coding: utf-8 -*-
1206
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
2 # This program is free software: you can redistribute it and/or modify
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
3 # it under the terms of the GNU General Public License as published by
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
4 # the Free Software Foundation, either version 3 of the License, or
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
5 # (at your option) any later version.
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
6 #
252
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
7 # This program is distributed in the hope that it will be useful,
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
8 # but WITHOUT ANY WARRANTY; without even the implied warranty of
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
9 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
10 # GNU General Public License for more details.
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
11 #
252
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
12 # You should have received a copy of the GNU General Public License
1206
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
13 # along with this program. If not, see <http://www.gnu.org/licenses/>.
4116
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 2776
diff changeset
14 """
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 2776
diff changeset
15 rhodecode.bin.backup_manager
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 2776
diff changeset
16 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 2776
diff changeset
17
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 2776
diff changeset
18 Repositories backup manager, it allows to backups all
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 2776
diff changeset
19 repositories and send it to backup server using RSA key via ssh.
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 2776
diff changeset
20
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 2776
diff changeset
21 :created_on: Feb 28, 2010
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 2776
diff changeset
22 :author: marcink
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 2776
diff changeset
23 :copyright: (c) 2013 RhodeCode GmbH.
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 2776
diff changeset
24 :license: GPLv3, see LICENSE for more details.
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 2776
diff changeset
25 """
252
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
26
1307
c1516b35f91d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
27 import os
c1516b35f91d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
28 import sys
252
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
29
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
30 import logging
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
31 import tarfile
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
32 import datetime
28
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
33 import subprocess
1307
c1516b35f91d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
34
33
22b2234e51cd Added removing of files from data
Marcin Kuzminski <marcin@python-blog.com>
parents: 29
diff changeset
35 logging.basicConfig(level=logging.DEBUG,
22b2234e51cd Added removing of files from data
Marcin Kuzminski <marcin@python-blog.com>
parents: 29
diff changeset
36 format="%(asctime)s %(levelname)-5.5s %(message)s")
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
37
1307
c1516b35f91d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
38
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
39 class BackupManager(object):
434
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
40 def __init__(self, repos_location, rsa_key, backup_server):
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
41 today = datetime.datetime.now().weekday() + 1
2379
7ac09514a178 created rhodecode-api binary script for working with api via cli
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
42 self.backup_file_name = "rhodecode_repos.%s.tar.gz" % today
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
43
434
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
44 self.id_rsa_path = self.get_id_rsa(rsa_key)
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
45 self.repos_path = self.get_repos_path(repos_location)
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
46 self.backup_server = backup_server
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
47
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
48 self.backup_file_path = '/tmp'
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
49
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
50 logging.info('starting backup for %s', self.repos_path)
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
51 logging.info('backup target %s', self.backup_file_path)
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
52
434
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
53 def get_id_rsa(self, rsa_key):
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
54 if not os.path.isfile(rsa_key):
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
55 logging.error('Could not load id_rsa key file in %s', rsa_key)
28
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
56 sys.exit()
440
9fe5ca6d72f6 fixed stupid typo in backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 434
diff changeset
57 return rsa_key
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
58
434
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
59 def get_repos_path(self, path):
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
60 if not os.path.isdir(path):
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
61 logging.error('Wrong location for repositories in %s', path)
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
62 sys.exit()
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
63 return path
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
64
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
65 def backup_repos(self):
28
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
66 bckp_file = os.path.join(self.backup_file_path, self.backup_file_name)
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
67 tar = tarfile.open(bckp_file, "w:gz")
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
68
34
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
69 for dir_name in os.listdir(self.repos_path):
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
70 logging.info('backing up %s', dir_name)
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
71 tar.add(os.path.join(self.repos_path, dir_name), dir_name)
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
72 tar.close()
26
b3307ca60bfb Litle info messages update
Marcin Kuzminski
parents: 25
diff changeset
73 logging.info('finished backup of mercurial repositories')
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
74
28
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
75 def transfer_files(self):
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
76 params = {
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
77 'id_rsa_key': self.id_rsa_path,
1307
c1516b35f91d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
78 'backup_file': os.path.join(self.backup_file_path,
434
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
79 self.backup_file_name),
1307
c1516b35f91d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
80 'backup_server': self.backup_server
28
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
81 }
39
787e7d307b69 fix bug in arguments for command
marcink
parents: 38
diff changeset
82 cmd = ['scp', '-l', '40000', '-i', '%(id_rsa_key)s' % params,
434
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
83 '%(backup_file)s' % params,
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
84 '%(backup_server)s' % params]
28
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
85
35
c92f827280a3 Changed to blocking operation
Marcin Kuzminski <marcin@python-blog.com>
parents: 34
diff changeset
86 subprocess.call(cmd)
28
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
87 logging.info('Transfered file %s to %s', self.backup_file_name, cmd[4])
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
88
33
22b2234e51cd Added removing of files from data
Marcin Kuzminski <marcin@python-blog.com>
parents: 29
diff changeset
89 def rm_file(self):
36
846e2d8672c4 Added message when file removing
Marcin Kuzminski <marcin@python-blog.com>
parents: 35
diff changeset
90 logging.info('Removing file %s', self.backup_file_name)
34
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
91 os.remove(os.path.join(self.backup_file_path, self.backup_file_name))
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
92
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
93 if __name__ == "__main__":
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
94
434
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
95 repo_location = '/home/repo_path'
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
96 backup_server = 'root@192.168.1.100:/backups/mercurial'
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
97 rsa_key = '/home/id_rsa'
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
98
434
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
99 B_MANAGER = BackupManager(repo_location, rsa_key, backup_server)
34
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
100 B_MANAGER.backup_repos()
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
101 B_MANAGER.transfer_files()
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
102 B_MANAGER.rm_file()