Mercurial > kallithea

annotate rhodecode/bin/rhodecode_backup.py @ 3574:000653f7cdf9 beta

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
avoid displaying repr of internal classes in user facing messages The context of the message will tell where the problem was and there is no reason to show ... does not exist for this repository <MercurialRepository at /home/marcink/repos/rhodecode>
author Mads Kiilerich <madski@unity3d.com>
date Thu, 21 Mar 2013 11:17:01 +0100
parents 7ac09514a178
children 63e58ef80ef1
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1307
c1516b35f91d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
1 # -*- coding: utf-8 -*-
c1516b35f91d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
2 """
2379
7ac09514a178 created rhodecode-api binary script for working with api via cli
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
3 rhodecode.bin.backup_manager
1307
c1516b35f91d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
5
2379
7ac09514a178 created rhodecode-api binary script for working with api via cli
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
6 Repositories backup manager, it allows to backups all
1307
c1516b35f91d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
7 repositories and send it to backup server using RSA key via ssh.
c1516b35f91d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
8
c1516b35f91d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
9 :created_on: Feb 28, 2010
1824
89efedac4e6c 2012 copyrights
Marcin Kuzminski <marcin@python-works.com>
parents: 1818
diff changeset
10 :author: marcink
89efedac4e6c 2012 copyrights
Marcin Kuzminski <marcin@python-works.com>
parents: 1818
diff changeset
11 :copyright: (C) 2010-2012 Marcin Kuzminski <marcin@python-works.com>
1532
2afe9320d5e6 updated docstrings
Marcin Kuzminski <marcin@python-works.com>
parents: 1307
diff changeset
12 :license: GPLv3, see COPYING for more details.
1307
c1516b35f91d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
13 """
1206
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
14 # This program is free software: you can redistribute it and/or modify
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
15 # it under the terms of the GNU General Public License as published by
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
16 # the Free Software Foundation, either version 3 of the License, or
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
17 # (at your option) any later version.
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
18 #
252
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
19 # This program is distributed in the hope that it will be useful,
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
20 # but WITHOUT ANY WARRANTY; without even the implied warranty of
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
21 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
22 # GNU General Public License for more details.
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
23 #
252
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
24 # You should have received a copy of the GNU General Public License
1206
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
25 # along with this program. If not, see <http://www.gnu.org/licenses/>.
252
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
26
1307
c1516b35f91d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
27 import os
c1516b35f91d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
28 import sys
252
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
29
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
30 import logging
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
31 import tarfile
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
32 import datetime
28
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
33 import subprocess
1307
c1516b35f91d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
34
33
22b2234e51cd Added removing of files from data
Marcin Kuzminski <marcin@python-blog.com>
parents: 29
diff changeset
35 logging.basicConfig(level=logging.DEBUG,
22b2234e51cd Added removing of files from data
Marcin Kuzminski <marcin@python-blog.com>
parents: 29
diff changeset
36 format="%(asctime)s %(levelname)-5.5s %(message)s")
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
37
1307
c1516b35f91d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
38
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
39 class BackupManager(object):
434
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
40 def __init__(self, repos_location, rsa_key, backup_server):
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
41 today = datetime.datetime.now().weekday() + 1
2379
7ac09514a178 created rhodecode-api binary script for working with api via cli
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
42 self.backup_file_name = "rhodecode_repos.%s.tar.gz" % today
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
43
434
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
44 self.id_rsa_path = self.get_id_rsa(rsa_key)
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
45 self.repos_path = self.get_repos_path(repos_location)
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
46 self.backup_server = backup_server
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
47
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
48 self.backup_file_path = '/tmp'
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
49
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
50 logging.info('starting backup for %s', self.repos_path)
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
51 logging.info('backup target %s', self.backup_file_path)
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
52
434
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
53 def get_id_rsa(self, rsa_key):
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
54 if not os.path.isfile(rsa_key):
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
55 logging.error('Could not load id_rsa key file in %s', rsa_key)
28
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
56 sys.exit()
440
9fe5ca6d72f6 fixed stupid typo in backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 434
diff changeset
57 return rsa_key
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
58
434
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
59 def get_repos_path(self, path):
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
60 if not os.path.isdir(path):
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
61 logging.error('Wrong location for repositories in %s', path)
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
62 sys.exit()
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
63 return path
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
64
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
65 def backup_repos(self):
28
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
66 bckp_file = os.path.join(self.backup_file_path, self.backup_file_name)
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
67 tar = tarfile.open(bckp_file, "w:gz")
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
68
34
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
69 for dir_name in os.listdir(self.repos_path):
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
70 logging.info('backing up %s', dir_name)
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
71 tar.add(os.path.join(self.repos_path, dir_name), dir_name)
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
72 tar.close()
26
b3307ca60bfb Litle info messages update
Marcin Kuzminski
parents: 25
diff changeset
73 logging.info('finished backup of mercurial repositories')
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
74
28
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
75 def transfer_files(self):
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
76 params = {
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
77 'id_rsa_key': self.id_rsa_path,
1307
c1516b35f91d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
78 'backup_file': os.path.join(self.backup_file_path,
434
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
79 self.backup_file_name),
1307
c1516b35f91d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
80 'backup_server': self.backup_server
28
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
81 }
39
787e7d307b69 fix bug in arguments for command
marcink
parents: 38
diff changeset
82 cmd = ['scp', '-l', '40000', '-i', '%(id_rsa_key)s' % params,
434
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
83 '%(backup_file)s' % params,
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
84 '%(backup_server)s' % params]
28
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
85
35
c92f827280a3 Changed to blocking operation
Marcin Kuzminski <marcin@python-blog.com>
parents: 34
diff changeset
86 subprocess.call(cmd)
28
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
87 logging.info('Transfered file %s to %s', self.backup_file_name, cmd[4])
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
88
33
22b2234e51cd Added removing of files from data
Marcin Kuzminski <marcin@python-blog.com>
parents: 29
diff changeset
89 def rm_file(self):
36
846e2d8672c4 Added message when file removing
Marcin Kuzminski <marcin@python-blog.com>
parents: 35
diff changeset
90 logging.info('Removing file %s', self.backup_file_name)
34
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
91 os.remove(os.path.join(self.backup_file_path, self.backup_file_name))
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
92
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
93 if __name__ == "__main__":
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
94
434
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
95 repo_location = '/home/repo_path'
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
96 backup_server = 'root@192.168.1.100:/backups/mercurial'
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
97 rsa_key = '/home/id_rsa'
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
98
434
bd3f341cad45 simplified and updated backup manager
Marcin Kuzminski <marcin@python-works.com>
parents: 252
diff changeset
99 B_MANAGER = BackupManager(repo_location, rsa_key, backup_server)
34
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
100 B_MANAGER.backup_repos()
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
101 B_MANAGER.transfer_files()
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
102 B_MANAGER.rm_file()