Mercurial > kallithea

annotate pylons_app/lib/backup_manager.py @ 415:04e8b31fb245

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Changed password crypting scheme to bcrypt, added dependency for setup
author Marcin Kuzminski <marcin@python-works.com>
date Fri, 20 Aug 2010 10:59:18 +0200
parents 3782a6d698af
children bd3f341cad45
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
252
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
1 #!/usr/bin/env python
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
2 # encoding: utf-8
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
3 # mercurial repository backup manager
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
4 # Copyright (C) 2009-2010 Marcin Kuzminski <marcin@python-works.com>
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
5
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
6 # This program is free software; you can redistribute it and/or
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
7 # modify it under the terms of the GNU General Public License
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
8 # as published by the Free Software Foundation; version 2
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
9 # of the License or (at your opinion) any later version of the license.
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
10 #
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
11 # This program is distributed in the hope that it will be useful,
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
12 # but WITHOUT ANY WARRANTY; without even the implied warranty of
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
13 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
14 # GNU General Public License for more details.
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
15 #
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
16 # You should have received a copy of the GNU General Public License
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
17 # along with this program; if not, write to the Free Software
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
18 # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
19 # MA 02110-1301, USA.
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
20
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
21 """
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
22 Created on Feb 28, 2010
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
23 Mercurial repositories backup manager
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
24 @author: marcink
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
25 """
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
26
3782a6d698af licensing updates, code cleanups
Marcin Kuzminski <marcin@python-works.com>
parents: 241
diff changeset
27
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
28 import logging
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
29 from mercurial import config
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
30 import tarfile
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
31 import os
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
32 import datetime
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
33 import sys
28
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
34 import subprocess
33
22b2234e51cd Added removing of files from data
Marcin Kuzminski <marcin@python-blog.com>
parents: 29
diff changeset
35 logging.basicConfig(level=logging.DEBUG,
22b2234e51cd Added removing of files from data
Marcin Kuzminski <marcin@python-blog.com>
parents: 29
diff changeset
36 format="%(asctime)s %(levelname)-5.5s %(message)s")
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
37
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
38 class BackupManager(object):
241
48727add84c9 Made repos path config configurable from pylons app configs. update Readme
Marcin Kuzminski <marcin@python-works.com>
parents: 39
diff changeset
39 def __init__(self, id_rsa_path, repo_conf):
34
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
40 self.repos_path = None
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
41 self.backup_file_name = None
241
48727add84c9 Made repos path config configurable from pylons app configs. update Readme
Marcin Kuzminski <marcin@python-works.com>
parents: 39
diff changeset
42 self.id_rsa_path = id_rsa_path
28
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
43 self.check_id_rsa()
27
8f29ddc4c147 fixed the backup script path problems
Marcin Kuzminski
parents: 26
diff changeset
44 cur_dir = os.path.realpath(__file__)
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
45 dn = os.path.dirname
27
8f29ddc4c147 fixed the backup script path problems
Marcin Kuzminski
parents: 26
diff changeset
46 self.backup_file_path = os.path.join(dn(dn(dn(cur_dir))), 'data')
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
47 cfg = config.config()
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
48 try:
241
48727add84c9 Made repos path config configurable from pylons app configs. update Readme
Marcin Kuzminski <marcin@python-works.com>
parents: 39
diff changeset
49 cfg.read(os.path.join(dn(dn(dn(cur_dir))), repo_conf))
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
50 except IOError:
241
48727add84c9 Made repos path config configurable from pylons app configs. update Readme
Marcin Kuzminski <marcin@python-works.com>
parents: 39
diff changeset
51 logging.error('Could not read %s', repo_conf)
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
52 sys.exit()
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
53 self.set_repos_path(cfg.items('paths'))
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
54 logging.info('starting backup for %s', self.repos_path)
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
55 logging.info('backup target %s', self.backup_file_path)
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
56
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
57 if not os.path.isdir(self.repos_path):
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
58 raise Exception('Not a valid directory in %s' % self.repos_path)
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
59
28
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
60 def check_id_rsa(self):
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
61 if not os.path.isfile(self.id_rsa_path):
34
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
62 logging.error('Could not load id_rsa key file in %s',
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
63 self.id_rsa_path)
28
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
64 sys.exit()
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
65
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
66 def set_repos_path(self, paths):
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
67 repos_path = paths[0][1].split('/')
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
68 if repos_path[-1] in ['*', '**']:
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
69 repos_path = repos_path[:-1]
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
70 if repos_path[0] != '/':
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
71 repos_path[0] = '/'
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
72 self.repos_path = os.path.join(*repos_path)
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
73
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
74 def backup_repos(self):
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
75 today = datetime.datetime.now().weekday() + 1
28
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
76 self.backup_file_name = "mercurial_repos.%s.tar.gz" % today
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
77 bckp_file = os.path.join(self.backup_file_path, self.backup_file_name)
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
78 tar = tarfile.open(bckp_file, "w:gz")
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
79
34
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
80 for dir_name in os.listdir(self.repos_path):
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
81 logging.info('backing up %s', dir_name)
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
82 tar.add(os.path.join(self.repos_path, dir_name), dir_name)
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
83 tar.close()
26
b3307ca60bfb Litle info messages update
Marcin Kuzminski
parents: 25
diff changeset
84 logging.info('finished backup of mercurial repositories')
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
85
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
86
28
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
87
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
88 def transfer_files(self):
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
89 params = {
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
90 'id_rsa_key': self.id_rsa_path,
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
91 'backup_file_path':self.backup_file_path,
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
92 'backup_file_name':self.backup_file_name,
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
93 }
39
787e7d307b69 fix bug in arguments for command
marcink
parents: 38
diff changeset
94 cmd = ['scp', '-l', '40000', '-i', '%(id_rsa_key)s' % params,
28
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
95 '%(backup_file_path)s/%(backup_file_name)s' % params,
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
96 'root@192.168.2.102:/backups/mercurial' % params]
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
97
35
c92f827280a3 Changed to blocking operation
Marcin Kuzminski <marcin@python-blog.com>
parents: 34
diff changeset
98 subprocess.call(cmd)
28
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
99 logging.info('Transfered file %s to %s', self.backup_file_name, cmd[4])
33
22b2234e51cd Added removing of files from data
Marcin Kuzminski <marcin@python-blog.com>
parents: 29
diff changeset
100
22b2234e51cd Added removing of files from data
Marcin Kuzminski <marcin@python-blog.com>
parents: 29
diff changeset
101
22b2234e51cd Added removing of files from data
Marcin Kuzminski <marcin@python-blog.com>
parents: 29
diff changeset
102 def rm_file(self):
36
846e2d8672c4 Added message when file removing
Marcin Kuzminski <marcin@python-blog.com>
parents: 35
diff changeset
103 logging.info('Removing file %s', self.backup_file_name)
34
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
104 os.remove(os.path.join(self.backup_file_path, self.backup_file_name))
33
22b2234e51cd Added removing of files from data
Marcin Kuzminski <marcin@python-blog.com>
parents: 29
diff changeset
105
28
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
106
fd0c6fa2df82 backup script update
Marcin Kuzminski
parents: 27
diff changeset
107
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
108 if __name__ == "__main__":
241
48727add84c9 Made repos path config configurable from pylons app configs. update Readme
Marcin Kuzminski <marcin@python-works.com>
parents: 39
diff changeset
109 B_MANAGER = BackupManager('/home/pylons/id_rsa', 'repositories.config')
34
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
110 B_MANAGER.backup_repos()
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
111 B_MANAGER.transfer_files()
b4b25ece7797 Code refactoring, and changed proper way of removing files
Marcin Kuzminski <marcin@python-blog.com>
parents: 33
diff changeset
112 B_MANAGER.rm_file()
25
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
113
e89967ca7f68 backup script for repositories
Marcin Kuzminski
parents:
diff changeset
114