Mercurial > kallithea

annotate rhodecode/lib/celerylib/__init__.py @ 1723:64e91067b996 beta

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
- refactoring to overcome poor usage of global pylons config - db transaction fixes - fixed tests - garden
author Marcin Kuzminski <marcin@python-works.com>
date Sat, 26 Nov 2011 02:16:21 +0200
parents 2755c11c90d8
children 89efedac4e6c
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
1 # -*- coding: utf-8 -*-
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
2 """
903
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
3 rhodecode.lib.celerylib.__init__
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
5
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
6 celery libs for RhodeCode
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 1082
diff changeset
7
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
8 :created_on: Nov 27, 2010
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
9 :author: marcink
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 1082
diff changeset
10 :copyright: (C) 2009-2011 Marcin Kuzminski <marcin@python-works.com>
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
11 :license: GPLv3, see COPYING for more details.
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
12 """
1206
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
13 # This program is free software: you can redistribute it and/or modify
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
14 # it under the terms of the GNU General Public License as published by
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
15 # the Free Software Foundation, either version 3 of the License, or
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
16 # (at your option) any later version.
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 1082
diff changeset
17 #
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
18 # This program is distributed in the hope that it will be useful,
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
19 # but WITHOUT ANY WARRANTY; without even the implied warranty of
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
20 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
21 # GNU General Public License for more details.
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 1082
diff changeset
22 #
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
23 # You should have received a copy of the GNU General Public License
1206
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
24 # along with this program. If not, see <http://www.gnu.org/licenses/>.
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
25
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
26 import os
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
27 import sys
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
28 import socket
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
29 import traceback
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
30 import logging
1354
ed309b1fbaa4 fixes issue #197 Relative paths for pidlocks
Marcin Kuzminski <marcin@python-works.com>
parents: 1264
diff changeset
31 from os.path import dirname as dn, join as jn
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
32
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
33 from hashlib import md5
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
34 from decorator import decorator
1082
c2440badab84 fixes for vcs 0.2.0and new slicing methods
Marcin Kuzminski <marcin@python-works.com>
parents: 1006
diff changeset
35
467
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
36 from vcs.utils.lazy import LazyProperty
1723
64e91067b996 - refactoring to overcome poor usage of global pylons config
Marcin Kuzminski <marcin@python-works.com>
parents: 1611
diff changeset
37 from rhodecode import CELERY_ON
1611
2755c11c90d8 Fixed problems with unicode cache keys in celery
Marcin Kuzminski <marcin@python-works.com>
parents: 1540
diff changeset
38 from rhodecode.lib import str2bool, safe_str
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
39 from rhodecode.lib.pidlock import DaemonLock, LockHeld
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
40
1003
9037456bb17f Another better solution for establishing connection with messaging broker in celery.
Marcin Kuzminski <marcin@python-works.com>
parents: 914
diff changeset
41 from celery.messaging import establish_connection
1082
c2440badab84 fixes for vcs 0.2.0and new slicing methods
Marcin Kuzminski <marcin@python-works.com>
parents: 1006
diff changeset
42
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
43
467
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
44 log = logging.getLogger(__name__)
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
45
1723
64e91067b996 - refactoring to overcome poor usage of global pylons config
Marcin Kuzminski <marcin@python-works.com>
parents: 1611
diff changeset
46
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
47
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
48
467
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
49 class ResultWrapper(object):
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
50 def __init__(self, task):
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
51 self.task = task
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
52
467
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
53 @LazyProperty
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
54 def result(self):
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
55 return self.task
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
56
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
57
487
b12ea84fb906 Some fixes to summary, and total rewrite of summary graphs implemented more interactive graph.
Marcin Kuzminski <marcin@python-works.com>
parents: 474
diff changeset
58 def run_task(task, *args, **kwargs):
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
59 if CELERY_ON:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
60 try:
1004
7fd45bf17d07 fixed celery issues, default loader was not set as PylonsLoader
Marcin Kuzminski <marcin@python-works.com>
parents: 1003
diff changeset
61 t = task.apply_async(args=args, kwargs=kwargs)
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
62 log.info('running task %s:%s', t.task_id, task)
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
63 return t
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
64
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
65 except socket.error, e:
1414
4f2c4fcc770b fixed Python2.5 socket error
Marcin Kuzminski <marcin@python-works.com>
parents: 1354
diff changeset
66 if isinstance(e, IOError) and e.errno == 111:
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
67 log.debug('Unable to connect to celeryd. Sync execution')
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
68 else:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
69 log.error(traceback.format_exc())
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
70 except KeyError, e:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
71 log.debug('Unable to connect to celeryd. Sync execution')
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
72 except Exception, e:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
73 log.error(traceback.format_exc())
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
74
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
75 log.debug('executing task %s in sync mode', task)
558
14559eb34003 more error catching on celery run_task
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
76 return ResultWrapper(task(*args, **kwargs))
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
77
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
78
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
79 def __get_lockkey(func, *fargs, **fkwargs):
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
80 params = list(fargs)
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
81 params.extend(['%s-%s' % ar for ar in fkwargs.items()])
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
82
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
83 func_name = str(func.__name__) if hasattr(func, '__name__') else str(func)
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
84
1354
ed309b1fbaa4 fixes issue #197 Relative paths for pidlocks
Marcin Kuzminski <marcin@python-works.com>
parents: 1264
diff changeset
85 lockkey = 'task_%s.lock' % \
1611
2755c11c90d8 Fixed problems with unicode cache keys in celery
Marcin Kuzminski <marcin@python-works.com>
parents: 1540
diff changeset
86 md5(func_name + '-' + '-'.join(map(safe_str, params))).hexdigest()
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
87 return lockkey
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
88
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
89
502
ac32a026c306 simplified task locking, and fixed some bugs for keyworded arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 497
diff changeset
90 def locked_task(func):
ac32a026c306 simplified task locking, and fixed some bugs for keyworded arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 497
diff changeset
91 def __wrapper(func, *fargs, **fkwargs):
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
92 lockkey = __get_lockkey(func, *fargs, **fkwargs)
1540
191f3f08236d fixes #258 RhodeCode 1.2 assumes egg folder is writable
Marcin Kuzminski <marcin@python-works.com>
parents: 1414
diff changeset
93 lockkey_path = config['here']
1354
ed309b1fbaa4 fixes issue #197 Relative paths for pidlocks
Marcin Kuzminski <marcin@python-works.com>
parents: 1264
diff changeset
94
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
95 log.info('running task with lockkey %s', lockkey)
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
96 try:
1540
191f3f08236d fixes #258 RhodeCode 1.2 assumes egg folder is writable
Marcin Kuzminski <marcin@python-works.com>
parents: 1414
diff changeset
97 l = DaemonLock(file_=jn(lockkey_path, lockkey))
510
9bedaa073c23 fixed lockdecrator to return executed function data
Marcin Kuzminski <marcin@python-works.com>
parents: 506
diff changeset
98 ret = func(*fargs, **fkwargs)
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
99 l.release()
510
9bedaa073c23 fixed lockdecrator to return executed function data
Marcin Kuzminski <marcin@python-works.com>
parents: 506
diff changeset
100 return ret
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
101 except LockHeld:
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
102 log.info('LockHeld')
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
103 return 'Task with key %s already running' % lockkey
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
104
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
105 return decorator(__wrapper, func)