Mercurial > kallithea

annotate rhodecode/lib/celerylib/__init__.py @ 903:04c9bb9ca6d6 beta

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
code docs, updates
author Marcin Kuzminski <marcin@python-works.com>
date Fri, 31 Dec 2010 19:58:43 +0100
parents 07a6e8c65526
children 110a00c181de
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
1 # -*- coding: utf-8 -*-
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
2 """
903
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
3 rhodecode.lib.celerylib.__init__
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
5
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
6 celery libs for RhodeCode
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
7
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
8 :created_on: Nov 27, 2010
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
9 :author: marcink
902
07a6e8c65526 fixed copyright year to 2011
Marcin Kuzminski <marcin@python-works.com>
parents: 783
diff changeset
10 :copyright: (C) 2009-2011 Marcin Kuzminski <marcin@python-works.com>
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
11 :license: GPLv3, see COPYING for more details.
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
12 """
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
13 # This program is free software; you can redistribute it and/or
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
14 # modify it under the terms of the GNU General Public License
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
15 # as published by the Free Software Foundation; version 2
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
16 # of the License or (at your opinion) any later version of the license.
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
17 #
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
18 # This program is distributed in the hope that it will be useful,
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
19 # but WITHOUT ANY WARRANTY; without even the implied warranty of
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
20 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
21 # GNU General Public License for more details.
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
22 #
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
23 # You should have received a copy of the GNU General Public License
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
24 # along with this program; if not, write to the Free Software
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
25 # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
26 # MA 02110-1301, USA.
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
27
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
28 import os
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
29 import sys
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
30 import socket
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
31 import traceback
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
32 import logging
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
33
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
34 from hashlib import md5
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
35 from decorator import decorator
467
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
36 from vcs.utils.lazy import LazyProperty
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
37
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
38 from rhodecode.lib.pidlock import DaemonLock, LockHeld
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
39
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
40 from pylons import config
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
41
467
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
42 log = logging.getLogger(__name__)
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
43
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
44 def str2bool(v):
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
45 return v.lower() in ["yes", "true", "t", "1"] if v else None
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
46
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
47 try:
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
48 CELERY_ON = str2bool(config['app_conf'].get('use_celery'))
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
49 except KeyError:
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
50 CELERY_ON = False
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
51
467
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
52 class ResultWrapper(object):
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
53 def __init__(self, task):
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
54 self.task = task
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
55
467
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
56 @LazyProperty
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
57 def result(self):
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
58 return self.task
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
59
487
b12ea84fb906 Some fixes to summary, and total rewrite of summary graphs implemented more interactive graph.
Marcin Kuzminski <marcin@python-works.com>
parents: 474
diff changeset
60 def run_task(task, *args, **kwargs):
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
61 if CELERY_ON:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
62 try:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
63 t = task.delay(*args, **kwargs)
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
64 log.info('running task %s:%s', t.task_id, task)
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
65 return t
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
66 except socket.error, e:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
67 if e.errno == 111:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
68 log.debug('Unable to connect to celeryd. Sync execution')
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
69 else:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
70 log.error(traceback.format_exc())
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
71 except KeyError, e:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
72 log.debug('Unable to connect to celeryd. Sync execution')
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
73 except Exception, e:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
74 log.error(traceback.format_exc())
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
75
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
76 log.debug('executing task %s in sync mode', task)
558
14559eb34003 more error catching on celery run_task
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
77 return ResultWrapper(task(*args, **kwargs))
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
78
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
79
502
ac32a026c306 simplified task locking, and fixed some bugs for keyworded arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 497
diff changeset
80 def locked_task(func):
ac32a026c306 simplified task locking, and fixed some bugs for keyworded arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 497
diff changeset
81 def __wrapper(func, *fargs, **fkwargs):
ac32a026c306 simplified task locking, and fixed some bugs for keyworded arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 497
diff changeset
82 params = list(fargs)
ac32a026c306 simplified task locking, and fixed some bugs for keyworded arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 497
diff changeset
83 params.extend(['%s-%s' % ar for ar in fkwargs.items()])
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
84
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
85 lockkey = 'task_%s' % \
502
ac32a026c306 simplified task locking, and fixed some bugs for keyworded arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 497
diff changeset
86 md5(str(func.__name__) + '-' + \
ac32a026c306 simplified task locking, and fixed some bugs for keyworded arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 497
diff changeset
87 '-'.join(map(str, params))).hexdigest()
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
88 log.info('running task with lockkey %s', lockkey)
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
89 try:
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
90 l = DaemonLock(lockkey)
510
9bedaa073c23 fixed lockdecrator to return executed function data
Marcin Kuzminski <marcin@python-works.com>
parents: 506
diff changeset
91 ret = func(*fargs, **fkwargs)
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
92 l.release()
510
9bedaa073c23 fixed lockdecrator to return executed function data
Marcin Kuzminski <marcin@python-works.com>
parents: 506
diff changeset
93 return ret
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
94 except LockHeld:
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
95 log.info('LockHeld')
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
96 return 'Task with key %s already running' % lockkey
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
97
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
98 return decorator(__wrapper, func)
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
99
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
100
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
101
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
102
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
103
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
104
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
105
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
106