Mercurial > kallithea

annotate rhodecode/lib/celerylib/__init__.py @ 1003:9037456bb17f beta

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Another better solution for establishing connection with messaging broker in celery. This one doesn't require celeryconfig.py
author Marcin Kuzminski <marcin@python-works.com>
date Tue, 08 Feb 2011 03:45:22 +0100
parents 110a00c181de
children 7fd45bf17d07
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
1 # -*- coding: utf-8 -*-
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
2 """
903
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
3 rhodecode.lib.celerylib.__init__
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
5
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
6 celery libs for RhodeCode
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
7
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
8 :created_on: Nov 27, 2010
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
9 :author: marcink
902
07a6e8c65526 fixed copyright year to 2011
Marcin Kuzminski <marcin@python-works.com>
parents: 783
diff changeset
10 :copyright: (C) 2009-2011 Marcin Kuzminski <marcin@python-works.com>
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
11 :license: GPLv3, see COPYING for more details.
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
12 """
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
13 # This program is free software; you can redistribute it and/or
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
14 # modify it under the terms of the GNU General Public License
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
15 # as published by the Free Software Foundation; version 2
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
16 # of the License or (at your opinion) any later version of the license.
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
17 #
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
18 # This program is distributed in the hope that it will be useful,
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
19 # but WITHOUT ANY WARRANTY; without even the implied warranty of
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
20 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
21 # GNU General Public License for more details.
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
22 #
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
23 # You should have received a copy of the GNU General Public License
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
24 # along with this program; if not, write to the Free Software
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
25 # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
26 # MA 02110-1301, USA.
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
27
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
28 import os
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
29 import sys
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
30 import socket
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
31 import traceback
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
32 import logging
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
33
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
34 from hashlib import md5
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
35 from decorator import decorator
467
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
36 from vcs.utils.lazy import LazyProperty
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
37
914
110a00c181de Added force https option into config files
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
38 from rhodecode.lib import str2bool
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
39 from rhodecode.lib.pidlock import DaemonLock, LockHeld
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
40
1003
9037456bb17f Another better solution for establishing connection with messaging broker in celery.
Marcin Kuzminski <marcin@python-works.com>
parents: 914
diff changeset
41 from celery.messaging import establish_connection
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
42 from pylons import config
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
43
467
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
44 log = logging.getLogger(__name__)
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
45
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
46 try:
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
47 CELERY_ON = str2bool(config['app_conf'].get('use_celery'))
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
48 except KeyError:
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
49 CELERY_ON = False
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
50
467
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
51 class ResultWrapper(object):
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
52 def __init__(self, task):
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
53 self.task = task
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
54
467
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
55 @LazyProperty
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
56 def result(self):
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
57 return self.task
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
58
487
b12ea84fb906 Some fixes to summary, and total rewrite of summary graphs implemented more interactive graph.
Marcin Kuzminski <marcin@python-works.com>
parents: 474
diff changeset
59 def run_task(task, *args, **kwargs):
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
60 if CELERY_ON:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
61 try:
1003
9037456bb17f Another better solution for establishing connection with messaging broker in celery.
Marcin Kuzminski <marcin@python-works.com>
parents: 914
diff changeset
62 kw = {
9037456bb17f Another better solution for establishing connection with messaging broker in celery.
Marcin Kuzminski <marcin@python-works.com>
parents: 914
diff changeset
63 'hostname':config['app_conf'].get('broker.host'),
9037456bb17f Another better solution for establishing connection with messaging broker in celery.
Marcin Kuzminski <marcin@python-works.com>
parents: 914
diff changeset
64 'userid':config['app_conf'].get('broker.user'),
9037456bb17f Another better solution for establishing connection with messaging broker in celery.
Marcin Kuzminski <marcin@python-works.com>
parents: 914
diff changeset
65 'password':config['app_conf'].get('broker.password'),
9037456bb17f Another better solution for establishing connection with messaging broker in celery.
Marcin Kuzminski <marcin@python-works.com>
parents: 914
diff changeset
66 'virtual_host':config['app_conf'].get('broker.vhost'),
9037456bb17f Another better solution for establishing connection with messaging broker in celery.
Marcin Kuzminski <marcin@python-works.com>
parents: 914
diff changeset
67 'port':config['app_conf'].get('broker.port'),
9037456bb17f Another better solution for establishing connection with messaging broker in celery.
Marcin Kuzminski <marcin@python-works.com>
parents: 914
diff changeset
68 }
9037456bb17f Another better solution for establishing connection with messaging broker in celery.
Marcin Kuzminski <marcin@python-works.com>
parents: 914
diff changeset
69 conn = establish_connection(**kw)
9037456bb17f Another better solution for establishing connection with messaging broker in celery.
Marcin Kuzminski <marcin@python-works.com>
parents: 914
diff changeset
70 publisher = task.get_publisher(connection=conn)
9037456bb17f Another better solution for establishing connection with messaging broker in celery.
Marcin Kuzminski <marcin@python-works.com>
parents: 914
diff changeset
71 t = task.apply_async(args=args, kwargs=kwargs, publisher=publisher)
9037456bb17f Another better solution for establishing connection with messaging broker in celery.
Marcin Kuzminski <marcin@python-works.com>
parents: 914
diff changeset
72
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
73 log.info('running task %s:%s', t.task_id, task)
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
74 return t
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
75 except socket.error, e:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
76 if e.errno == 111:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
77 log.debug('Unable to connect to celeryd. Sync execution')
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
78 else:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
79 log.error(traceback.format_exc())
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
80 except KeyError, e:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
81 log.debug('Unable to connect to celeryd. Sync execution')
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
82 except Exception, e:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
83 log.error(traceback.format_exc())
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
84
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
85 log.debug('executing task %s in sync mode', task)
558
14559eb34003 more error catching on celery run_task
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
86 return ResultWrapper(task(*args, **kwargs))
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
87
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
88
502
ac32a026c306 simplified task locking, and fixed some bugs for keyworded arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 497
diff changeset
89 def locked_task(func):
ac32a026c306 simplified task locking, and fixed some bugs for keyworded arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 497
diff changeset
90 def __wrapper(func, *fargs, **fkwargs):
ac32a026c306 simplified task locking, and fixed some bugs for keyworded arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 497
diff changeset
91 params = list(fargs)
ac32a026c306 simplified task locking, and fixed some bugs for keyworded arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 497
diff changeset
92 params.extend(['%s-%s' % ar for ar in fkwargs.items()])
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
93
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
94 lockkey = 'task_%s' % \
502
ac32a026c306 simplified task locking, and fixed some bugs for keyworded arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 497
diff changeset
95 md5(str(func.__name__) + '-' + \
ac32a026c306 simplified task locking, and fixed some bugs for keyworded arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 497
diff changeset
96 '-'.join(map(str, params))).hexdigest()
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
97 log.info('running task with lockkey %s', lockkey)
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
98 try:
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
99 l = DaemonLock(lockkey)
510
9bedaa073c23 fixed lockdecrator to return executed function data
Marcin Kuzminski <marcin@python-works.com>
parents: 506
diff changeset
100 ret = func(*fargs, **fkwargs)
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
101 l.release()
510
9bedaa073c23 fixed lockdecrator to return executed function data
Marcin Kuzminski <marcin@python-works.com>
parents: 506
diff changeset
102 return ret
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
103 except LockHeld:
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
104 log.info('LockHeld')
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
105 return 'Task with key %s already running' % lockkey
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
106
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
107 return decorator(__wrapper, func)
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
108
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
109
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
110
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
111
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
112
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
113
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
114
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
115