Mercurial > kallithea

annotate rhodecode/lib/celerylib/__init__.py @ 1036:405b80e4ccd5 beta

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Major refactoring, removed when possible calls to app globals. Refactored models to fetch the paths needed for scans from database directly small fixes in base
author Marcin Kuzminski <marcin@python-works.com>
date Sat, 12 Feb 2011 21:21:23 +0100
parents 02246101c846
children c2440badab84
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
1 # -*- coding: utf-8 -*-
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
2 """
903
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
3 rhodecode.lib.celerylib.__init__
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
5
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
6 celery libs for RhodeCode
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
7
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
8 :created_on: Nov 27, 2010
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
9 :author: marcink
902
07a6e8c65526 fixed copyright year to 2011
Marcin Kuzminski <marcin@python-works.com>
parents: 783
diff changeset
10 :copyright: (C) 2009-2011 Marcin Kuzminski <marcin@python-works.com>
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
11 :license: GPLv3, see COPYING for more details.
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
12 """
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
13 # This program is free software; you can redistribute it and/or
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
14 # modify it under the terms of the GNU General Public License
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
15 # as published by the Free Software Foundation; version 2
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
16 # of the License or (at your opinion) any later version of the license.
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
17 #
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
18 # This program is distributed in the hope that it will be useful,
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
19 # but WITHOUT ANY WARRANTY; without even the implied warranty of
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
20 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
21 # GNU General Public License for more details.
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
22 #
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
23 # You should have received a copy of the GNU General Public License
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
24 # along with this program; if not, write to the Free Software
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
25 # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
26 # MA 02110-1301, USA.
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
27
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
28 import os
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
29 import sys
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
30 import socket
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
31 import traceback
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
32 import logging
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
33
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
34 from hashlib import md5
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
35 from decorator import decorator
467
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
36 from vcs.utils.lazy import LazyProperty
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
37
914
110a00c181de Added force https option into config files
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
38 from rhodecode.lib import str2bool
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
39 from rhodecode.lib.pidlock import DaemonLock, LockHeld
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
40
1003
9037456bb17f Another better solution for establishing connection with messaging broker in celery.
Marcin Kuzminski <marcin@python-works.com>
parents: 914
diff changeset
41 from celery.messaging import establish_connection
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
42 from pylons import config
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
43
467
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
44 log = logging.getLogger(__name__)
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
45
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
46 try:
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
47 CELERY_ON = str2bool(config['app_conf'].get('use_celery'))
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
48 except KeyError:
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
49 CELERY_ON = False
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
50
467
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
51 class ResultWrapper(object):
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
52 def __init__(self, task):
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
53 self.task = task
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
54
467
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
55 @LazyProperty
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
56 def result(self):
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
57 return self.task
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
58
487
b12ea84fb906 Some fixes to summary, and total rewrite of summary graphs implemented more interactive graph.
Marcin Kuzminski <marcin@python-works.com>
parents: 474
diff changeset
59 def run_task(task, *args, **kwargs):
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
60 if CELERY_ON:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
61 try:
1004
7fd45bf17d07 fixed celery issues, default loader was not set as PylonsLoader
Marcin Kuzminski <marcin@python-works.com>
parents: 1003
diff changeset
62 t = task.apply_async(args=args, kwargs=kwargs)
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
63 log.info('running task %s:%s', t.task_id, task)
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
64 return t
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
65 except socket.error, e:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
66 if e.errno == 111:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
67 log.debug('Unable to connect to celeryd. Sync execution')
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
68 else:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
69 log.error(traceback.format_exc())
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
70 except KeyError, e:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
71 log.debug('Unable to connect to celeryd. Sync execution')
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
72 except Exception, e:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
73 log.error(traceback.format_exc())
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
74
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
75 log.debug('executing task %s in sync mode', task)
558
14559eb34003 more error catching on celery run_task
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
76 return ResultWrapper(task(*args, **kwargs))
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
77
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
78
502
ac32a026c306 simplified task locking, and fixed some bugs for keyworded arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 497
diff changeset
79 def locked_task(func):
ac32a026c306 simplified task locking, and fixed some bugs for keyworded arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 497
diff changeset
80 def __wrapper(func, *fargs, **fkwargs):
ac32a026c306 simplified task locking, and fixed some bugs for keyworded arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 497
diff changeset
81 params = list(fargs)
ac32a026c306 simplified task locking, and fixed some bugs for keyworded arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 497
diff changeset
82 params.extend(['%s-%s' % ar for ar in fkwargs.items()])
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
83
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
84 lockkey = 'task_%s' % \
502
ac32a026c306 simplified task locking, and fixed some bugs for keyworded arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 497
diff changeset
85 md5(str(func.__name__) + '-' + \
ac32a026c306 simplified task locking, and fixed some bugs for keyworded arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 497
diff changeset
86 '-'.join(map(str, params))).hexdigest()
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
87 log.info('running task with lockkey %s', lockkey)
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
88 try:
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
89 l = DaemonLock(lockkey)
510
9bedaa073c23 fixed lockdecrator to return executed function data
Marcin Kuzminski <marcin@python-works.com>
parents: 506
diff changeset
90 ret = func(*fargs, **fkwargs)
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
91 l.release()
510
9bedaa073c23 fixed lockdecrator to return executed function data
Marcin Kuzminski <marcin@python-works.com>
parents: 506
diff changeset
92 return ret
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
93 except LockHeld:
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
94 log.info('LockHeld')
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
95 return 'Task with key %s already running' % lockkey
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
96
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
97 return decorator(__wrapper, func)
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
98
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
99
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
100
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
101
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
102
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
103
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
104
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
105