Mercurial > kallithea

annotate rhodecode/lib/celerylib/__init__.py @ 1976:a76e9bacbedc beta

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
garden - unified logging formatting to use only %
author Marcin Kuzminski <marcin@python-works.com>
date Thu, 02 Feb 2012 00:31:00 +0200
parents a69573cfcb00
children 324ac367a4da
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
1 # -*- coding: utf-8 -*-
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
2 """
903
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
3 rhodecode.lib.celerylib.__init__
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
5
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
6 celery libs for RhodeCode
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 1082
diff changeset
7
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
8 :created_on: Nov 27, 2010
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
9 :author: marcink
1824
89efedac4e6c 2012 copyrights
Marcin Kuzminski <marcin@python-works.com>
parents: 1723
diff changeset
10 :copyright: (C) 2010-2012 Marcin Kuzminski <marcin@python-works.com>
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
11 :license: GPLv3, see COPYING for more details.
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
12 """
1206
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
13 # This program is free software: you can redistribute it and/or modify
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
14 # it under the terms of the GNU General Public License as published by
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
15 # the Free Software Foundation, either version 3 of the License, or
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
16 # (at your option) any later version.
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 1082
diff changeset
17 #
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
18 # This program is distributed in the hope that it will be useful,
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
19 # but WITHOUT ANY WARRANTY; without even the implied warranty of
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
20 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
21 # GNU General Public License for more details.
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 1082
diff changeset
22 #
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
23 # You should have received a copy of the GNU General Public License
1206
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
24 # along with this program. If not, see <http://www.gnu.org/licenses/>.
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
25
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
26 import os
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
27 import sys
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
28 import socket
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
29 import traceback
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
30 import logging
1354
ed309b1fbaa4 fixes issue #197 Relative paths for pidlocks
Marcin Kuzminski <marcin@python-works.com>
parents: 1264
diff changeset
31 from os.path import dirname as dn, join as jn
1929
cd8a7e3698bc fixes #340 session cleanup for celery tasks
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
32 from pylons import config
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
33
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
34 from hashlib import md5
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
35 from decorator import decorator
1082
c2440badab84 fixes for vcs 0.2.0and new slicing methods
Marcin Kuzminski <marcin@python-works.com>
parents: 1006
diff changeset
36
467
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
37 from vcs.utils.lazy import LazyProperty
1723
64e91067b996 - refactoring to overcome poor usage of global pylons config
Marcin Kuzminski <marcin@python-works.com>
parents: 1611
diff changeset
38 from rhodecode import CELERY_ON
1611
2755c11c90d8 Fixed problems with unicode cache keys in celery
Marcin Kuzminski <marcin@python-works.com>
parents: 1540
diff changeset
39 from rhodecode.lib import str2bool, safe_str
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
40 from rhodecode.lib.pidlock import DaemonLock, LockHeld
1929
cd8a7e3698bc fixes #340 session cleanup for celery tasks
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
41 from rhodecode.model import init_model
cd8a7e3698bc fixes #340 session cleanup for celery tasks
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
42 from rhodecode.model import meta
cd8a7e3698bc fixes #340 session cleanup for celery tasks
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
43 from rhodecode.model.db import Statistics, Repository, User
cd8a7e3698bc fixes #340 session cleanup for celery tasks
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
44
cd8a7e3698bc fixes #340 session cleanup for celery tasks
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
45 from sqlalchemy import engine_from_config
783
71113f64b2d8 fidex corrent variables passed to dnconfig,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
46
1003
9037456bb17f Another better solution for establishing connection with messaging broker in celery.
Marcin Kuzminski <marcin@python-works.com>
parents: 914
diff changeset
47 from celery.messaging import establish_connection
1082
c2440badab84 fixes for vcs 0.2.0and new slicing methods
Marcin Kuzminski <marcin@python-works.com>
parents: 1006
diff changeset
48
467
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
49 log = logging.getLogger(__name__)
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
50
1723
64e91067b996 - refactoring to overcome poor usage of global pylons config
Marcin Kuzminski <marcin@python-works.com>
parents: 1611
diff changeset
51
467
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
52 class ResultWrapper(object):
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
53 def __init__(self, task):
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
54 self.task = task
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
55
467
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
56 @LazyProperty
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
57 def result(self):
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
58 return self.task
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
59
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
60
487
b12ea84fb906 Some fixes to summary, and total rewrite of summary graphs implemented more interactive graph.
Marcin Kuzminski <marcin@python-works.com>
parents: 474
diff changeset
61 def run_task(task, *args, **kwargs):
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
62 if CELERY_ON:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
63 try:
1004
7fd45bf17d07 fixed celery issues, default loader was not set as PylonsLoader
Marcin Kuzminski <marcin@python-works.com>
parents: 1003
diff changeset
64 t = task.apply_async(args=args, kwargs=kwargs)
1976
Marcin Kuzminski <marcin@python-works.com>
parents: 1930
diff changeset
65 log.info('running task %s:%s' % (t.task_id, task))
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
66 return t
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
67
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
68 except socket.error, e:
1414
4f2c4fcc770b fixed Python2.5 socket error
Marcin Kuzminski <marcin@python-works.com>
parents: 1354
diff changeset
69 if isinstance(e, IOError) and e.errno == 111:
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
70 log.debug('Unable to connect to celeryd. Sync execution')
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
71 else:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
72 log.error(traceback.format_exc())
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
73 except KeyError, e:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
74 log.debug('Unable to connect to celeryd. Sync execution')
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
75 except Exception, e:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
76 log.error(traceback.format_exc())
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
77
1976
Marcin Kuzminski <marcin@python-works.com>
parents: 1930
diff changeset
78 log.debug('executing task %s in sync mode' % task)
558
14559eb34003 more error catching on celery run_task
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
79 return ResultWrapper(task(*args, **kwargs))
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
80
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
81
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
82 def __get_lockkey(func, *fargs, **fkwargs):
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
83 params = list(fargs)
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
84 params.extend(['%s-%s' % ar for ar in fkwargs.items()])
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
85
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
86 func_name = str(func.__name__) if hasattr(func, '__name__') else str(func)
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
87
1354
ed309b1fbaa4 fixes issue #197 Relative paths for pidlocks
Marcin Kuzminski <marcin@python-works.com>
parents: 1264
diff changeset
88 lockkey = 'task_%s.lock' % \
1611
2755c11c90d8 Fixed problems with unicode cache keys in celery
Marcin Kuzminski <marcin@python-works.com>
parents: 1540
diff changeset
89 md5(func_name + '-' + '-'.join(map(safe_str, params))).hexdigest()
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
90 return lockkey
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
91
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
92
502
ac32a026c306 simplified task locking, and fixed some bugs for keyworded arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 497
diff changeset
93 def locked_task(func):
ac32a026c306 simplified task locking, and fixed some bugs for keyworded arguments
Marcin Kuzminski <marcin@python-works.com>
parents: 497
diff changeset
94 def __wrapper(func, *fargs, **fkwargs):
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
95 lockkey = __get_lockkey(func, *fargs, **fkwargs)
1540
191f3f08236d fixes #258 RhodeCode 1.2 assumes egg folder is writable
Marcin Kuzminski <marcin@python-works.com>
parents: 1414
diff changeset
96 lockkey_path = config['here']
1354
ed309b1fbaa4 fixes issue #197 Relative paths for pidlocks
Marcin Kuzminski <marcin@python-works.com>
parents: 1264
diff changeset
97
1976
Marcin Kuzminski <marcin@python-works.com>
parents: 1930
diff changeset
98 log.info('running task with lockkey %s' % lockkey)
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
99 try:
1540
191f3f08236d fixes #258 RhodeCode 1.2 assumes egg folder is writable
Marcin Kuzminski <marcin@python-works.com>
parents: 1414
diff changeset
100 l = DaemonLock(file_=jn(lockkey_path, lockkey))
510
9bedaa073c23 fixed lockdecrator to return executed function data
Marcin Kuzminski <marcin@python-works.com>
parents: 506
diff changeset
101 ret = func(*fargs, **fkwargs)
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
102 l.release()
510
9bedaa073c23 fixed lockdecrator to return executed function data
Marcin Kuzminski <marcin@python-works.com>
parents: 506
diff changeset
103 return ret
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
104 except LockHeld:
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
105 log.info('LockHeld')
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
106 return 'Task with key %s already running' % lockkey
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
107
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 558
diff changeset
108 return decorator(__wrapper, func)
1929
cd8a7e3698bc fixes #340 session cleanup for celery tasks
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
109
cd8a7e3698bc fixes #340 session cleanup for celery tasks
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
110
cd8a7e3698bc fixes #340 session cleanup for celery tasks
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
111 def get_session():
cd8a7e3698bc fixes #340 session cleanup for celery tasks
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
112 if CELERY_ON:
cd8a7e3698bc fixes #340 session cleanup for celery tasks
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
113 engine = engine_from_config(config, 'sqlalchemy.db1.')
cd8a7e3698bc fixes #340 session cleanup for celery tasks
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
114 init_model(engine)
cd8a7e3698bc fixes #340 session cleanup for celery tasks
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
115 sa = meta.Session
cd8a7e3698bc fixes #340 session cleanup for celery tasks
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
116 return sa
cd8a7e3698bc fixes #340 session cleanup for celery tasks
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
117
cd8a7e3698bc fixes #340 session cleanup for celery tasks
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
118
cd8a7e3698bc fixes #340 session cleanup for celery tasks
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
119 def dbsession(func):
cd8a7e3698bc fixes #340 session cleanup for celery tasks
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
120 def __wrapper(func, *fargs, **fkwargs):
cd8a7e3698bc fixes #340 session cleanup for celery tasks
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
121 try:
cd8a7e3698bc fixes #340 session cleanup for celery tasks
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
122 ret = func(*fargs, **fkwargs)
cd8a7e3698bc fixes #340 session cleanup for celery tasks
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
123 return ret
cd8a7e3698bc fixes #340 session cleanup for celery tasks
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
124 finally:
1930
a69573cfcb00 cleanup task session only if celery is on
Marcin Kuzminski <marcin@python-works.com>
parents: 1929
diff changeset
125 if CELERY_ON:
a69573cfcb00 cleanup task session only if celery is on
Marcin Kuzminski <marcin@python-works.com>
parents: 1929
diff changeset
126 meta.Session.remove()
1929
cd8a7e3698bc fixes #340 session cleanup for celery tasks
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
127
cd8a7e3698bc fixes #340 session cleanup for celery tasks
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
128 return decorator(__wrapper, func)