Mercurial > kallithea

annotate rhodecode/lib/celerypylons/commands.py @ 3774:60335b702a00 beta

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
invalidation: don't create CacheInvalidation records on startup Creating the records early gave an advantage before lightweight was introduced. With lightweight it is no longer necessary. The records will be created on demand anyway and there is no reason to create and maintain them before they are used.
author Mads Kiilerich <madski@unity3d.com>
date Wed, 03 Apr 2013 15:56:12 +0200
parents 8ecfed1d8f8b
children 7aa0ff5b4e09
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1728
07e56179633e - fixes celery sqlalchemy session issues for async forking
Marcin Kuzminski <marcin@python-works.com>
parents: 1726
diff changeset
1 import rhodecode
2109
8ecfed1d8f8b utils/conf
Marcin Kuzminski <marcin@python-works.com>
parents: 1728
diff changeset
2 from rhodecode.lib.utils import BasePasterCommand, Command, load_rcextensions
1002
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
3 from celery.app import app_or_default
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
4 from celery.bin import camqadm, celerybeat, celeryd, celeryev
1726
fe8c2e881403 moved pylons config out of global scope for celerypylons
Marcin Kuzminski <marcin@python-works.com>
parents: 1672
diff changeset
5
2109
8ecfed1d8f8b utils/conf
Marcin Kuzminski <marcin@python-works.com>
parents: 1728
diff changeset
6 from rhodecode.lib.utils2 import str2bool
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
7
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
8 __all__ = ['CeleryDaemonCommand', 'CeleryBeatCommand',
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
9 'CAMQPAdminCommand', 'CeleryEventCommand']
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
10
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
11
1002
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
12 class CeleryCommand(BasePasterCommand):
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
13 """Abstract class implements run methods needed for celery
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
14
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
15 Starts the celery worker that uses a paste.deploy configuration
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
16 file.
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
17 """
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
18
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
19 def update_parser(self):
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
20 """
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
21 Abstract method. Allows for the class's parser to be updated
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
22 before the superclass's `run` method is called. Necessary to
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
23 allow options/arguments to be passed through to the underlying
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
24 celery command.
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
25 """
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
26
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
27 cmd = self.celery_command(app_or_default())
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
28 for x in cmd.get_options():
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
29 self.parser.add_option(x)
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
30
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
31 def command(self):
1726
fe8c2e881403 moved pylons config out of global scope for celerypylons
Marcin Kuzminski <marcin@python-works.com>
parents: 1672
diff changeset
32 from pylons import config
1672
13d6da57b0cf Added explicit error message about running celeryd without enable_celery flag set to true.
Marcin Kuzminski <marcin@python-works.com>
parents: 1002
diff changeset
33 try:
13d6da57b0cf Added explicit error message about running celeryd without enable_celery flag set to true.
Marcin Kuzminski <marcin@python-works.com>
parents: 1002
diff changeset
34 CELERY_ON = str2bool(config['app_conf'].get('use_celery'))
13d6da57b0cf Added explicit error message about running celeryd without enable_celery flag set to true.
Marcin Kuzminski <marcin@python-works.com>
parents: 1002
diff changeset
35 except KeyError:
13d6da57b0cf Added explicit error message about running celeryd without enable_celery flag set to true.
Marcin Kuzminski <marcin@python-works.com>
parents: 1002
diff changeset
36 CELERY_ON = False
13d6da57b0cf Added explicit error message about running celeryd without enable_celery flag set to true.
Marcin Kuzminski <marcin@python-works.com>
parents: 1002
diff changeset
37
13d6da57b0cf Added explicit error message about running celeryd without enable_celery flag set to true.
Marcin Kuzminski <marcin@python-works.com>
parents: 1002
diff changeset
38 if CELERY_ON == False:
13d6da57b0cf Added explicit error message about running celeryd without enable_celery flag set to true.
Marcin Kuzminski <marcin@python-works.com>
parents: 1002
diff changeset
39 raise Exception('Please enable celery_on in .ini config '
13d6da57b0cf Added explicit error message about running celeryd without enable_celery flag set to true.
Marcin Kuzminski <marcin@python-works.com>
parents: 1002
diff changeset
40 'file before running celeryd')
1728
07e56179633e - fixes celery sqlalchemy session issues for async forking
Marcin Kuzminski <marcin@python-works.com>
parents: 1726
diff changeset
41 rhodecode.CELERY_ON = CELERY_ON
2109
8ecfed1d8f8b utils/conf
Marcin Kuzminski <marcin@python-works.com>
parents: 1728
diff changeset
42 load_rcextensions(config['here'])
1002
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
43 cmd = self.celery_command(app_or_default())
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
44 return cmd.run(**vars(self.options))
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
45
2109
8ecfed1d8f8b utils/conf
Marcin Kuzminski <marcin@python-works.com>
parents: 1728
diff changeset
46
1002
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
47 class CeleryDaemonCommand(CeleryCommand):
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
48 """Start the celery worker
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
49
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
50 Starts the celery worker that uses a paste.deploy configuration
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
51 file.
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
52 """
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
53 usage = 'CONFIG_FILE [celeryd options...]'
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
54 summary = __doc__.splitlines()[0]
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
55 description = "".join(__doc__.splitlines()[2:])
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
56
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
57 parser = Command.standard_parser(quiet=True)
1002
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
58 celery_command = celeryd.WorkerCommand
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
59
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
60
1002
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
61 class CeleryBeatCommand(CeleryCommand):
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
62 """Start the celery beat server
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
63
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
64 Starts the celery beat server using a paste.deploy configuration
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
65 file.
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
66 """
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
67 usage = 'CONFIG_FILE [celerybeat options...]'
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
68 summary = __doc__.splitlines()[0]
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
69 description = "".join(__doc__.splitlines()[2:])
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
70
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
71 parser = Command.standard_parser(quiet=True)
1002
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
72 celery_command = celerybeat.BeatCommand
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
73
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
74
1002
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
75 class CAMQPAdminCommand(CeleryCommand):
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
76 """CAMQP Admin
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
77
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
78 CAMQP celery admin tool.
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
79 """
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
80 usage = 'CONFIG_FILE [camqadm options...]'
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
81 summary = __doc__.splitlines()[0]
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
82 description = "".join(__doc__.splitlines()[2:])
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
83
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
84 parser = Command.standard_parser(quiet=True)
1002
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
85 celery_command = camqadm.AMQPAdminCommand
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
86
2109
8ecfed1d8f8b utils/conf
Marcin Kuzminski <marcin@python-works.com>
parents: 1728
diff changeset
87
1002
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
88 class CeleryEventCommand(CeleryCommand):
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
89 """Celery event command.
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
90
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
91 Capture celery events.
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
92 """
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
93 usage = 'CONFIG_FILE [celeryev options...]'
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
94 summary = __doc__.splitlines()[0]
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
95 description = "".join(__doc__.splitlines()[2:])
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
96
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
97 parser = Command.standard_parser(quiet=True)
1002
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
98 celery_command = celeryev.EvCommand