Mercurial > kallithea

annotate kallithea/lib/celery_app.py @ 8833:aafca212c8e2

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
celery: move send_email task to a better home in notification model Avoid bundling everything from many different layers in one big task library. This is more feasible now when we don't need kallithea.CELERY_APP set at import time.
author Mads Kiilerich <mads@kiilerich.com>
date Tue, 29 Dec 2020 22:23:01 +0100
parents 1d3b67443aac
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
4116
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 776
diff changeset
1 # -*- coding: utf-8 -*-
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 776
diff changeset
2
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
3 """
6172
7e7db11d4e4d celerypylons: wrap celery import so we always get the right environment variables set and check configuration
Mads Kiilerich <madski@unity3d.com>
parents: 4187
diff changeset
4 Kallithea wrapper of Celery
7e7db11d4e4d celerypylons: wrap celery import so we always get the right environment variables set and check configuration
Mads Kiilerich <madski@unity3d.com>
parents: 4187
diff changeset
5
6803
3fb0ce6de10d celery: replace loader.PylonsSettingsProxy with a simple configuration object
Mads Kiilerich <mads@kiilerich.com>
parents: 6802
diff changeset
6 The Celery configuration is in the Kallithea ini file but must be converted to an
3fb0ce6de10d celery: replace loader.PylonsSettingsProxy with a simple configuration object
Mads Kiilerich <mads@kiilerich.com>
parents: 6802
diff changeset
7 entirely different format before Celery can use it.
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
8
6803
3fb0ce6de10d celery: replace loader.PylonsSettingsProxy with a simple configuration object
Mads Kiilerich <mads@kiilerich.com>
parents: 6802
diff changeset
9 We read the configuration from tg.config at module import time. This module can
3fb0ce6de10d celery: replace loader.PylonsSettingsProxy with a simple configuration object
Mads Kiilerich <mads@kiilerich.com>
parents: 6802
diff changeset
10 thus not be imported in global scope but must be imported on demand in function
3fb0ce6de10d celery: replace loader.PylonsSettingsProxy with a simple configuration object
Mads Kiilerich <mads@kiilerich.com>
parents: 6802
diff changeset
11 scope after tg.config has been initialized.
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
12
6803
3fb0ce6de10d celery: replace loader.PylonsSettingsProxy with a simple configuration object
Mads Kiilerich <mads@kiilerich.com>
parents: 6802
diff changeset
13 To make sure that the config really has been initialized, we check one of the
3fb0ce6de10d celery: replace loader.PylonsSettingsProxy with a simple configuration object
Mads Kiilerich <mads@kiilerich.com>
parents: 6802
diff changeset
14 mandatory settings.
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
15 """
4116
ffd45b185016 Imported some of the GPLv3'd changes from RhodeCode v2.2.5.
Bradley M. Kuhn <bkuhn@sfconservancy.org>
parents: 776
diff changeset
16
8266
b2f7d1c4e001 celery: log an error for some deprecated config settings
Mads Kiilerich <mads@kiilerich.com>
parents: 8265
diff changeset
17 import logging
b2f7d1c4e001 celery: log an error for some deprecated config settings
Mads Kiilerich <mads@kiilerich.com>
parents: 8265
diff changeset
18
8239
9d6cc55384fe cleanup: mute some pyflakes "imported but unused" by adding asserts
Mads Kiilerich <mads@kiilerich.com>
parents: 8184
diff changeset
19
8262
66670aff96c5 celery: set default config values in code and remove them from the generated .ini
Mads Kiilerich <mads@kiilerich.com>
parents: 8244
diff changeset
20 class CeleryConfig(object):
8831
1f92cded1bd2 celery: move Whoosh indexing task to whoosh library where it belongs
Mads Kiilerich <mads@kiilerich.com>
parents: 8827
diff changeset
21 imports = [
1f92cded1bd2 celery: move Whoosh indexing task to whoosh library where it belongs
Mads Kiilerich <mads@kiilerich.com>
parents: 8827
diff changeset
22 'kallithea.lib.indexers.daemon',
1f92cded1bd2 celery: move Whoosh indexing task to whoosh library where it belongs
Mads Kiilerich <mads@kiilerich.com>
parents: 8827
diff changeset
23 'kallithea.model.async_tasks',
8833
aafca212c8e2 celery: move send_email task to a better home in notification model
Mads Kiilerich <mads@kiilerich.com>
parents: 8832
diff changeset
24 'kallithea.model.notification',
8832
1d3b67443aac celery: move repo create tasks to repo model - that's where it belongs
Mads Kiilerich <mads@kiilerich.com>
parents: 8831
diff changeset
25 'kallithea.model.repo',
8831
1f92cded1bd2 celery: move Whoosh indexing task to whoosh library where it belongs
Mads Kiilerich <mads@kiilerich.com>
parents: 8827
diff changeset
26 ]
8268
f8f50d3b6512 celery: upgrade to Celery 4
Mads Kiilerich <mads@kiilerich.com>
parents: 8267
diff changeset
27 task_always_eager = False
f8f50d3b6512 celery: upgrade to Celery 4
Mads Kiilerich <mads@kiilerich.com>
parents: 8267
diff changeset
28
8780
eed44652346d celery: drop pre-celery-4 compatibility
Mads Kiilerich <mads@kiilerich.com>
parents: 8779
diff changeset
29 list_config_names = {'imports', 'accept_content'}
8262
66670aff96c5 celery: set default config values in code and remove them from the generated .ini
Mads Kiilerich <mads@kiilerich.com>
parents: 8244
diff changeset
30
66670aff96c5 celery: set default config values in code and remove them from the generated .ini
Mads Kiilerich <mads@kiilerich.com>
parents: 8244
diff changeset
31
8266
b2f7d1c4e001 celery: log an error for some deprecated config settings
Mads Kiilerich <mads@kiilerich.com>
parents: 8265
diff changeset
32 desupported = set([
8780
eed44652346d celery: drop pre-celery-4 compatibility
Mads Kiilerich <mads@kiilerich.com>
parents: 8779
diff changeset
33 'broker.url',
eed44652346d celery: drop pre-celery-4 compatibility
Mads Kiilerich <mads@kiilerich.com>
parents: 8779
diff changeset
34 'celery.accept.content',
eed44652346d celery: drop pre-celery-4 compatibility
Mads Kiilerich <mads@kiilerich.com>
parents: 8779
diff changeset
35 'celery.always.eager',
eed44652346d celery: drop pre-celery-4 compatibility
Mads Kiilerich <mads@kiilerich.com>
parents: 8779
diff changeset
36 'celery.amqp.task.result.expires',
eed44652346d celery: drop pre-celery-4 compatibility
Mads Kiilerich <mads@kiilerich.com>
parents: 8779
diff changeset
37 'celeryd.concurrency',
eed44652346d celery: drop pre-celery-4 compatibility
Mads Kiilerich <mads@kiilerich.com>
parents: 8779
diff changeset
38 'celeryd.max.tasks.per.child',
eed44652346d celery: drop pre-celery-4 compatibility
Mads Kiilerich <mads@kiilerich.com>
parents: 8779
diff changeset
39 'celery.result.backend', # Note: the .ini template used this instead of 'celery.result_backend' in 0.6
8266
b2f7d1c4e001 celery: log an error for some deprecated config settings
Mads Kiilerich <mads@kiilerich.com>
parents: 8265
diff changeset
40 'celery.result.dburi',
b2f7d1c4e001 celery: log an error for some deprecated config settings
Mads Kiilerich <mads@kiilerich.com>
parents: 8265
diff changeset
41 'celery.result.serialier',
8780
eed44652346d celery: drop pre-celery-4 compatibility
Mads Kiilerich <mads@kiilerich.com>
parents: 8779
diff changeset
42 'celery.result.serializer',
8266
b2f7d1c4e001 celery: log an error for some deprecated config settings
Mads Kiilerich <mads@kiilerich.com>
parents: 8265
diff changeset
43 'celery.send.task.error.emails',
8820
0270c714fa17 celery: drop traces of support for task_always_eager
Mads Kiilerich <mads@kiilerich.com>
parents: 8798
diff changeset
44 'celery.task_always_eager', # still a valid configuration in celery, but not supported in Kallithea
8780
eed44652346d celery: drop pre-celery-4 compatibility
Mads Kiilerich <mads@kiilerich.com>
parents: 8779
diff changeset
45 'celery.task.serializer',
8266
b2f7d1c4e001 celery: log an error for some deprecated config settings
Mads Kiilerich <mads@kiilerich.com>
parents: 8265
diff changeset
46 ])
b2f7d1c4e001 celery: log an error for some deprecated config settings
Mads Kiilerich <mads@kiilerich.com>
parents: 8265
diff changeset
47
b2f7d1c4e001 celery: log an error for some deprecated config settings
Mads Kiilerich <mads@kiilerich.com>
parents: 8265
diff changeset
48
b2f7d1c4e001 celery: log an error for some deprecated config settings
Mads Kiilerich <mads@kiilerich.com>
parents: 8265
diff changeset
49 log = logging.getLogger(__name__)
b2f7d1c4e001 celery: log an error for some deprecated config settings
Mads Kiilerich <mads@kiilerich.com>
parents: 8265
diff changeset
50
b2f7d1c4e001 celery: log an error for some deprecated config settings
Mads Kiilerich <mads@kiilerich.com>
parents: 8265
diff changeset
51
8267
e432a6a7dd7c celery: move initialization of global kallithea.CELERY_EAGER to the place where we read celery configuration
Mads Kiilerich <mads@kiilerich.com>
parents: 8266
diff changeset
52 def make_celery_config(config):
6803
3fb0ce6de10d celery: replace loader.PylonsSettingsProxy with a simple configuration object
Mads Kiilerich <mads@kiilerich.com>
parents: 6802
diff changeset
53 """Return Celery config object populated from relevant settings in a config dict, such as tg.config"""
3fb0ce6de10d celery: replace loader.PylonsSettingsProxy with a simple configuration object
Mads Kiilerich <mads@kiilerich.com>
parents: 6802
diff changeset
54
3fb0ce6de10d celery: replace loader.PylonsSettingsProxy with a simple configuration object
Mads Kiilerich <mads@kiilerich.com>
parents: 6802
diff changeset
55 celery_config = CeleryConfig()
3fb0ce6de10d celery: replace loader.PylonsSettingsProxy with a simple configuration object
Mads Kiilerich <mads@kiilerich.com>
parents: 6802
diff changeset
56
3fb0ce6de10d celery: replace loader.PylonsSettingsProxy with a simple configuration object
Mads Kiilerich <mads@kiilerich.com>
parents: 6802
diff changeset
57 for config_key, config_value in sorted(config.items()):
8266
b2f7d1c4e001 celery: log an error for some deprecated config settings
Mads Kiilerich <mads@kiilerich.com>
parents: 8265
diff changeset
58 if config_key in desupported and config_value:
b2f7d1c4e001 celery: log an error for some deprecated config settings
Mads Kiilerich <mads@kiilerich.com>
parents: 8265
diff changeset
59 log.error('Celery configuration setting %r is no longer supported', config_key)
8268
f8f50d3b6512 celery: upgrade to Celery 4
Mads Kiilerich <mads@kiilerich.com>
parents: 8267
diff changeset
60 parts = config_key.split('.', 1)
8780
eed44652346d celery: drop pre-celery-4 compatibility
Mads Kiilerich <mads@kiilerich.com>
parents: 8779
diff changeset
61 if parts[0] == 'celery' and len(parts) == 2: # Celery 4 config key
8268
f8f50d3b6512 celery: upgrade to Celery 4
Mads Kiilerich <mads@kiilerich.com>
parents: 8267
diff changeset
62 celery_key = parts[1]
f8f50d3b6512 celery: upgrade to Celery 4
Mads Kiilerich <mads@kiilerich.com>
parents: 8267
diff changeset
63 else:
6803
3fb0ce6de10d celery: replace loader.PylonsSettingsProxy with a simple configuration object
Mads Kiilerich <mads@kiilerich.com>
parents: 6802
diff changeset
64 continue
8184
fb4b72c1c0f1 py3: automatic migration with 2to3 -f basestring
Mads Kiilerich <mads@kiilerich.com>
parents: 8162
diff changeset
65 if not isinstance(config_value, str):
6803
3fb0ce6de10d celery: replace loader.PylonsSettingsProxy with a simple configuration object
Mads Kiilerich <mads@kiilerich.com>
parents: 6802
diff changeset
66 continue
8268
f8f50d3b6512 celery: upgrade to Celery 4
Mads Kiilerich <mads@kiilerich.com>
parents: 8267
diff changeset
67 if celery_key in list_config_names:
6803
3fb0ce6de10d celery: replace loader.PylonsSettingsProxy with a simple configuration object
Mads Kiilerich <mads@kiilerich.com>
parents: 6802
diff changeset
68 celery_value = config_value.split()
3fb0ce6de10d celery: replace loader.PylonsSettingsProxy with a simple configuration object
Mads Kiilerich <mads@kiilerich.com>
parents: 6802
diff changeset
69 elif config_value.isdigit():
3fb0ce6de10d celery: replace loader.PylonsSettingsProxy with a simple configuration object
Mads Kiilerich <mads@kiilerich.com>
parents: 6802
diff changeset
70 celery_value = int(config_value)
3fb0ce6de10d celery: replace loader.PylonsSettingsProxy with a simple configuration object
Mads Kiilerich <mads@kiilerich.com>
parents: 6802
diff changeset
71 elif config_value.lower() in ['true', 'false']:
3fb0ce6de10d celery: replace loader.PylonsSettingsProxy with a simple configuration object
Mads Kiilerich <mads@kiilerich.com>
parents: 6802
diff changeset
72 celery_value = config_value.lower() == 'true'
3fb0ce6de10d celery: replace loader.PylonsSettingsProxy with a simple configuration object
Mads Kiilerich <mads@kiilerich.com>
parents: 6802
diff changeset
73 else:
3fb0ce6de10d celery: replace loader.PylonsSettingsProxy with a simple configuration object
Mads Kiilerich <mads@kiilerich.com>
parents: 6802
diff changeset
74 celery_value = config_value
3fb0ce6de10d celery: replace loader.PylonsSettingsProxy with a simple configuration object
Mads Kiilerich <mads@kiilerich.com>
parents: 6802
diff changeset
75 setattr(celery_config, celery_key, celery_value)
3fb0ce6de10d celery: replace loader.PylonsSettingsProxy with a simple configuration object
Mads Kiilerich <mads@kiilerich.com>
parents: 6802
diff changeset
76 return celery_config