Mercurial > kallithea

annotate rhodecode/lib/celerylib/tasks.py @ 1354:ed309b1fbaa4 beta

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
fixes issue #197 Relative paths for pidlocks
author Marcin Kuzminski <marcin@python-works.com>
date Fri, 03 Jun 2011 13:44:10 +0200
parents 37625d304a16
children 9c0f5d558789
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
903
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 854
diff changeset
1 # -*- coding: utf-8 -*-
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 854
diff changeset
2 """
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 854
diff changeset
3 rhodecode.lib.celerylib.tasks
1002
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
903
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 854
diff changeset
5
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 854
diff changeset
6 RhodeCode task modules, containing all task that suppose to be run
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 854
diff changeset
7 by celery daemon
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 1169
diff changeset
8
903
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 854
diff changeset
9 :created_on: Oct 6, 2010
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 854
diff changeset
10 :author: marcink
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 1169
diff changeset
11 :copyright: (C) 2009-2011 Marcin Kuzminski <marcin@python-works.com>
903
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 854
diff changeset
12 :license: GPLv3, see COPYING for more details.
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 854
diff changeset
13 """
1206
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
14 # This program is free software: you can redistribute it and/or modify
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
15 # it under the terms of the GNU General Public License as published by
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
16 # the Free Software Foundation, either version 3 of the License, or
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
17 # (at your option) any later version.
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 1169
diff changeset
18 #
903
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 854
diff changeset
19 # This program is distributed in the hope that it will be useful,
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 854
diff changeset
20 # but WITHOUT ANY WARRANTY; without even the implied warranty of
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 854
diff changeset
21 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 854
diff changeset
22 # GNU General Public License for more details.
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 1169
diff changeset
23 #
903
04c9bb9ca6d6 code docs, updates
Marcin Kuzminski <marcin@python-works.com>
parents: 854
diff changeset
24 # You should have received a copy of the GNU General Public License
1206
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
25 # along with this program. If not, see <http://www.gnu.org/licenses/>.
467
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
26 from celery.decorators import task
555
03676d39dd0a added fault tolerant case when celeryconfig is not present in the directory.
Marcin Kuzminski <marcin@python-works.com>
parents: 549
diff changeset
27
692
cb0d9ce6ac5c #50 on point cache invalidation changes.
Marcin Kuzminski <marcin@python-works.com>
parents: 691
diff changeset
28 import os
cb0d9ce6ac5c #50 on point cache invalidation changes.
Marcin Kuzminski <marcin@python-works.com>
parents: 691
diff changeset
29 import traceback
1002
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
30 import logging
1354
ed309b1fbaa4 fixes issue #197 Relative paths for pidlocks
Marcin Kuzminski <marcin@python-works.com>
parents: 1337
diff changeset
31 from os.path import dirname as dn, join as jn
1002
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
32
692
cb0d9ce6ac5c #50 on point cache invalidation changes.
Marcin Kuzminski <marcin@python-works.com>
parents: 691
diff changeset
33 from time import mktime
506
d5efb83590ef fixed lock decorator bug which didn't release the lock after func execution and rewrote the pidlock a little with Ask Solem suggestions!
Marcin Kuzminski <marcin@python-works.com>
parents: 504
diff changeset
34 from operator import itemgetter
1244
0eceb478c720 fixed issue #165 trending source files are now stored in cache as ext only, and translated to description only when displaying, so future changes of mappings will take affect right away.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
35 from string import lower
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 752
diff changeset
36
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 752
diff changeset
37 from pylons import config
474
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
38 from pylons.i18n.translation import _
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 752
diff changeset
39
1302
f0e904651f21 moved LANGUAGE_EXTENSION_MAP to lib, and made whoosh indexer use the same map
Marcin Kuzminski <marcin@python-works.com>
parents: 1264
diff changeset
40 from rhodecode.lib import LANGUAGES_EXTENSIONS_MAP
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
41 from rhodecode.lib.celerylib import run_task, locked_task, str2bool, \
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
42 __get_lockkey, LockHeld, DaemonLock
547
1e757ac98988 renamed project to rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 535
diff changeset
43 from rhodecode.lib.helpers import person
1e757ac98988 renamed project to rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 535
diff changeset
44 from rhodecode.lib.smtp_mailer import SmtpMailer
1337
37625d304a16 Changed OrderedDict implementation to pypy odict, in general it's the fastest and most reliable solution. Added OrderedTuple from python foundation.
Marcin Kuzminski <marcin@python-works.com>
parents: 1302
diff changeset
45 from rhodecode.lib.utils import add_cache
37625d304a16 Changed OrderedDict implementation to pypy odict, in general it's the fastest and most reliable solution. Added OrderedTuple from python foundation.
Marcin Kuzminski <marcin@python-works.com>
parents: 1302
diff changeset
46 from rhodecode.lib.odict import OrderedDict
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 752
diff changeset
47 from rhodecode.model import init_model
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 752
diff changeset
48 from rhodecode.model import meta
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
49 from rhodecode.model.db import RhodeCodeUi, Statistics, Repository
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 752
diff changeset
50
631
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 630
diff changeset
51 from vcs.backends import get_repo
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 752
diff changeset
52
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 752
diff changeset
53 from sqlalchemy import engine_from_config
692
cb0d9ce6ac5c #50 on point cache invalidation changes.
Marcin Kuzminski <marcin@python-works.com>
parents: 691
diff changeset
54
777
aac24db58ce8 fixed cache problem,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
55 add_cache(config)
aac24db58ce8 fixed cache problem,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
56
555
03676d39dd0a added fault tolerant case when celeryconfig is not present in the directory.
Marcin Kuzminski <marcin@python-works.com>
parents: 549
diff changeset
57 try:
564
ba7e24cd4786 refactor codes and setup for python 2.5
Marcin Kuzminski <marcin@python-works.com>
parents: 562
diff changeset
58 import json
ba7e24cd4786 refactor codes and setup for python 2.5
Marcin Kuzminski <marcin@python-works.com>
parents: 562
diff changeset
59 except ImportError:
ba7e24cd4786 refactor codes and setup for python 2.5
Marcin Kuzminski <marcin@python-works.com>
parents: 562
diff changeset
60 #python 2.5 compatibility
ba7e24cd4786 refactor codes and setup for python 2.5
Marcin Kuzminski <marcin@python-works.com>
parents: 562
diff changeset
61 import simplejson as json
ba7e24cd4786 refactor codes and setup for python 2.5
Marcin Kuzminski <marcin@python-works.com>
parents: 562
diff changeset
62
474
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
63 __all__ = ['whoosh_index', 'get_commits_stats',
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
64 'reset_user_password', 'send_email']
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
65
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 752
diff changeset
66 CELERY_ON = str2bool(config['app_conf'].get('use_celery'))
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 752
diff changeset
67
1244
0eceb478c720 fixed issue #165 trending source files are now stored in cache as ext only, and translated to description only when displaying, so future changes of mappings will take affect right away.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
68
0eceb478c720 fixed issue #165 trending source files are now stored in cache as ext only, and translated to description only when displaying, so future changes of mappings will take affect right away.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
69
474
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
70 def get_session():
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 752
diff changeset
71 if CELERY_ON:
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 752
diff changeset
72 engine = engine_from_config(config, 'sqlalchemy.db1.')
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 752
diff changeset
73 init_model(engine)
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 752
diff changeset
74 sa = meta.Session()
474
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
75 return sa
467
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
76
1244
0eceb478c720 fixed issue #165 trending source files are now stored in cache as ext only, and translated to description only when displaying, so future changes of mappings will take affect right away.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
77
692
cb0d9ce6ac5c #50 on point cache invalidation changes.
Marcin Kuzminski <marcin@python-works.com>
parents: 691
diff changeset
78 def get_repos_path():
cb0d9ce6ac5c #50 on point cache invalidation changes.
Marcin Kuzminski <marcin@python-works.com>
parents: 691
diff changeset
79 sa = get_session()
cb0d9ce6ac5c #50 on point cache invalidation changes.
Marcin Kuzminski <marcin@python-works.com>
parents: 691
diff changeset
80 q = sa.query(RhodeCodeUi).filter(RhodeCodeUi.ui_key == '/').one()
cb0d9ce6ac5c #50 on point cache invalidation changes.
Marcin Kuzminski <marcin@python-works.com>
parents: 691
diff changeset
81 return q.ui_value
cb0d9ce6ac5c #50 on point cache invalidation changes.
Marcin Kuzminski <marcin@python-works.com>
parents: 691
diff changeset
82
1244
0eceb478c720 fixed issue #165 trending source files are now stored in cache as ext only, and translated to description only when displaying, so future changes of mappings will take affect right away.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
83
1002
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
84 @task(ignore_result=True)
504
d280aa1c85c6 removed pidlock from whoosh and added it as locked_task decorator
Marcin Kuzminski <marcin@python-works.com>
parents: 502
diff changeset
85 @locked_task
474
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
86 def whoosh_index(repo_location, full_index):
1002
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
87 #log = whoosh_index.get_logger()
547
1e757ac98988 renamed project to rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 535
diff changeset
88 from rhodecode.lib.indexers.daemon import WhooshIndexingDaemon
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 752
diff changeset
89 index_location = config['index_dir']
662
373ee7031003 fixed annotation bug, added history to annotation.
Marcin Kuzminski <marcin@python-works.com>
parents: 659
diff changeset
90 WhooshIndexingDaemon(index_location=index_location,
777
aac24db58ce8 fixed cache problem,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
91 repo_location=repo_location, sa=get_session())\
aac24db58ce8 fixed cache problem,
Marcin Kuzminski <marcin@python-works.com>
parents: 776
diff changeset
92 .run(full_index=full_index)
497
fb0c3af6031b Implemented locking for task, to prevent for running the same tasks,
Marcin Kuzminski <marcin@python-works.com>
parents: 493
diff changeset
93
1244
0eceb478c720 fixed issue #165 trending source files are now stored in cache as ext only, and translated to description only when displaying, so future changes of mappings will take affect right away.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
94
1002
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
95 @task(ignore_result=True)
493
2256c78afe53 implemented basic autoupdating statistics fetched from database
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
96 def get_commits_stats(repo_name, ts_min_y, ts_max_y):
1002
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
97 try:
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
98 log = get_commits_stats.get_logger()
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
99 except:
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
100 log = logging.getLogger(__name__)
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
101
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
102 lockkey = __get_lockkey('get_commits_stats', repo_name, ts_min_y,
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
103 ts_max_y)
1354
ed309b1fbaa4 fixes issue #197 Relative paths for pidlocks
Marcin Kuzminski <marcin@python-works.com>
parents: 1337
diff changeset
104 lockkey_path = dn(dn(dn(dn(os.path.abspath(__file__)))))
ed309b1fbaa4 fixes issue #197 Relative paths for pidlocks
Marcin Kuzminski <marcin@python-works.com>
parents: 1337
diff changeset
105 print jn(lockkey_path, lockkey)
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
106 log.info('running task with lockkey %s', lockkey)
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
107 try:
1354
ed309b1fbaa4 fixes issue #197 Relative paths for pidlocks
Marcin Kuzminski <marcin@python-works.com>
parents: 1337
diff changeset
108 lock = l = DaemonLock(jn(lockkey_path, lockkey))
699
52da7cba88a6 Code refactor for auth func, preparing for ldap support
Marcin Kuzminski <marcin@python-works.com>
parents: 692
diff changeset
109
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
110 #for js data compatibilty cleans the key for person from '
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
111 akc = lambda k: person(k).replace('"', "")
629
7e536d1af60d Code refactoring,models renames
Marcin Kuzminski <marcin@python-works.com>
parents: 603
diff changeset
112
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
113 co_day_auth_aggr = {}
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
114 commits_by_day_aggregate = {}
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
115 repos_path = get_repos_path()
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
116 p = os.path.join(repos_path, repo_name)
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
117 repo = get_repo(p)
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
118 repo_size = len(repo.revisions)
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
119 #return if repo have no revisions
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
120 if repo_size < 1:
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
121 lock.release()
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
122 return True
493
2256c78afe53 implemented basic autoupdating statistics fetched from database
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
123
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
124 skip_date_limit = True
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
125 parse_limit = int(config['app_conf'].get('commit_parse_limit'))
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
126 last_rev = 0
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
127 last_cs = None
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
128 timegetter = itemgetter('time')
629
7e536d1af60d Code refactoring,models renames
Marcin Kuzminski <marcin@python-works.com>
parents: 603
diff changeset
129
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
130 sa = get_session()
629
7e536d1af60d Code refactoring,models renames
Marcin Kuzminski <marcin@python-works.com>
parents: 603
diff changeset
131
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
132 dbrepo = sa.query(Repository)\
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
133 .filter(Repository.repo_name == repo_name).scalar()
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
134 cur_stats = sa.query(Statistics)\
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
135 .filter(Statistics.repository == dbrepo).scalar()
1105
61c5f0ab24b9 Added branch filter to repo pager
Marcin Kuzminski <marcin@python-works.com>
parents: 1076
diff changeset
136
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
137 if cur_stats is not None:
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
138 last_rev = cur_stats.stat_on_revision
1105
61c5f0ab24b9 Added branch filter to repo pager
Marcin Kuzminski <marcin@python-works.com>
parents: 1076
diff changeset
139
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
140 if last_rev == repo.get_changeset().revision and repo_size > 1:
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
141 #pass silently without any work if we're not on first revision or
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
142 #current state of parsing revision(from db marker) is the
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
143 #last revision
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
144 lock.release()
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
145 return True
629
7e536d1af60d Code refactoring,models renames
Marcin Kuzminski <marcin@python-works.com>
parents: 603
diff changeset
146
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
147 if cur_stats:
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
148 commits_by_day_aggregate = OrderedDict(json.loads(
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
149 cur_stats.commit_activity_combined))
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
150 co_day_auth_aggr = json.loads(cur_stats.commit_activity)
629
7e536d1af60d Code refactoring,models renames
Marcin Kuzminski <marcin@python-works.com>
parents: 603
diff changeset
151
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
152 log.debug('starting parsing %s', parse_limit)
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
153 lmktime = mktime
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
154
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
155 last_rev = last_rev + 1 if last_rev > 0 else last_rev
629
7e536d1af60d Code refactoring,models renames
Marcin Kuzminski <marcin@python-works.com>
parents: 603
diff changeset
156
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
157 for cs in repo[last_rev:last_rev + parse_limit]:
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
158 last_cs = cs # remember last parsed changeset
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
159 k = lmktime([cs.date.timetuple()[0], cs.date.timetuple()[1],
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
160 cs.date.timetuple()[2], 0, 0, 0, 0, 0, 0])
1076
db71228a4c23 moved statistics parse_limit into .ini files
Marcin Kuzminski <marcin@python-works.com>
parents: 1002
diff changeset
161
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
162 if akc(cs.author) in co_day_auth_aggr:
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
163 try:
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
164 l = [timegetter(x) for x in
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
165 co_day_auth_aggr[akc(cs.author)]['data']]
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
166 time_pos = l.index(k)
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
167 except ValueError:
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
168 time_pos = False
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
169
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
170 if time_pos >= 0 and time_pos is not False:
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
171
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
172 datadict = \
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
173 co_day_auth_aggr[akc(cs.author)]['data'][time_pos]
804
b92d9a0321e0 code stats speed improvments
Marcin Kuzminski <marcin@python-works.com>
parents: 803
diff changeset
174
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
175 datadict["commits"] += 1
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
176 datadict["added"] += len(cs.added)
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
177 datadict["changed"] += len(cs.changed)
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
178 datadict["removed"] += len(cs.removed)
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
179
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
180 else:
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
181 if k >= ts_min_y and k <= ts_max_y or skip_date_limit:
629
7e536d1af60d Code refactoring,models renames
Marcin Kuzminski <marcin@python-works.com>
parents: 603
diff changeset
182
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
183 datadict = {"time": k,
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
184 "commits": 1,
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
185 "added": len(cs.added),
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
186 "changed": len(cs.changed),
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
187 "removed": len(cs.removed),
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
188 }
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
189 co_day_auth_aggr[akc(cs.author)]['data']\
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
190 .append(datadict)
629
7e536d1af60d Code refactoring,models renames
Marcin Kuzminski <marcin@python-works.com>
parents: 603
diff changeset
191
467
3fc3ce53659b starting celery branch
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
192 else:
487
b12ea84fb906 Some fixes to summary, and total rewrite of summary graphs implemented more interactive graph.
Marcin Kuzminski <marcin@python-works.com>
parents: 486
diff changeset
193 if k >= ts_min_y and k <= ts_max_y or skip_date_limit:
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
194 co_day_auth_aggr[akc(cs.author)] = {
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
195 "label": akc(cs.author),
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
196 "data": [{"time":k,
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
197 "commits":1,
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
198 "added":len(cs.added),
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
199 "changed":len(cs.changed),
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
200 "removed":len(cs.removed),
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
201 }],
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
202 "schema": ["commits"],
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
203 }
629
7e536d1af60d Code refactoring,models renames
Marcin Kuzminski <marcin@python-works.com>
parents: 603
diff changeset
204
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
205 #gather all data by day
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
206 if k in commits_by_day_aggregate:
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
207 commits_by_day_aggregate[k] += 1
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
208 else:
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
209 commits_by_day_aggregate[k] = 1
629
7e536d1af60d Code refactoring,models renames
Marcin Kuzminski <marcin@python-works.com>
parents: 603
diff changeset
210
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
211 overview_data = sorted(commits_by_day_aggregate.items(),
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
212 key=itemgetter(0))
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
213
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
214 if not co_day_auth_aggr:
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
215 co_day_auth_aggr[akc(repo.contact)] = {
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
216 "label": akc(repo.contact),
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
217 "data": [0, 1],
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
218 "schema": ["commits"],
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
219 }
629
7e536d1af60d Code refactoring,models renames
Marcin Kuzminski <marcin@python-works.com>
parents: 603
diff changeset
220
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
221 stats = cur_stats if cur_stats else Statistics()
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
222 stats.commit_activity = json.dumps(co_day_auth_aggr)
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
223 stats.commit_activity_combined = json.dumps(overview_data)
493
2256c78afe53 implemented basic autoupdating statistics fetched from database
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
224
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
225 log.debug('last revison %s', last_rev)
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
226 leftovers = len(repo.revisions[last_rev:])
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
227 log.debug('revisions to parse %s', leftovers)
506
d5efb83590ef fixed lock decorator bug which didn't release the lock after func execution and rewrote the pidlock a little with Ask Solem suggestions!
Marcin Kuzminski <marcin@python-works.com>
parents: 504
diff changeset
228
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
229 if last_rev == 0 or leftovers < parse_limit:
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
230 log.debug('getting code trending stats')
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
231 stats.languages = json.dumps(__get_codes_stats(repo_name))
629
7e536d1af60d Code refactoring,models renames
Marcin Kuzminski <marcin@python-works.com>
parents: 603
diff changeset
232
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
233 try:
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
234 stats.repository = dbrepo
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
235 stats.stat_on_revision = last_cs.revision if last_cs else 0
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
236 sa.add(stats)
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
237 sa.commit()
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
238 except:
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
239 log.error(traceback.format_exc())
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
240 sa.rollback()
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
241 lock.release()
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
242 return False
629
7e536d1af60d Code refactoring,models renames
Marcin Kuzminski <marcin@python-works.com>
parents: 603
diff changeset
243
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
244 #final release
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
245 lock.release()
629
7e536d1af60d Code refactoring,models renames
Marcin Kuzminski <marcin@python-works.com>
parents: 603
diff changeset
246
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
247 #execute another task if celery is enabled
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
248 if len(repo.revisions) > 1 and CELERY_ON:
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
249 run_task(get_commits_stats, repo_name, ts_min_y, ts_max_y)
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
250 return True
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
251 except LockHeld:
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
252 log.info('LockHeld')
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
253 return 'Task with key %s already running' % lockkey
474
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
254
1244
0eceb478c720 fixed issue #165 trending source files are now stored in cache as ext only, and translated to description only when displaying, so future changes of mappings will take affect right away.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
255
1002
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
256 @task(ignore_result=True)
474
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
257 def reset_user_password(user_email):
1002
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
258 try:
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
259 log = reset_user_password.get_logger()
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
260 except:
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
261 log = logging.getLogger(__name__)
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
262
547
1e757ac98988 renamed project to rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 535
diff changeset
263 from rhodecode.lib import auth
1e757ac98988 renamed project to rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 535
diff changeset
264 from rhodecode.model.db import User
629
7e536d1af60d Code refactoring,models renames
Marcin Kuzminski <marcin@python-works.com>
parents: 603
diff changeset
265
474
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
266 try:
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
267 try:
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
268 sa = get_session()
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
269 user = sa.query(User).filter(User.email == user_email).scalar()
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
270 new_passwd = auth.PasswordGenerator().gen_password(8,
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
271 auth.PasswordGenerator.ALPHABETS_BIG_SMALL)
493
2256c78afe53 implemented basic autoupdating statistics fetched from database
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
272 if user:
2256c78afe53 implemented basic autoupdating statistics fetched from database
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
273 user.password = auth.get_crypt_password(new_passwd)
1116
716911af91e1 Added api_key into user, api key get's generated again after password change
Marcin Kuzminski <marcin@python-works.com>
parents: 1105
diff changeset
274 user.api_key = auth.generate_api_key(user.username)
493
2256c78afe53 implemented basic autoupdating statistics fetched from database
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
275 sa.add(user)
2256c78afe53 implemented basic autoupdating statistics fetched from database
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
276 sa.commit()
2256c78afe53 implemented basic autoupdating statistics fetched from database
Marcin Kuzminski <marcin@python-works.com>
parents: 487
diff changeset
277 log.info('change password for %s', user_email)
474
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
278 if new_passwd is None:
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
279 raise Exception('unable to generate new password')
629
7e536d1af60d Code refactoring,models renames
Marcin Kuzminski <marcin@python-works.com>
parents: 603
diff changeset
280
474
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
281 except:
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
282 log.error(traceback.format_exc())
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
283 sa.rollback()
629
7e536d1af60d Code refactoring,models renames
Marcin Kuzminski <marcin@python-works.com>
parents: 603
diff changeset
284
474
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
285 run_task(send_email, user_email,
549
f99075170eb4 more renames for rhode code !!
Marcin Kuzminski <marcin@python-works.com>
parents: 548
diff changeset
286 "Your new rhodecode password",
f99075170eb4 more renames for rhode code !!
Marcin Kuzminski <marcin@python-works.com>
parents: 548
diff changeset
287 'Your new rhodecode password:%s' % (new_passwd))
474
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
288 log.info('send new password mail to %s', user_email)
629
7e536d1af60d Code refactoring,models renames
Marcin Kuzminski <marcin@python-works.com>
parents: 603
diff changeset
289
474
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
290 except:
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
291 log.error('Failed to update user password')
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
292 log.error(traceback.format_exc())
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 752
diff changeset
293
474
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
294 return True
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
295
1244
0eceb478c720 fixed issue #165 trending source files are now stored in cache as ext only, and translated to description only when displaying, so future changes of mappings will take affect right away.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
296
1002
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
297 @task(ignore_result=True)
474
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
298 def send_email(recipients, subject, body):
689
ecc566f8b69f fixes #59, notifications for user registrations + some changes to mailer
Marcin Kuzminski <marcin@python-works.com>
parents: 685
diff changeset
299 """
ecc566f8b69f fixes #59, notifications for user registrations + some changes to mailer
Marcin Kuzminski <marcin@python-works.com>
parents: 685
diff changeset
300 Sends an email with defined parameters from the .ini files.
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 1169
diff changeset
301
689
ecc566f8b69f fixes #59, notifications for user registrations + some changes to mailer
Marcin Kuzminski <marcin@python-works.com>
parents: 685
diff changeset
302 :param recipients: list of recipients, it this is empty the defined email
ecc566f8b69f fixes #59, notifications for user registrations + some changes to mailer
Marcin Kuzminski <marcin@python-works.com>
parents: 685
diff changeset
303 address from field 'email_to' is used instead
ecc566f8b69f fixes #59, notifications for user registrations + some changes to mailer
Marcin Kuzminski <marcin@python-works.com>
parents: 685
diff changeset
304 :param subject: subject of the mail
ecc566f8b69f fixes #59, notifications for user registrations + some changes to mailer
Marcin Kuzminski <marcin@python-works.com>
parents: 685
diff changeset
305 :param body: body of the mail
ecc566f8b69f fixes #59, notifications for user registrations + some changes to mailer
Marcin Kuzminski <marcin@python-works.com>
parents: 685
diff changeset
306 """
1002
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
307 try:
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
308 log = send_email.get_logger()
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
309 except:
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
310 log = logging.getLogger(__name__)
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
311
776
f6c613fba757 Celery is configured by the .ini files and run from paster now
Marcin Kuzminski <marcin@python-works.com>
parents: 752
diff changeset
312 email_config = config
689
ecc566f8b69f fixes #59, notifications for user registrations + some changes to mailer
Marcin Kuzminski <marcin@python-works.com>
parents: 685
diff changeset
313
ecc566f8b69f fixes #59, notifications for user registrations + some changes to mailer
Marcin Kuzminski <marcin@python-works.com>
parents: 685
diff changeset
314 if not recipients:
ecc566f8b69f fixes #59, notifications for user registrations + some changes to mailer
Marcin Kuzminski <marcin@python-works.com>
parents: 685
diff changeset
315 recipients = [email_config.get('email_to')]
ecc566f8b69f fixes #59, notifications for user registrations + some changes to mailer
Marcin Kuzminski <marcin@python-works.com>
parents: 685
diff changeset
316
474
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
317 mail_from = email_config.get('app_email_from')
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
318 user = email_config.get('smtp_username')
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
319 passwd = email_config.get('smtp_password')
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
320 mail_server = email_config.get('smtp_server')
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
321 mail_port = email_config.get('smtp_port')
689
ecc566f8b69f fixes #59, notifications for user registrations + some changes to mailer
Marcin Kuzminski <marcin@python-works.com>
parents: 685
diff changeset
322 tls = str2bool(email_config.get('smtp_use_tls'))
ecc566f8b69f fixes #59, notifications for user registrations + some changes to mailer
Marcin Kuzminski <marcin@python-works.com>
parents: 685
diff changeset
323 ssl = str2bool(email_config.get('smtp_use_ssl'))
1169
f6dca275c5a8 control mailer debug with the .ini file
Marcin Kuzminski <marcin@python-works.com>
parents: 1116
diff changeset
324 debug = str2bool(config.get('debug'))
629
7e536d1af60d Code refactoring,models renames
Marcin Kuzminski <marcin@python-works.com>
parents: 603
diff changeset
325
474
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
326 try:
483
a9e50dce3081 Removed config names from whoosh and celery,
Marcin Kuzminski <marcin@python-works.com>
parents: 474
diff changeset
327 m = SmtpMailer(mail_from, user, passwd, mail_server,
1169
f6dca275c5a8 control mailer debug with the .ini file
Marcin Kuzminski <marcin@python-works.com>
parents: 1116
diff changeset
328 mail_port, ssl, tls, debug=debug)
629
7e536d1af60d Code refactoring,models renames
Marcin Kuzminski <marcin@python-works.com>
parents: 603
diff changeset
329 m.send(recipients, subject, body)
474
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
330 except:
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
331 log.error('Mail sending failed')
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
332 log.error(traceback.format_exc())
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
333 return False
a3d9d24acbec Implemented password reset(forms/models/ tasks) and mailing tasks.
Marcin Kuzminski <marcin@python-works.com>
parents: 467
diff changeset
334 return True
506
d5efb83590ef fixed lock decorator bug which didn't release the lock after func execution and rewrote the pidlock a little with Ask Solem suggestions!
Marcin Kuzminski <marcin@python-works.com>
parents: 504
diff changeset
335
1244
0eceb478c720 fixed issue #165 trending source files are now stored in cache as ext only, and translated to description only when displaying, so future changes of mappings will take affect right away.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
336
1002
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
337 @task(ignore_result=True)
530
a08f610e545e Implemented server side forks
Marcin Kuzminski <marcin@python-works.com>
parents: 509
diff changeset
338 def create_repo_fork(form_data, cur_user):
1264
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
339 from rhodecode.model.repo import RepoModel
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
340 from vcs import get_backend
0c43c6671815 moved locking of commit stats into the task itself to remove race conditions when lock was not removed before starting another task.
Marcin Kuzminski <marcin@python-works.com>
parents: 1244
diff changeset
341
1002
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
342 try:
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
343 log = create_repo_fork.get_logger()
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
344 except:
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
345 log = logging.getLogger(__name__)
3a7f5b1a19dd made rhodecode work with celery 2.2, made some tasks optimizations(forget results)
Marcin Kuzminski <marcin@python-works.com>
parents: 903
diff changeset
346
752
89b9037d68b7 fixed Example celery config to ampq,
Marcin Kuzminski <marcin@python-works.com>
parents: 722
diff changeset
347 repo_model = RepoModel(get_session())
630
a9e72f7148c8 some small fixes
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
348 repo_model.create(form_data, cur_user, just_db=True, fork=True)
659
758f64f3fbda extended repo creation by repo type. fixed fork creation to maintain repo type.
Marcin Kuzminski <marcin@python-works.com>
parents: 635
diff changeset
349 repo_name = form_data['repo_name']
692
cb0d9ce6ac5c #50 on point cache invalidation changes.
Marcin Kuzminski <marcin@python-works.com>
parents: 691
diff changeset
350 repos_path = get_repos_path()
659
758f64f3fbda extended repo creation by repo type. fixed fork creation to maintain repo type.
Marcin Kuzminski <marcin@python-works.com>
parents: 635
diff changeset
351 repo_path = os.path.join(repos_path, repo_name)
530
a08f610e545e Implemented server side forks
Marcin Kuzminski <marcin@python-works.com>
parents: 509
diff changeset
352 repo_fork_path = os.path.join(repos_path, form_data['fork_name'])
659
758f64f3fbda extended repo creation by repo type. fixed fork creation to maintain repo type.
Marcin Kuzminski <marcin@python-works.com>
parents: 635
diff changeset
353 alias = form_data['repo_type']
629
7e536d1af60d Code refactoring,models renames
Marcin Kuzminski <marcin@python-works.com>
parents: 603
diff changeset
354
659
758f64f3fbda extended repo creation by repo type. fixed fork creation to maintain repo type.
Marcin Kuzminski <marcin@python-works.com>
parents: 635
diff changeset
355 log.info('creating repo fork %s as %s', repo_name, repo_path)
758f64f3fbda extended repo creation by repo type. fixed fork creation to maintain repo type.
Marcin Kuzminski <marcin@python-works.com>
parents: 635
diff changeset
356 backend = get_backend(alias)
758f64f3fbda extended repo creation by repo type. fixed fork creation to maintain repo type.
Marcin Kuzminski <marcin@python-works.com>
parents: 635
diff changeset
357 backend(str(repo_fork_path), create=True, src_url=str(repo_path))
629
7e536d1af60d Code refactoring,models renames
Marcin Kuzminski <marcin@python-works.com>
parents: 603
diff changeset
358
1244
0eceb478c720 fixed issue #165 trending source files are now stored in cache as ext only, and translated to description only when displaying, so future changes of mappings will take affect right away.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
359
506
d5efb83590ef fixed lock decorator bug which didn't release the lock after func execution and rewrote the pidlock a little with Ask Solem suggestions!
Marcin Kuzminski <marcin@python-works.com>
parents: 504
diff changeset
360 def __get_codes_stats(repo_name):
692
cb0d9ce6ac5c #50 on point cache invalidation changes.
Marcin Kuzminski <marcin@python-works.com>
parents: 691
diff changeset
361 repos_path = get_repos_path()
631
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 630
diff changeset
362 p = os.path.join(repos_path, repo_name)
635
fd63782c4426 Fixed age, for new vcs implementation. Removed all obsolete date formatters
Marcin Kuzminski <marcin@python-works.com>
parents: 631
diff changeset
363 repo = get_repo(p)
603
95a502d94860 removed soon deprecated walk method on repository instance
Marcin Kuzminski <marcin@python-works.com>
parents: 564
diff changeset
364 tip = repo.get_changeset()
506
d5efb83590ef fixed lock decorator bug which didn't release the lock after func execution and rewrote the pidlock a little with Ask Solem suggestions!
Marcin Kuzminski <marcin@python-works.com>
parents: 504
diff changeset
365 code_stats = {}
630
a9e72f7148c8 some small fixes
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
366
a9e72f7148c8 some small fixes
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
367 def aggregate(cs):
a9e72f7148c8 some small fixes
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
368 for f in cs[2]:
1244
0eceb478c720 fixed issue #165 trending source files are now stored in cache as ext only, and translated to description only when displaying, so future changes of mappings will take affect right away.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
369 ext = lower(f.extension)
789
9fec2207f91c removed binary files from trending sources
Marcin Kuzminski <marcin@python-works.com>
parents: 785
diff changeset
370 if ext in LANGUAGES_EXTENSIONS_MAP.keys() and not f.is_binary:
1244
0eceb478c720 fixed issue #165 trending source files are now stored in cache as ext only, and translated to description only when displaying, so future changes of mappings will take affect right away.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
371 if ext in code_stats:
0eceb478c720 fixed issue #165 trending source files are now stored in cache as ext only, and translated to description only when displaying, so future changes of mappings will take affect right away.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
372 code_stats[ext] += 1
506
d5efb83590ef fixed lock decorator bug which didn't release the lock after func execution and rewrote the pidlock a little with Ask Solem suggestions!
Marcin Kuzminski <marcin@python-works.com>
parents: 504
diff changeset
373 else:
1244
0eceb478c720 fixed issue #165 trending source files are now stored in cache as ext only, and translated to description only when displaying, so future changes of mappings will take affect right away.
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
374 code_stats[ext] = 1
629
7e536d1af60d Code refactoring,models renames
Marcin Kuzminski <marcin@python-works.com>
parents: 603
diff changeset
375
630
a9e72f7148c8 some small fixes
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
376 map(aggregate, tip.walk('/'))
a9e72f7148c8 some small fixes
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
377
506
d5efb83590ef fixed lock decorator bug which didn't release the lock after func execution and rewrote the pidlock a little with Ask Solem suggestions!
Marcin Kuzminski <marcin@python-works.com>
parents: 504
diff changeset
378 return code_stats or {}