Mercurial > kallithea
annotate rhodecode/lib/indexers/daemon.py @ 3997:156cb1cdd7ad
Added handling of copied files diff parsing
author | Marcin Kuzminski <marcin@python-works.com> |
---|---|
date | Sat, 15 Jun 2013 20:47:41 +0200 |
parents | 5293d4bbb1ea |
children | ffd45b185016 |
rev | line source |
---|---|
885
94f7585af8a1
fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents:
777
diff
changeset
|
1 # -*- coding: utf-8 -*- |
94f7585af8a1
fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents:
777
diff
changeset
|
2 """ |
94f7585af8a1
fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents:
777
diff
changeset
|
3 rhodecode.lib.indexers.daemon |
94f7585af8a1
fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents:
777
diff
changeset
|
4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
94f7585af8a1
fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents:
777
diff
changeset
|
5 |
1377
78e5853df5c8
fixed daemon typos
Marcin Kuzminski <marcin@python-works.com>
parents:
1206
diff
changeset
|
6 A daemon will read from task table and run tasks |
947
99850ac883d1
Fixed whoosh daemon, for depracated walk method
Marcin Kuzminski <marcin@python-works.com>
parents:
902
diff
changeset
|
7 |
885
94f7585af8a1
fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents:
777
diff
changeset
|
8 :created_on: Jan 26, 2010 |
94f7585af8a1
fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents:
777
diff
changeset
|
9 :author: marcink |
1824
89efedac4e6c
2012 copyrights
Marcin Kuzminski <marcin@python-works.com>
parents:
1711
diff
changeset
|
10 :copyright: (C) 2010-2012 Marcin Kuzminski <marcin@python-works.com> |
885
94f7585af8a1
fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents:
777
diff
changeset
|
11 :license: GPLv3, see COPYING for more details. |
94f7585af8a1
fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents:
777
diff
changeset
|
12 """ |
1206
a671db5bdd58
fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents:
1183
diff
changeset
|
13 # This program is free software: you can redistribute it and/or modify |
a671db5bdd58
fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents:
1183
diff
changeset
|
14 # it under the terms of the GNU General Public License as published by |
a671db5bdd58
fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents:
1183
diff
changeset
|
15 # the Free Software Foundation, either version 3 of the License, or |
a671db5bdd58
fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents:
1183
diff
changeset
|
16 # (at your option) any later version. |
947
99850ac883d1
Fixed whoosh daemon, for depracated walk method
Marcin Kuzminski <marcin@python-works.com>
parents:
902
diff
changeset
|
17 # |
406
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
18 # This program is distributed in the hope that it will be useful, |
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
19 # but WITHOUT ANY WARRANTY; without even the implied warranty of |
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
20 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
21 # GNU General Public License for more details. |
947
99850ac883d1
Fixed whoosh daemon, for depracated walk method
Marcin Kuzminski <marcin@python-works.com>
parents:
902
diff
changeset
|
22 # |
406
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
23 # You should have received a copy of the GNU General Public License |
1206
a671db5bdd58
fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents:
1183
diff
changeset
|
24 # along with this program. If not, see <http://www.gnu.org/licenses/>. |
2641
cfcd981d6679
import with_statment to make daemon.py python 2.5 compatible
Indra Talip <indra.talip@gmail.com>
parents:
2640
diff
changeset
|
25 from __future__ import with_statement |
406
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
26 |
1154
36fe593dfe4b
simplified str2bool, and moved safe_unicode out of helpers since it was not html specific function
Marcin Kuzminski <marcin@python-works.com>
parents:
1036
diff
changeset
|
27 import os |
406
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
28 import sys |
1154
36fe593dfe4b
simplified str2bool, and moved safe_unicode out of helpers since it was not html specific function
Marcin Kuzminski <marcin@python-works.com>
parents:
1036
diff
changeset
|
29 import logging |
885
94f7585af8a1
fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents:
777
diff
changeset
|
30 import traceback |
1154
36fe593dfe4b
simplified str2bool, and moved safe_unicode out of helpers since it was not html specific function
Marcin Kuzminski <marcin@python-works.com>
parents:
1036
diff
changeset
|
31 |
36fe593dfe4b
simplified str2bool, and moved safe_unicode out of helpers since it was not html specific function
Marcin Kuzminski <marcin@python-works.com>
parents:
1036
diff
changeset
|
32 from shutil import rmtree |
36fe593dfe4b
simplified str2bool, and moved safe_unicode out of helpers since it was not html specific function
Marcin Kuzminski <marcin@python-works.com>
parents:
1036
diff
changeset
|
33 from time import mktime |
36fe593dfe4b
simplified str2bool, and moved safe_unicode out of helpers since it was not html specific function
Marcin Kuzminski <marcin@python-works.com>
parents:
1036
diff
changeset
|
34 |
406
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
35 from os.path import dirname as dn |
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
36 from os.path import join as jn |
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
37 |
547
1e757ac98988
renamed project to rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents:
497
diff
changeset
|
38 #to get the rhodecode import |
411
9b67cebe6609
some fixes to whoosh indexer daemon
Marcin Kuzminski <marcin@python-works.com>
parents:
407
diff
changeset
|
39 project_path = dn(dn(dn(dn(os.path.realpath(__file__))))) |
9b67cebe6609
some fixes to whoosh indexer daemon
Marcin Kuzminski <marcin@python-works.com>
parents:
407
diff
changeset
|
40 sys.path.append(project_path) |
406
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
41 |
2109 | 42 from rhodecode.config.conf import INDEX_EXTENSIONS |
691
7486da5f0628
Refactor codes for scm model
Marcin Kuzminski <marcin@python-works.com>
parents:
683
diff
changeset
|
43 from rhodecode.model.scm import ScmModel |
3916
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
44 from rhodecode.model.db import Repository |
3016
b3c8a3a5ce5f
fixed issues some people had with whoosh indexers and unicode decode
Marcin Kuzminski <marcin@python-works.com>
parents:
2841
diff
changeset
|
45 from rhodecode.lib.utils2 import safe_unicode, safe_str |
2648
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
46 from rhodecode.lib.indexers import SCHEMA, IDX_NAME, CHGSETS_SCHEMA, \ |
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
47 CHGSET_IDX_NAME |
411
9b67cebe6609
some fixes to whoosh indexer daemon
Marcin Kuzminski <marcin@python-works.com>
parents:
407
diff
changeset
|
48 |
2007
324ac367a4da
Added VCS into rhodecode core for faster and easier deployments of new versions
Marcin Kuzminski <marcin@python-works.com>
parents:
1995
diff
changeset
|
49 from rhodecode.lib.vcs.exceptions import ChangesetError, RepositoryError, \ |
1711
b369bec5d468
fixes issue with whoosh reindexing files that were removed or renamed
Marcin Kuzminski <marcin@python-works.com>
parents:
1451
diff
changeset
|
50 NodeDoesNotExistError |
560
3072935bdeed
rewrote whoosh indexing to run internal repository.walk() instead of filesystem.
Marcin Kuzminski <marcin@python-works.com>
parents:
557
diff
changeset
|
51 |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
52 from whoosh.index import create_in, open_dir, exists_in |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
53 from whoosh.query import * |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
54 from whoosh.qparser import QueryParser |
1154
36fe593dfe4b
simplified str2bool, and moved safe_unicode out of helpers since it was not html specific function
Marcin Kuzminski <marcin@python-works.com>
parents:
1036
diff
changeset
|
55 |
2101 | 56 log = logging.getLogger('whoosh_indexer') |
406
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
57 |
1995
b6c902d88472
bumbed whoosh to 2.3.X series
Marcin Kuzminski <marcin@python-works.com>
parents:
1824
diff
changeset
|
58 |
406
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
59 class WhooshIndexingDaemon(object): |
560
3072935bdeed
rewrote whoosh indexing to run internal repository.walk() instead of filesystem.
Marcin Kuzminski <marcin@python-works.com>
parents:
557
diff
changeset
|
60 """ |
2373
1828eb7fa688
#469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents:
2372
diff
changeset
|
61 Daemon for atomic indexing jobs |
560
3072935bdeed
rewrote whoosh indexing to run internal repository.walk() instead of filesystem.
Marcin Kuzminski <marcin@python-works.com>
parents:
557
diff
changeset
|
62 """ |
406
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
63 |
1995
b6c902d88472
bumbed whoosh to 2.3.X series
Marcin Kuzminski <marcin@python-works.com>
parents:
1824
diff
changeset
|
64 def __init__(self, indexname=IDX_NAME, index_location=None, |
2373
1828eb7fa688
#469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents:
2372
diff
changeset
|
65 repo_location=None, sa=None, repo_list=None, |
1828eb7fa688
#469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents:
2372
diff
changeset
|
66 repo_update_list=None): |
406
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
67 self.indexname = indexname |
631
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
68 |
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
69 self.index_location = index_location |
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
70 if not index_location: |
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
71 raise Exception('You have to provide index location') |
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
72 |
411
9b67cebe6609
some fixes to whoosh indexer daemon
Marcin Kuzminski <marcin@python-works.com>
parents:
407
diff
changeset
|
73 self.repo_location = repo_location |
631
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
74 if not repo_location: |
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
75 raise Exception('You have to provide repositories location') |
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
76 |
1036
405b80e4ccd5
Major refactoring, removed when possible calls to app globals.
Marcin Kuzminski <marcin@python-works.com>
parents:
947
diff
changeset
|
77 self.repo_paths = ScmModel(sa).repo_scan(self.repo_location) |
894
1fed3c9161bb
fixes #90 + docs update
Marcin Kuzminski <marcin@python-works.com>
parents:
886
diff
changeset
|
78 |
2373
1828eb7fa688
#469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents:
2372
diff
changeset
|
79 #filter repo list |
894
1fed3c9161bb
fixes #90 + docs update
Marcin Kuzminski <marcin@python-works.com>
parents:
886
diff
changeset
|
80 if repo_list: |
2841
2fa3c09f63e0
fixed problems with re-indexing non-ascii names of repositories
Marcin Kuzminski <marcin@python-works.com>
parents:
2840
diff
changeset
|
81 #Fix non-ascii repo names to unicode |
2fa3c09f63e0
fixed problems with re-indexing non-ascii names of repositories
Marcin Kuzminski <marcin@python-works.com>
parents:
2840
diff
changeset
|
82 repo_list = map(safe_unicode, repo_list) |
2373
1828eb7fa688
#469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents:
2372
diff
changeset
|
83 self.filtered_repo_paths = {} |
894
1fed3c9161bb
fixes #90 + docs update
Marcin Kuzminski <marcin@python-works.com>
parents:
886
diff
changeset
|
84 for repo_name, repo in self.repo_paths.items(): |
1fed3c9161bb
fixes #90 + docs update
Marcin Kuzminski <marcin@python-works.com>
parents:
886
diff
changeset
|
85 if repo_name in repo_list: |
2373
1828eb7fa688
#469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents:
2372
diff
changeset
|
86 self.filtered_repo_paths[repo_name] = repo |
1828eb7fa688
#469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents:
2372
diff
changeset
|
87 |
1828eb7fa688
#469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents:
2372
diff
changeset
|
88 self.repo_paths = self.filtered_repo_paths |
894
1fed3c9161bb
fixes #90 + docs update
Marcin Kuzminski <marcin@python-works.com>
parents:
886
diff
changeset
|
89 |
2373
1828eb7fa688
#469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents:
2372
diff
changeset
|
90 #filter update repo list |
1828eb7fa688
#469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents:
2372
diff
changeset
|
91 self.filtered_repo_update_paths = {} |
1828eb7fa688
#469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents:
2372
diff
changeset
|
92 if repo_update_list: |
1828eb7fa688
#469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents:
2372
diff
changeset
|
93 self.filtered_repo_update_paths = {} |
1828eb7fa688
#469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents:
2372
diff
changeset
|
94 for repo_name, repo in self.repo_paths.items(): |
1828eb7fa688
#469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents:
2372
diff
changeset
|
95 if repo_name in repo_update_list: |
1828eb7fa688
#469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents:
2372
diff
changeset
|
96 self.filtered_repo_update_paths[repo_name] = repo |
1828eb7fa688
#469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents:
2372
diff
changeset
|
97 self.repo_paths = self.filtered_repo_update_paths |
894
1fed3c9161bb
fixes #90 + docs update
Marcin Kuzminski <marcin@python-works.com>
parents:
886
diff
changeset
|
98 |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
99 self.initial = True |
631
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
100 if not os.path.isdir(self.index_location): |
763
0dad296d2a57
extended trending languages to more entries, implemented new faster and "fancy"
Marcin Kuzminski <marcin@python-works.com>
parents:
691
diff
changeset
|
101 os.makedirs(self.index_location) |
3916
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
102 log.info('Cannot run incremental index since it does not ' |
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
103 'yet exist running full build') |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
104 elif not exists_in(self.index_location, IDX_NAME): |
3916
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
105 log.info('Running full index build as the file content ' |
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
106 'index does not exist') |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
107 elif not exists_in(self.index_location, CHGSET_IDX_NAME): |
3916
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
108 log.info('Running full index build as the changeset ' |
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
109 'index does not exist') |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
110 else: |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
111 self.initial = False |
631
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
112 |
3916
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
113 def _get_index_revision(self, repo): |
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
114 db_repo = Repository.get_by_repo_name(repo.name) |
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
115 landing_rev = 'tip' |
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
116 if db_repo: |
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
117 landing_rev = db_repo.landing_rev |
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
118 return landing_rev |
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
119 |
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
120 def _get_index_changeset(self, repo): |
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
121 index_rev = self._get_index_revision(repo) |
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
122 cs = repo.get_changeset(index_rev) |
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
123 return cs |
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
124 |
561
5f3b967d9d10
fixed reindexing, and made some optimizations to reuse repo instances from repo scann list.
Marcin Kuzminski <marcin@python-works.com>
parents:
560
diff
changeset
|
125 def get_paths(self, repo): |
2101 | 126 """ |
127 recursive walk in root dir and return a set of all path in that dir | |
560
3072935bdeed
rewrote whoosh indexing to run internal repository.walk() instead of filesystem.
Marcin Kuzminski <marcin@python-works.com>
parents:
557
diff
changeset
|
128 based on repository walk function |
3072935bdeed
rewrote whoosh indexing to run internal repository.walk() instead of filesystem.
Marcin Kuzminski <marcin@python-works.com>
parents:
557
diff
changeset
|
129 """ |
406
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
130 index_paths_ = set() |
567
80dc0a23edf7
fixed whoosh failure on new repository
Marcin Kuzminski <marcin@python-works.com>
parents:
561
diff
changeset
|
131 try: |
3916
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
132 cs = self._get_index_changeset(repo) |
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
133 for _topnode, _dirs, files in cs.walk('/'): |
406
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
134 for f in files: |
3016
b3c8a3a5ce5f
fixed issues some people had with whoosh indexers and unicode decode
Marcin Kuzminski <marcin@python-works.com>
parents:
2841
diff
changeset
|
135 index_paths_.add(jn(safe_str(repo.path), safe_str(f.path))) |
631
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
136 |
2648
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
137 except RepositoryError: |
885
94f7585af8a1
fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents:
777
diff
changeset
|
138 log.debug(traceback.format_exc()) |
567
80dc0a23edf7
fixed whoosh failure on new repository
Marcin Kuzminski <marcin@python-works.com>
parents:
561
diff
changeset
|
139 pass |
631
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
140 return index_paths_ |
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
141 |
561
5f3b967d9d10
fixed reindexing, and made some optimizations to reuse repo instances from repo scann list.
Marcin Kuzminski <marcin@python-works.com>
parents:
560
diff
changeset
|
142 def get_node(self, repo, path): |
3921
932c84e8fa92
fixed #851 and #563 make-index crashes on non-ascii files
Marcin Kuzminski <marcin@python-works.com>
parents:
3916
diff
changeset
|
143 """ |
932c84e8fa92
fixed #851 and #563 make-index crashes on non-ascii files
Marcin Kuzminski <marcin@python-works.com>
parents:
3916
diff
changeset
|
144 gets a filenode based on given full path.It operates on string for |
932c84e8fa92
fixed #851 and #563 make-index crashes on non-ascii files
Marcin Kuzminski <marcin@python-works.com>
parents:
3916
diff
changeset
|
145 hg git compatability. |
932c84e8fa92
fixed #851 and #563 make-index crashes on non-ascii files
Marcin Kuzminski <marcin@python-works.com>
parents:
3916
diff
changeset
|
146 |
932c84e8fa92
fixed #851 and #563 make-index crashes on non-ascii files
Marcin Kuzminski <marcin@python-works.com>
parents:
3916
diff
changeset
|
147 :param repo: scm repo instance |
932c84e8fa92
fixed #851 and #563 make-index crashes on non-ascii files
Marcin Kuzminski <marcin@python-works.com>
parents:
3916
diff
changeset
|
148 :param path: full path including root location |
932c84e8fa92
fixed #851 and #563 make-index crashes on non-ascii files
Marcin Kuzminski <marcin@python-works.com>
parents:
3916
diff
changeset
|
149 :return: FileNode |
932c84e8fa92
fixed #851 and #563 make-index crashes on non-ascii files
Marcin Kuzminski <marcin@python-works.com>
parents:
3916
diff
changeset
|
150 """ |
932c84e8fa92
fixed #851 and #563 make-index crashes on non-ascii files
Marcin Kuzminski <marcin@python-works.com>
parents:
3916
diff
changeset
|
151 root_path = safe_str(repo.path)+'/' |
932c84e8fa92
fixed #851 and #563 make-index crashes on non-ascii files
Marcin Kuzminski <marcin@python-works.com>
parents:
3916
diff
changeset
|
152 parts = safe_str(path).partition(root_path) |
3916
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
153 cs = self._get_index_changeset(repo) |
3921
932c84e8fa92
fixed #851 and #563 make-index crashes on non-ascii files
Marcin Kuzminski <marcin@python-works.com>
parents:
3916
diff
changeset
|
154 node = cs.get_node(parts[-1]) |
561
5f3b967d9d10
fixed reindexing, and made some optimizations to reuse repo instances from repo scann list.
Marcin Kuzminski <marcin@python-works.com>
parents:
560
diff
changeset
|
155 return node |
631
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
156 |
561
5f3b967d9d10
fixed reindexing, and made some optimizations to reuse repo instances from repo scann list.
Marcin Kuzminski <marcin@python-works.com>
parents:
560
diff
changeset
|
157 def get_node_mtime(self, node): |
5f3b967d9d10
fixed reindexing, and made some optimizations to reuse repo instances from repo scann list.
Marcin Kuzminski <marcin@python-works.com>
parents:
560
diff
changeset
|
158 return mktime(node.last_changeset.date.timetuple()) |
631
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
159 |
1171
2ab211e0aecd
changes for #56
Marcin Kuzminski <marcin@python-works.com>
parents:
1154
diff
changeset
|
160 def add_doc(self, writer, path, repo, repo_name): |
2101 | 161 """ |
162 Adding doc to writer this function itself fetches data from | |
163 the instance of vcs backend | |
164 """ | |
165 | |
561
5f3b967d9d10
fixed reindexing, and made some optimizations to reuse repo instances from repo scann list.
Marcin Kuzminski <marcin@python-works.com>
parents:
560
diff
changeset
|
166 node = self.get_node(repo, path) |
2109 | 167 indexed = indexed_w_content = 0 |
2101 | 168 # we just index the content of chosen files, and skip binary files |
886
0736230c7d91
#92 removed content of binary files for whoosh indexer
Marcin Kuzminski <marcin@python-works.com>
parents:
885
diff
changeset
|
169 if node.extension in INDEX_EXTENSIONS and not node.is_binary: |
560
3072935bdeed
rewrote whoosh indexing to run internal repository.walk() instead of filesystem.
Marcin Kuzminski <marcin@python-works.com>
parents:
557
diff
changeset
|
170 u_content = node.content |
885
94f7585af8a1
fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents:
777
diff
changeset
|
171 if not isinstance(u_content, unicode): |
94f7585af8a1
fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents:
777
diff
changeset
|
172 log.warning(' >> %s Could not get this content as unicode ' |
2101 | 173 'replacing with empty content' % path) |
885
94f7585af8a1
fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents:
777
diff
changeset
|
174 u_content = u'' |
94f7585af8a1
fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents:
777
diff
changeset
|
175 else: |
94f7585af8a1
fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents:
777
diff
changeset
|
176 log.debug(' >> %s [WITH CONTENT]' % path) |
2109 | 177 indexed_w_content += 1 |
885
94f7585af8a1
fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents:
777
diff
changeset
|
178 |
406
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
179 else: |
436
28f19fa562df
updated config files,
Marcin Kuzminski <marcin@python-works.com>
parents:
411
diff
changeset
|
180 log.debug(' >> %s' % path) |
2101 | 181 # just index file name without it's content |
436
28f19fa562df
updated config files,
Marcin Kuzminski <marcin@python-works.com>
parents:
411
diff
changeset
|
182 u_content = u'' |
2109 | 183 indexed += 1 |
631
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
184 |
2388
a0ef98f2520b
#453 added ID field in whoosh SCHEMA that solves the issue of reindexing modified files
Marcin Kuzminski <marcin@python-works.com>
parents:
2373
diff
changeset
|
185 p = safe_unicode(path) |
2101 | 186 writer.add_document( |
2388
a0ef98f2520b
#453 added ID field in whoosh SCHEMA that solves the issue of reindexing modified files
Marcin Kuzminski <marcin@python-works.com>
parents:
2373
diff
changeset
|
187 fileid=p, |
2101 | 188 owner=unicode(repo.contact), |
189 repository=safe_unicode(repo_name), | |
2388
a0ef98f2520b
#453 added ID field in whoosh SCHEMA that solves the issue of reindexing modified files
Marcin Kuzminski <marcin@python-works.com>
parents:
2373
diff
changeset
|
190 path=p, |
2101 | 191 content=u_content, |
192 modtime=self.get_node_mtime(node), | |
193 extension=node.extension | |
194 ) | |
2109 | 195 return indexed, indexed_w_content |
631
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
196 |
2643
2ad50c44b025
when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents:
2642
diff
changeset
|
197 def index_changesets(self, writer, repo_name, repo, start_rev=None): |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
198 """ |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
199 Add all changeset in the vcs repo starting at start_rev |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
200 to the index writer |
2643
2ad50c44b025
when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents:
2642
diff
changeset
|
201 |
2ad50c44b025
when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents:
2642
diff
changeset
|
202 :param writer: the whoosh index writer to add to |
2ad50c44b025
when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents:
2642
diff
changeset
|
203 :param repo_name: name of the repository from whence the |
2ad50c44b025
when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents:
2642
diff
changeset
|
204 changeset originates including the repository group |
2ad50c44b025
when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents:
2642
diff
changeset
|
205 :param repo: the vcs repository instance to index changesets for, |
2ad50c44b025
when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents:
2642
diff
changeset
|
206 the presumption is the repo has changesets to index |
2ad50c44b025
when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents:
2642
diff
changeset
|
207 :param start_rev=None: the full sha id to start indexing from |
2ad50c44b025
when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents:
2642
diff
changeset
|
208 if start_rev is None then index from the first changeset in |
2ad50c44b025
when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents:
2642
diff
changeset
|
209 the repo |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
210 """ |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
211 |
2643
2ad50c44b025
when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents:
2642
diff
changeset
|
212 if start_rev is None: |
2ad50c44b025
when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents:
2642
diff
changeset
|
213 start_rev = repo[0].raw_id |
2ad50c44b025
when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents:
2642
diff
changeset
|
214 |
2648
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
215 log.debug('indexing changesets in %s starting at rev: %s' % |
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
216 (repo_name, start_rev)) |
631
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
217 |
2648
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
218 indexed = 0 |
3922
d8e02de53bbc
show number of revisions to parse in whoosh indexing deamon logging
Marcin Kuzminski <marcin@python-works.com>
parents:
3921
diff
changeset
|
219 cs_iter = repo.get_changesets(start=start_rev) |
d8e02de53bbc
show number of revisions to parse in whoosh indexing deamon logging
Marcin Kuzminski <marcin@python-works.com>
parents:
3921
diff
changeset
|
220 total = len(cs_iter) |
d8e02de53bbc
show number of revisions to parse in whoosh indexing deamon logging
Marcin Kuzminski <marcin@python-works.com>
parents:
3921
diff
changeset
|
221 for cs in cs_iter: |
d8e02de53bbc
show number of revisions to parse in whoosh indexing deamon logging
Marcin Kuzminski <marcin@python-works.com>
parents:
3921
diff
changeset
|
222 log.debug(' >> %s/%s' % (cs, total)) |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
223 writer.add_document( |
2642
88b0e82bcba4
rename changeset index key to match raw_id rather than path for greater consistency
Indra Talip <indra.talip@gmail.com>
parents:
2641
diff
changeset
|
224 raw_id=unicode(cs.raw_id), |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
225 owner=unicode(repo.contact), |
2693
66c778b8cb54
Extended commit search schema with date of commit
Marcin Kuzminski <marcin@python-works.com>
parents:
2648
diff
changeset
|
226 date=cs._timestamp, |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
227 repository=safe_unicode(repo_name), |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
228 author=cs.author, |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
229 message=cs.message, |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
230 last=cs.last, |
2763
81624c8a1035
#548 Fixed issue with non-ascii paths in whoosh indexer
Marcin Kuzminski <marcin@python-works.com>
parents:
2693
diff
changeset
|
231 added=u' '.join([safe_unicode(node.path) for node in cs.added]).lower(), |
81624c8a1035
#548 Fixed issue with non-ascii paths in whoosh indexer
Marcin Kuzminski <marcin@python-works.com>
parents:
2693
diff
changeset
|
232 removed=u' '.join([safe_unicode(node.path) for node in cs.removed]).lower(), |
81624c8a1035
#548 Fixed issue with non-ascii paths in whoosh indexer
Marcin Kuzminski <marcin@python-works.com>
parents:
2693
diff
changeset
|
233 changed=u' '.join([safe_unicode(node.path) for node in cs.changed]).lower(), |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
234 parents=u' '.join([cs.raw_id for cs in cs.parents]), |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
235 ) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
236 indexed += 1 |
631
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
237 |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
238 log.debug('indexed %d changesets for repo %s' % (indexed, repo_name)) |
2648
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
239 return indexed |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
240 |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
241 def index_files(self, file_idx_writer, repo_name, repo): |
2648
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
242 """ |
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
243 Index files for given repo_name |
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
244 |
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
245 :param file_idx_writer: the whoosh index writer to add to |
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
246 :param repo_name: name of the repository we're indexing |
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
247 :param repo: instance of vcs repo |
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
248 """ |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
249 i_cnt = iwc_cnt = 0 |
3916
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
250 log.debug('building index for %s @revision:%s' % (repo.path, |
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
251 self._get_index_revision(repo))) |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
252 for idx_path in self.get_paths(repo): |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
253 i, iwc = self.add_doc(file_idx_writer, idx_path, repo, repo_name) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
254 i_cnt += i |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
255 iwc_cnt += iwc |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
256 |
2648
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
257 log.debug('added %s files %s with content for repo %s' % |
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
258 (i_cnt + iwc_cnt, iwc_cnt, repo.path)) |
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
259 return i_cnt, iwc_cnt |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
260 |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
261 def update_changeset_index(self): |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
262 idx = open_dir(self.index_location, indexname=CHGSET_IDX_NAME) |
2569
b98fd6fc67f9
Little better logging in whoosh indexer
Marcin Kuzminski <marcin@python-works.com>
parents:
2388
diff
changeset
|
263 |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
264 with idx.searcher() as searcher: |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
265 writer = idx.writer() |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
266 writer_is_dirty = False |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
267 try: |
2648
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
268 indexed_total = 0 |
2839
c0ddc86b4654
Fix possible exception about repo_name not defined, on whoosh indexer when using index-only option
Marcin Kuzminski <marcin@python-works.com>
parents:
2763
diff
changeset
|
269 repo_name = None |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
270 for repo_name, repo in self.repo_paths.items(): |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
271 # skip indexing if there aren't any revs in the repo |
2643
2ad50c44b025
when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents:
2642
diff
changeset
|
272 num_of_revs = len(repo) |
2ad50c44b025
when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents:
2642
diff
changeset
|
273 if num_of_revs < 1: |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
274 continue |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
275 |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
276 qp = QueryParser('repository', schema=CHGSETS_SCHEMA) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
277 q = qp.parse(u"last:t AND %s" % repo_name) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
278 |
2643
2ad50c44b025
when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents:
2642
diff
changeset
|
279 results = searcher.search(q) |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
280 |
2643
2ad50c44b025
when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents:
2642
diff
changeset
|
281 # default to scanning the entire repo |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
282 last_rev = 0 |
2643
2ad50c44b025
when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents:
2642
diff
changeset
|
283 start_id = None |
2ad50c44b025
when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents:
2642
diff
changeset
|
284 |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
285 if len(results) > 0: |
2643
2ad50c44b025
when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents:
2642
diff
changeset
|
286 # assuming that there is only one result, if not this |
2ad50c44b025
when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents:
2642
diff
changeset
|
287 # may require a full re-index. |
2ad50c44b025
when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents:
2642
diff
changeset
|
288 start_id = results[0]['raw_id'] |
2ad50c44b025
when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents:
2642
diff
changeset
|
289 last_rev = repo.get_changeset(revision=start_id).revision |
631
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
290 |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
291 # there are new changesets to index or a new repo to index |
2643
2ad50c44b025
when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents:
2642
diff
changeset
|
292 if last_rev == 0 or num_of_revs > last_rev + 1: |
2648
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
293 # delete the docs in the index for the previous |
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
294 # last changeset(s) |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
295 for hit in results: |
2648
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
296 q = qp.parse(u"last:t AND %s AND raw_id:%s" % |
2642
88b0e82bcba4
rename changeset index key to match raw_id rather than path for greater consistency
Indra Talip <indra.talip@gmail.com>
parents:
2641
diff
changeset
|
297 (repo_name, hit['raw_id'])) |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
298 writer.delete_by_query(q) |
631
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
299 |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
300 # index from the previous last changeset + all new ones |
2648
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
301 indexed_total += self.index_changesets(writer, |
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
302 repo_name, repo, start_id) |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
303 writer_is_dirty = True |
2648
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
304 log.debug('indexed %s changesets for repo %s' % ( |
3916
ba08786c49ef
fixed #850 Whoosh indexer should use the default revision flag to make index
Marcin Kuzminski <marcin@python-works.com>
parents:
3016
diff
changeset
|
305 indexed_total, repo_name) |
2648
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
306 ) |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
307 finally: |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
308 if writer_is_dirty: |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
309 log.debug('>> COMMITING CHANGES TO CHANGESET INDEX<<') |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
310 writer.commit(merge=True) |
2840
c7c5825299fe
fixed logging messages on whoosh indexer
Marcin Kuzminski <marcin@python-works.com>
parents:
2839
diff
changeset
|
311 log.debug('>>> FINISHED REBUILDING CHANGESET INDEX <<<') |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
312 else: |
2840
c7c5825299fe
fixed logging messages on whoosh indexer
Marcin Kuzminski <marcin@python-works.com>
parents:
2839
diff
changeset
|
313 log.debug('>> NOTHING TO COMMIT TO CHANGESET INDEX<<') |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
314 |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
315 def update_file_index(self): |
2373
1828eb7fa688
#469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents:
2372
diff
changeset
|
316 log.debug((u'STARTING INCREMENTAL INDEXING UPDATE FOR EXTENSIONS %s ' |
1828eb7fa688
#469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents:
2372
diff
changeset
|
317 'AND REPOS %s') % (INDEX_EXTENSIONS, self.repo_paths.keys())) |
631
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
318 |
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
319 idx = open_dir(self.index_location, indexname=self.indexname) |
406
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
320 # The set of all paths in the index |
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
321 indexed_paths = set() |
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
322 # The set of all paths we need to re-index |
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
323 to_index = set() |
631
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
324 |
406
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
325 writer = idx.writer() |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
326 writer_is_dirty = False |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
327 try: |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
328 with idx.reader() as reader: |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
329 |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
330 # Loop over the stored fields in the index |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
331 for fields in reader.all_stored_fields(): |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
332 indexed_path = fields['path'] |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
333 indexed_repo_path = fields['repository'] |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
334 indexed_paths.add(indexed_path) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
335 |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
336 if not indexed_repo_path in self.filtered_repo_update_paths: |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
337 continue |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
338 |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
339 repo = self.repo_paths[indexed_repo_path] |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
340 |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
341 try: |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
342 node = self.get_node(repo, indexed_path) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
343 # Check if this file was changed since it was indexed |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
344 indexed_time = fields['modtime'] |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
345 mtime = self.get_node_mtime(node) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
346 if mtime > indexed_time: |
2648
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
347 # The file has changed, delete it and add it to |
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
348 # the list of files to reindex |
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
349 log.debug( |
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
350 'adding to reindex list %s mtime: %s vs %s' % ( |
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
351 indexed_path, mtime, indexed_time) |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
352 ) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
353 writer.delete_by_term('fileid', indexed_path) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
354 writer_is_dirty = True |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
355 |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
356 to_index.add(indexed_path) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
357 except (ChangesetError, NodeDoesNotExistError): |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
358 # This file was deleted since it was indexed |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
359 log.debug('removing from index %s' % indexed_path) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
360 writer.delete_by_term('path', indexed_path) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
361 writer_is_dirty = True |
631
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
362 |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
363 # Loop over the files in the filesystem |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
364 # Assume we have a function that gathers the filenames of the |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
365 # documents to be indexed |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
366 ri_cnt_total = 0 # indexed |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
367 riwc_cnt_total = 0 # indexed with content |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
368 for repo_name, repo in self.repo_paths.items(): |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
369 # skip indexing if there aren't any revisions |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
370 if len(repo) < 1: |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
371 continue |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
372 ri_cnt = 0 # indexed |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
373 riwc_cnt = 0 # indexed with content |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
374 for path in self.get_paths(repo): |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
375 path = safe_unicode(path) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
376 if path in to_index or path not in indexed_paths: |
631
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
377 |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
378 # This is either a file that's changed, or a new file |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
379 # that wasn't indexed before. So index it! |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
380 i, iwc = self.add_doc(writer, path, repo, repo_name) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
381 writer_is_dirty = True |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
382 log.debug('re indexing %s' % path) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
383 ri_cnt += i |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
384 ri_cnt_total += 1 |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
385 riwc_cnt += iwc |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
386 riwc_cnt_total += iwc |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
387 log.debug('added %s files %s with content for repo %s' % ( |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
388 ri_cnt + riwc_cnt, riwc_cnt, repo.path) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
389 ) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
390 log.debug('indexed %s files in total and %s with content' % ( |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
391 ri_cnt_total, riwc_cnt_total) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
392 ) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
393 finally: |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
394 if writer_is_dirty: |
2840
c7c5825299fe
fixed logging messages on whoosh indexer
Marcin Kuzminski <marcin@python-works.com>
parents:
2839
diff
changeset
|
395 log.debug('>> COMMITING CHANGES TO FILE INDEX <<') |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
396 writer.commit(merge=True) |
2840
c7c5825299fe
fixed logging messages on whoosh indexer
Marcin Kuzminski <marcin@python-works.com>
parents:
2839
diff
changeset
|
397 log.debug('>>> FINISHED REBUILDING FILE INDEX <<<') |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
398 else: |
2840
c7c5825299fe
fixed logging messages on whoosh indexer
Marcin Kuzminski <marcin@python-works.com>
parents:
2839
diff
changeset
|
399 log.debug('>> NOTHING TO COMMIT TO FILE INDEX <<') |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
400 writer.cancel() |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
401 |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
402 def build_indexes(self): |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
403 if os.path.exists(self.index_location): |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
404 log.debug('removing previous index') |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
405 rmtree(self.index_location) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
406 |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
407 if not os.path.exists(self.index_location): |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
408 os.mkdir(self.index_location) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
409 |
2648
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
410 chgset_idx = create_in(self.index_location, CHGSETS_SCHEMA, |
0911cf6940af
little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents:
2643
diff
changeset
|
411 indexname=CHGSET_IDX_NAME) |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
412 chgset_idx_writer = chgset_idx.writer() |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
413 |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
414 file_idx = create_in(self.index_location, SCHEMA, indexname=IDX_NAME) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
415 file_idx_writer = file_idx.writer() |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
416 log.debug('BUILDING INDEX FOR EXTENSIONS %s ' |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
417 'AND REPOS %s' % (INDEX_EXTENSIONS, self.repo_paths.keys())) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
418 |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
419 for repo_name, repo in self.repo_paths.items(): |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
420 # skip indexing if there aren't any revisions |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
421 if len(repo) < 1: |
2373
1828eb7fa688
#469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents:
2372
diff
changeset
|
422 continue |
1828eb7fa688
#469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents:
2372
diff
changeset
|
423 |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
424 self.index_files(file_idx_writer, repo_name, repo) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
425 self.index_changesets(chgset_idx_writer, repo_name, repo) |
631
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
426 |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
427 log.debug('>> COMMITING CHANGES <<') |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
428 file_idx_writer.commit(merge=True) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
429 chgset_idx_writer.commit(merge=True) |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
430 log.debug('>>> FINISHED BUILDING INDEX <<<') |
2388
a0ef98f2520b
#453 added ID field in whoosh SCHEMA that solves the issue of reindexing modified files
Marcin Kuzminski <marcin@python-works.com>
parents:
2373
diff
changeset
|
431 |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
432 def update_indexes(self): |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
433 self.update_file_index() |
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
434 self.update_changeset_index() |
631
05528ad948c4
Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents:
629
diff
changeset
|
435 |
406
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
436 def run(self, full_index=False): |
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
437 """Run daemon""" |
465
e01a85f9fc90
fixed initial whoosh indexer. Build full index on first run even with incremental flag
Marcin Kuzminski <marcin@python-works.com>
parents:
452
diff
changeset
|
438 if full_index or self.initial: |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
439 self.build_indexes() |
406
b153a51b1d3b
Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff
changeset
|
440 else: |
2640
5f21a9dcb09d
create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents:
2569
diff
changeset
|
441 self.update_indexes() |