Mercurial > kallithea

annotate rhodecode/lib/indexers/daemon.py @ 3774:60335b702a00 beta

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
invalidation: don't create CacheInvalidation records on startup Creating the records early gave an advantage before lightweight was introduced. With lightweight it is no longer necessary. The records will be created on demand anyway and there is no reason to create and maintain them before they are used.
author Mads Kiilerich <madski@unity3d.com>
date Wed, 03 Apr 2013 15:56:12 +0200
parents b3c8a3a5ce5f
children a5f0bc867edc ba08786c49ef
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
885
94f7585af8a1 fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents: 777
diff changeset
1 # -*- coding: utf-8 -*-
94f7585af8a1 fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents: 777
diff changeset
2 """
94f7585af8a1 fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents: 777
diff changeset
3 rhodecode.lib.indexers.daemon
94f7585af8a1 fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents: 777
diff changeset
4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
94f7585af8a1 fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents: 777
diff changeset
5
1377
78e5853df5c8 fixed daemon typos
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
6 A daemon will read from task table and run tasks
947
99850ac883d1 Fixed whoosh daemon, for depracated walk method
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
7
885
94f7585af8a1 fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents: 777
diff changeset
8 :created_on: Jan 26, 2010
94f7585af8a1 fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents: 777
diff changeset
9 :author: marcink
1824
89efedac4e6c 2012 copyrights
Marcin Kuzminski <marcin@python-works.com>
parents: 1711
diff changeset
10 :copyright: (C) 2010-2012 Marcin Kuzminski <marcin@python-works.com>
885
94f7585af8a1 fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents: 777
diff changeset
11 :license: GPLv3, see COPYING for more details.
94f7585af8a1 fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents: 777
diff changeset
12 """
1206
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1183
diff changeset
13 # This program is free software: you can redistribute it and/or modify
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1183
diff changeset
14 # it under the terms of the GNU General Public License as published by
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1183
diff changeset
15 # the Free Software Foundation, either version 3 of the License, or
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1183
diff changeset
16 # (at your option) any later version.
947
99850ac883d1 Fixed whoosh daemon, for depracated walk method
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
17 #
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
18 # This program is distributed in the hope that it will be useful,
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
19 # but WITHOUT ANY WARRANTY; without even the implied warranty of
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
20 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
21 # GNU General Public License for more details.
947
99850ac883d1 Fixed whoosh daemon, for depracated walk method
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
22 #
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
23 # You should have received a copy of the GNU General Public License
1206
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1183
diff changeset
24 # along with this program. If not, see <http://www.gnu.org/licenses/>.
2641
cfcd981d6679 import with_statment to make daemon.py python 2.5 compatible
Indra Talip <indra.talip@gmail.com>
parents: 2640
diff changeset
25 from __future__ import with_statement
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
26
1154
36fe593dfe4b simplified str2bool, and moved safe_unicode out of helpers since it was not html specific function
Marcin Kuzminski <marcin@python-works.com>
parents: 1036
diff changeset
27 import os
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
28 import sys
1154
36fe593dfe4b simplified str2bool, and moved safe_unicode out of helpers since it was not html specific function
Marcin Kuzminski <marcin@python-works.com>
parents: 1036
diff changeset
29 import logging
885
94f7585af8a1 fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents: 777
diff changeset
30 import traceback
1154
36fe593dfe4b simplified str2bool, and moved safe_unicode out of helpers since it was not html specific function
Marcin Kuzminski <marcin@python-works.com>
parents: 1036
diff changeset
31
36fe593dfe4b simplified str2bool, and moved safe_unicode out of helpers since it was not html specific function
Marcin Kuzminski <marcin@python-works.com>
parents: 1036
diff changeset
32 from shutil import rmtree
36fe593dfe4b simplified str2bool, and moved safe_unicode out of helpers since it was not html specific function
Marcin Kuzminski <marcin@python-works.com>
parents: 1036
diff changeset
33 from time import mktime
36fe593dfe4b simplified str2bool, and moved safe_unicode out of helpers since it was not html specific function
Marcin Kuzminski <marcin@python-works.com>
parents: 1036
diff changeset
34
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
35 from os.path import dirname as dn
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
36 from os.path import join as jn
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
37
547
1e757ac98988 renamed project to rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 497
diff changeset
38 #to get the rhodecode import
411
9b67cebe6609 some fixes to whoosh indexer daemon
Marcin Kuzminski <marcin@python-works.com>
parents: 407
diff changeset
39 project_path = dn(dn(dn(dn(os.path.realpath(__file__)))))
9b67cebe6609 some fixes to whoosh indexer daemon
Marcin Kuzminski <marcin@python-works.com>
parents: 407
diff changeset
40 sys.path.append(project_path)
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
41
2109
8ecfed1d8f8b utils/conf
Marcin Kuzminski <marcin@python-works.com>
parents: 2101
diff changeset
42 from rhodecode.config.conf import INDEX_EXTENSIONS
691
7486da5f0628 Refactor codes for scm model
Marcin Kuzminski <marcin@python-works.com>
parents: 683
diff changeset
43 from rhodecode.model.scm import ScmModel
3016
b3c8a3a5ce5f fixed issues some people had with whoosh indexers and unicode decode
Marcin Kuzminski <marcin@python-works.com>
parents: 2841
diff changeset
44 from rhodecode.lib.utils2 import safe_unicode, safe_str
2648
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
45 from rhodecode.lib.indexers import SCHEMA, IDX_NAME, CHGSETS_SCHEMA, \
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
46 CHGSET_IDX_NAME
411
9b67cebe6609 some fixes to whoosh indexer daemon
Marcin Kuzminski <marcin@python-works.com>
parents: 407
diff changeset
47
2007
324ac367a4da Added VCS into rhodecode core for faster and easier deployments of new versions
Marcin Kuzminski <marcin@python-works.com>
parents: 1995
diff changeset
48 from rhodecode.lib.vcs.exceptions import ChangesetError, RepositoryError, \
1711
b369bec5d468 fixes issue with whoosh reindexing files that were removed or renamed
Marcin Kuzminski <marcin@python-works.com>
parents: 1451
diff changeset
49 NodeDoesNotExistError
560
3072935bdeed rewrote whoosh indexing to run internal repository.walk() instead of filesystem.
Marcin Kuzminski <marcin@python-works.com>
parents: 557
diff changeset
50
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
51 from whoosh.index import create_in, open_dir, exists_in
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
52 from whoosh.query import *
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
53 from whoosh.qparser import QueryParser
1154
36fe593dfe4b simplified str2bool, and moved safe_unicode out of helpers since it was not html specific function
Marcin Kuzminski <marcin@python-works.com>
parents: 1036
diff changeset
54
2101
df96adcbb1f7 code garden
Marcin Kuzminski <marcin@python-works.com>
parents: 2007
diff changeset
55 log = logging.getLogger('whoosh_indexer')
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
56
1995
b6c902d88472 bumbed whoosh to 2.3.X series
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
57
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
58 class WhooshIndexingDaemon(object):
560
3072935bdeed rewrote whoosh indexing to run internal repository.walk() instead of filesystem.
Marcin Kuzminski <marcin@python-works.com>
parents: 557
diff changeset
59 """
2373
1828eb7fa688 #469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents: 2372
diff changeset
60 Daemon for atomic indexing jobs
560
3072935bdeed rewrote whoosh indexing to run internal repository.walk() instead of filesystem.
Marcin Kuzminski <marcin@python-works.com>
parents: 557
diff changeset
61 """
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
62
1995
b6c902d88472 bumbed whoosh to 2.3.X series
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
63 def __init__(self, indexname=IDX_NAME, index_location=None,
2373
1828eb7fa688 #469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents: 2372
diff changeset
64 repo_location=None, sa=None, repo_list=None,
1828eb7fa688 #469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents: 2372
diff changeset
65 repo_update_list=None):
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
66 self.indexname = indexname
631
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
67
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
68 self.index_location = index_location
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
69 if not index_location:
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
70 raise Exception('You have to provide index location')
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
71
411
9b67cebe6609 some fixes to whoosh indexer daemon
Marcin Kuzminski <marcin@python-works.com>
parents: 407
diff changeset
72 self.repo_location = repo_location
631
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
73 if not repo_location:
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
74 raise Exception('You have to provide repositories location')
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
75
1036
405b80e4ccd5 Major refactoring, removed when possible calls to app globals.
Marcin Kuzminski <marcin@python-works.com>
parents: 947
diff changeset
76 self.repo_paths = ScmModel(sa).repo_scan(self.repo_location)
894
1fed3c9161bb fixes #90 + docs update
Marcin Kuzminski <marcin@python-works.com>
parents: 886
diff changeset
77
2373
1828eb7fa688 #469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents: 2372
diff changeset
78 #filter repo list
894
1fed3c9161bb fixes #90 + docs update
Marcin Kuzminski <marcin@python-works.com>
parents: 886
diff changeset
79 if repo_list:
2841
2fa3c09f63e0 fixed problems with re-indexing non-ascii names of repositories
Marcin Kuzminski <marcin@python-works.com>
parents: 2840
diff changeset
80 #Fix non-ascii repo names to unicode
2fa3c09f63e0 fixed problems with re-indexing non-ascii names of repositories
Marcin Kuzminski <marcin@python-works.com>
parents: 2840
diff changeset
81 repo_list = map(safe_unicode, repo_list)
2373
1828eb7fa688 #469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents: 2372
diff changeset
82 self.filtered_repo_paths = {}
894
1fed3c9161bb fixes #90 + docs update
Marcin Kuzminski <marcin@python-works.com>
parents: 886
diff changeset
83 for repo_name, repo in self.repo_paths.items():
1fed3c9161bb fixes #90 + docs update
Marcin Kuzminski <marcin@python-works.com>
parents: 886
diff changeset
84 if repo_name in repo_list:
2373
1828eb7fa688 #469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents: 2372
diff changeset
85 self.filtered_repo_paths[repo_name] = repo
1828eb7fa688 #469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents: 2372
diff changeset
86
1828eb7fa688 #469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents: 2372
diff changeset
87 self.repo_paths = self.filtered_repo_paths
894
1fed3c9161bb fixes #90 + docs update
Marcin Kuzminski <marcin@python-works.com>
parents: 886
diff changeset
88
2373
1828eb7fa688 #469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents: 2372
diff changeset
89 #filter update repo list
1828eb7fa688 #469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents: 2372
diff changeset
90 self.filtered_repo_update_paths = {}
1828eb7fa688 #469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents: 2372
diff changeset
91 if repo_update_list:
1828eb7fa688 #469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents: 2372
diff changeset
92 self.filtered_repo_update_paths = {}
1828eb7fa688 #469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents: 2372
diff changeset
93 for repo_name, repo in self.repo_paths.items():
1828eb7fa688 #469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents: 2372
diff changeset
94 if repo_name in repo_update_list:
1828eb7fa688 #469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents: 2372
diff changeset
95 self.filtered_repo_update_paths[repo_name] = repo
1828eb7fa688 #469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents: 2372
diff changeset
96 self.repo_paths = self.filtered_repo_update_paths
894
1fed3c9161bb fixes #90 + docs update
Marcin Kuzminski <marcin@python-works.com>
parents: 886
diff changeset
97
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
98 self.initial = True
631
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
99 if not os.path.isdir(self.index_location):
763
0dad296d2a57 extended trending languages to more entries, implemented new faster and "fancy"
Marcin Kuzminski <marcin@python-works.com>
parents: 691
diff changeset
100 os.makedirs(self.index_location)
465
e01a85f9fc90 fixed initial whoosh indexer. Build full index on first run even with incremental flag
Marcin Kuzminski <marcin@python-works.com>
parents: 452
diff changeset
101 log.info('Cannot run incremental index since it does not'
e01a85f9fc90 fixed initial whoosh indexer. Build full index on first run even with incremental flag
Marcin Kuzminski <marcin@python-works.com>
parents: 452
diff changeset
102 ' yet exist running full build')
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
103 elif not exists_in(self.index_location, IDX_NAME):
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
104 log.info('Running full index build as the file content'
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
105 ' index does not exist')
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
106 elif not exists_in(self.index_location, CHGSET_IDX_NAME):
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
107 log.info('Running full index build as the changeset'
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
108 ' index does not exist')
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
109 else:
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
110 self.initial = False
631
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
111
561
5f3b967d9d10 fixed reindexing, and made some optimizations to reuse repo instances from repo scann list.
Marcin Kuzminski <marcin@python-works.com>
parents: 560
diff changeset
112 def get_paths(self, repo):
2101
df96adcbb1f7 code garden
Marcin Kuzminski <marcin@python-works.com>
parents: 2007
diff changeset
113 """
df96adcbb1f7 code garden
Marcin Kuzminski <marcin@python-works.com>
parents: 2007
diff changeset
114 recursive walk in root dir and return a set of all path in that dir
560
3072935bdeed rewrote whoosh indexing to run internal repository.walk() instead of filesystem.
Marcin Kuzminski <marcin@python-works.com>
parents: 557
diff changeset
115 based on repository walk function
3072935bdeed rewrote whoosh indexing to run internal repository.walk() instead of filesystem.
Marcin Kuzminski <marcin@python-works.com>
parents: 557
diff changeset
116 """
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
117 index_paths_ = set()
567
80dc0a23edf7 fixed whoosh failure on new repository
Marcin Kuzminski <marcin@python-works.com>
parents: 561
diff changeset
118 try:
947
99850ac883d1 Fixed whoosh daemon, for depracated walk method
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
119 tip = repo.get_changeset('tip')
2648
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
120 for _topnode, _dirs, files in tip.walk('/'):
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
121 for f in files:
3016
b3c8a3a5ce5f fixed issues some people had with whoosh indexers and unicode decode
Marcin Kuzminski <marcin@python-works.com>
parents: 2841
diff changeset
122 index_paths_.add(jn(safe_str(repo.path), safe_str(f.path)))
631
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
123
2648
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
124 except RepositoryError:
885
94f7585af8a1 fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents: 777
diff changeset
125 log.debug(traceback.format_exc())
567
80dc0a23edf7 fixed whoosh failure on new repository
Marcin Kuzminski <marcin@python-works.com>
parents: 561
diff changeset
126 pass
631
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
127 return index_paths_
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
128
561
5f3b967d9d10 fixed reindexing, and made some optimizations to reuse repo instances from repo scann list.
Marcin Kuzminski <marcin@python-works.com>
parents: 560
diff changeset
129 def get_node(self, repo, path):
5f3b967d9d10 fixed reindexing, and made some optimizations to reuse repo instances from repo scann list.
Marcin Kuzminski <marcin@python-works.com>
parents: 560
diff changeset
130 n_path = path[len(repo.path) + 1:]
5f3b967d9d10 fixed reindexing, and made some optimizations to reuse repo instances from repo scann list.
Marcin Kuzminski <marcin@python-works.com>
parents: 560
diff changeset
131 node = repo.get_changeset().get_node(n_path)
5f3b967d9d10 fixed reindexing, and made some optimizations to reuse repo instances from repo scann list.
Marcin Kuzminski <marcin@python-works.com>
parents: 560
diff changeset
132 return node
631
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
133
561
5f3b967d9d10 fixed reindexing, and made some optimizations to reuse repo instances from repo scann list.
Marcin Kuzminski <marcin@python-works.com>
parents: 560
diff changeset
134 def get_node_mtime(self, node):
5f3b967d9d10 fixed reindexing, and made some optimizations to reuse repo instances from repo scann list.
Marcin Kuzminski <marcin@python-works.com>
parents: 560
diff changeset
135 return mktime(node.last_changeset.date.timetuple())
631
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
136
1171
2ab211e0aecd changes for #56
Marcin Kuzminski <marcin@python-works.com>
parents: 1154
diff changeset
137 def add_doc(self, writer, path, repo, repo_name):
2101
df96adcbb1f7 code garden
Marcin Kuzminski <marcin@python-works.com>
parents: 2007
diff changeset
138 """
df96adcbb1f7 code garden
Marcin Kuzminski <marcin@python-works.com>
parents: 2007
diff changeset
139 Adding doc to writer this function itself fetches data from
df96adcbb1f7 code garden
Marcin Kuzminski <marcin@python-works.com>
parents: 2007
diff changeset
140 the instance of vcs backend
df96adcbb1f7 code garden
Marcin Kuzminski <marcin@python-works.com>
parents: 2007
diff changeset
141 """
df96adcbb1f7 code garden
Marcin Kuzminski <marcin@python-works.com>
parents: 2007
diff changeset
142
561
5f3b967d9d10 fixed reindexing, and made some optimizations to reuse repo instances from repo scann list.
Marcin Kuzminski <marcin@python-works.com>
parents: 560
diff changeset
143 node = self.get_node(repo, path)
2109
8ecfed1d8f8b utils/conf
Marcin Kuzminski <marcin@python-works.com>
parents: 2101
diff changeset
144 indexed = indexed_w_content = 0
2101
df96adcbb1f7 code garden
Marcin Kuzminski <marcin@python-works.com>
parents: 2007
diff changeset
145 # we just index the content of chosen files, and skip binary files
886
0736230c7d91 #92 removed content of binary files for whoosh indexer
Marcin Kuzminski <marcin@python-works.com>
parents: 885
diff changeset
146 if node.extension in INDEX_EXTENSIONS and not node.is_binary:
560
3072935bdeed rewrote whoosh indexing to run internal repository.walk() instead of filesystem.
Marcin Kuzminski <marcin@python-works.com>
parents: 557
diff changeset
147 u_content = node.content
885
94f7585af8a1 fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents: 777
diff changeset
148 if not isinstance(u_content, unicode):
94f7585af8a1 fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents: 777
diff changeset
149 log.warning(' >> %s Could not get this content as unicode '
2101
df96adcbb1f7 code garden
Marcin Kuzminski <marcin@python-works.com>
parents: 2007
diff changeset
150 'replacing with empty content' % path)
885
94f7585af8a1 fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents: 777
diff changeset
151 u_content = u''
94f7585af8a1 fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents: 777
diff changeset
152 else:
94f7585af8a1 fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents: 777
diff changeset
153 log.debug(' >> %s [WITH CONTENT]' % path)
2109
8ecfed1d8f8b utils/conf
Marcin Kuzminski <marcin@python-works.com>
parents: 2101
diff changeset
154 indexed_w_content += 1
885
94f7585af8a1 fixes to #92, updated changelog
Marcin Kuzminski <marcin@python-works.com>
parents: 777
diff changeset
155
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
156 else:
436
28f19fa562df updated config files,
Marcin Kuzminski <marcin@python-works.com>
parents: 411
diff changeset
157 log.debug(' >> %s' % path)
2101
df96adcbb1f7 code garden
Marcin Kuzminski <marcin@python-works.com>
parents: 2007
diff changeset
158 # just index file name without it's content
436
28f19fa562df updated config files,
Marcin Kuzminski <marcin@python-works.com>
parents: 411
diff changeset
159 u_content = u''
2109
8ecfed1d8f8b utils/conf
Marcin Kuzminski <marcin@python-works.com>
parents: 2101
diff changeset
160 indexed += 1
631
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
161
2388
a0ef98f2520b #453 added ID field in whoosh SCHEMA that solves the issue of reindexing modified files
Marcin Kuzminski <marcin@python-works.com>
parents: 2373
diff changeset
162 p = safe_unicode(path)
2101
df96adcbb1f7 code garden
Marcin Kuzminski <marcin@python-works.com>
parents: 2007
diff changeset
163 writer.add_document(
2388
a0ef98f2520b #453 added ID field in whoosh SCHEMA that solves the issue of reindexing modified files
Marcin Kuzminski <marcin@python-works.com>
parents: 2373
diff changeset
164 fileid=p,
2101
df96adcbb1f7 code garden
Marcin Kuzminski <marcin@python-works.com>
parents: 2007
diff changeset
165 owner=unicode(repo.contact),
df96adcbb1f7 code garden
Marcin Kuzminski <marcin@python-works.com>
parents: 2007
diff changeset
166 repository=safe_unicode(repo_name),
2388
a0ef98f2520b #453 added ID field in whoosh SCHEMA that solves the issue of reindexing modified files
Marcin Kuzminski <marcin@python-works.com>
parents: 2373
diff changeset
167 path=p,
2101
df96adcbb1f7 code garden
Marcin Kuzminski <marcin@python-works.com>
parents: 2007
diff changeset
168 content=u_content,
df96adcbb1f7 code garden
Marcin Kuzminski <marcin@python-works.com>
parents: 2007
diff changeset
169 modtime=self.get_node_mtime(node),
df96adcbb1f7 code garden
Marcin Kuzminski <marcin@python-works.com>
parents: 2007
diff changeset
170 extension=node.extension
df96adcbb1f7 code garden
Marcin Kuzminski <marcin@python-works.com>
parents: 2007
diff changeset
171 )
2109
8ecfed1d8f8b utils/conf
Marcin Kuzminski <marcin@python-works.com>
parents: 2101
diff changeset
172 return indexed, indexed_w_content
631
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
173
2643
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
174 def index_changesets(self, writer, repo_name, repo, start_rev=None):
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
175 """
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
176 Add all changeset in the vcs repo starting at start_rev
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
177 to the index writer
2643
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
178
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
179 :param writer: the whoosh index writer to add to
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
180 :param repo_name: name of the repository from whence the
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
181 changeset originates including the repository group
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
182 :param repo: the vcs repository instance to index changesets for,
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
183 the presumption is the repo has changesets to index
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
184 :param start_rev=None: the full sha id to start indexing from
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
185 if start_rev is None then index from the first changeset in
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
186 the repo
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
187 """
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
188
2643
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
189 if start_rev is None:
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
190 start_rev = repo[0].raw_id
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
191
2648
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
192 log.debug('indexing changesets in %s starting at rev: %s' %
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
193 (repo_name, start_rev))
631
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
194
2648
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
195 indexed = 0
2643
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
196 for cs in repo.get_changesets(start=start_rev):
2648
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
197 log.debug(' >> %s' % cs)
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
198 writer.add_document(
2642
88b0e82bcba4 rename changeset index key to match raw_id rather than path for greater consistency
Indra Talip <indra.talip@gmail.com>
parents: 2641
diff changeset
199 raw_id=unicode(cs.raw_id),
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
200 owner=unicode(repo.contact),
2693
66c778b8cb54 Extended commit search schema with date of commit
Marcin Kuzminski <marcin@python-works.com>
parents: 2648
diff changeset
201 date=cs._timestamp,
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
202 repository=safe_unicode(repo_name),
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
203 author=cs.author,
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
204 message=cs.message,
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
205 last=cs.last,
2763
81624c8a1035 #548 Fixed issue with non-ascii paths in whoosh indexer
Marcin Kuzminski <marcin@python-works.com>
parents: 2693
diff changeset
206 added=u' '.join([safe_unicode(node.path) for node in cs.added]).lower(),
81624c8a1035 #548 Fixed issue with non-ascii paths in whoosh indexer
Marcin Kuzminski <marcin@python-works.com>
parents: 2693
diff changeset
207 removed=u' '.join([safe_unicode(node.path) for node in cs.removed]).lower(),
81624c8a1035 #548 Fixed issue with non-ascii paths in whoosh indexer
Marcin Kuzminski <marcin@python-works.com>
parents: 2693
diff changeset
208 changed=u' '.join([safe_unicode(node.path) for node in cs.changed]).lower(),
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
209 parents=u' '.join([cs.raw_id for cs in cs.parents]),
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
210 )
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
211 indexed += 1
631
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
212
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
213 log.debug('indexed %d changesets for repo %s' % (indexed, repo_name))
2648
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
214 return indexed
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
215
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
216 def index_files(self, file_idx_writer, repo_name, repo):
2648
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
217 """
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
218 Index files for given repo_name
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
219
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
220 :param file_idx_writer: the whoosh index writer to add to
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
221 :param repo_name: name of the repository we're indexing
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
222 :param repo: instance of vcs repo
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
223 """
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
224 i_cnt = iwc_cnt = 0
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
225 log.debug('building index for [%s]' % repo.path)
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
226 for idx_path in self.get_paths(repo):
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
227 i, iwc = self.add_doc(file_idx_writer, idx_path, repo, repo_name)
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
228 i_cnt += i
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
229 iwc_cnt += iwc
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
230
2648
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
231 log.debug('added %s files %s with content for repo %s' %
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
232 (i_cnt + iwc_cnt, iwc_cnt, repo.path))
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
233 return i_cnt, iwc_cnt
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
234
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
235 def update_changeset_index(self):
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
236 idx = open_dir(self.index_location, indexname=CHGSET_IDX_NAME)
2569
b98fd6fc67f9 Little better logging in whoosh indexer
Marcin Kuzminski <marcin@python-works.com>
parents: 2388
diff changeset
237
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
238 with idx.searcher() as searcher:
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
239 writer = idx.writer()
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
240 writer_is_dirty = False
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
241 try:
2648
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
242 indexed_total = 0
2839
c0ddc86b4654 Fix possible exception about repo_name not defined, on whoosh indexer when using index-only option
Marcin Kuzminski <marcin@python-works.com>
parents: 2763
diff changeset
243 repo_name = None
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
244 for repo_name, repo in self.repo_paths.items():
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
245 # skip indexing if there aren't any revs in the repo
2643
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
246 num_of_revs = len(repo)
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
247 if num_of_revs < 1:
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
248 continue
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
249
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
250 qp = QueryParser('repository', schema=CHGSETS_SCHEMA)
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
251 q = qp.parse(u"last:t AND %s" % repo_name)
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
252
2643
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
253 results = searcher.search(q)
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
254
2643
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
255 # default to scanning the entire repo
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
256 last_rev = 0
2643
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
257 start_id = None
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
258
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
259 if len(results) > 0:
2643
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
260 # assuming that there is only one result, if not this
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
261 # may require a full re-index.
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
262 start_id = results[0]['raw_id']
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
263 last_rev = repo.get_changeset(revision=start_id).revision
631
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
264
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
265 # there are new changesets to index or a new repo to index
2643
2ad50c44b025 when indexing changesets use the raw_id to locate the point from
Indra Talip <indra.talip@gmail.com>
parents: 2642
diff changeset
266 if last_rev == 0 or num_of_revs > last_rev + 1:
2648
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
267 # delete the docs in the index for the previous
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
268 # last changeset(s)
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
269 for hit in results:
2648
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
270 q = qp.parse(u"last:t AND %s AND raw_id:%s" %
2642
88b0e82bcba4 rename changeset index key to match raw_id rather than path for greater consistency
Indra Talip <indra.talip@gmail.com>
parents: 2641
diff changeset
271 (repo_name, hit['raw_id']))
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
272 writer.delete_by_query(q)
631
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
273
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
274 # index from the previous last changeset + all new ones
2648
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
275 indexed_total += self.index_changesets(writer,
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
276 repo_name, repo, start_id)
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
277 writer_is_dirty = True
2648
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
278 log.debug('indexed %s changesets for repo %s' % (
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
279 indexed_total, repo_name)
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
280 )
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
281 finally:
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
282 if writer_is_dirty:
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
283 log.debug('>> COMMITING CHANGES TO CHANGESET INDEX<<')
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
284 writer.commit(merge=True)
2840
c7c5825299fe fixed logging messages on whoosh indexer
Marcin Kuzminski <marcin@python-works.com>
parents: 2839
diff changeset
285 log.debug('>>> FINISHED REBUILDING CHANGESET INDEX <<<')
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
286 else:
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
287 writer.cancel
2840
c7c5825299fe fixed logging messages on whoosh indexer
Marcin Kuzminski <marcin@python-works.com>
parents: 2839
diff changeset
288 log.debug('>> NOTHING TO COMMIT TO CHANGESET INDEX<<')
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
289
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
290 def update_file_index(self):
2373
1828eb7fa688 #469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents: 2372
diff changeset
291 log.debug((u'STARTING INCREMENTAL INDEXING UPDATE FOR EXTENSIONS %s '
1828eb7fa688 #469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents: 2372
diff changeset
292 'AND REPOS %s') % (INDEX_EXTENSIONS, self.repo_paths.keys()))
631
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
293
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
294 idx = open_dir(self.index_location, indexname=self.indexname)
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
295 # The set of all paths in the index
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
296 indexed_paths = set()
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
297 # The set of all paths we need to re-index
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
298 to_index = set()
631
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
299
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
300 writer = idx.writer()
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
301 writer_is_dirty = False
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
302 try:
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
303 with idx.reader() as reader:
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
304
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
305 # Loop over the stored fields in the index
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
306 for fields in reader.all_stored_fields():
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
307 indexed_path = fields['path']
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
308 indexed_repo_path = fields['repository']
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
309 indexed_paths.add(indexed_path)
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
310
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
311 if not indexed_repo_path in self.filtered_repo_update_paths:
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
312 continue
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
313
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
314 repo = self.repo_paths[indexed_repo_path]
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
315
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
316 try:
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
317 node = self.get_node(repo, indexed_path)
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
318 # Check if this file was changed since it was indexed
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
319 indexed_time = fields['modtime']
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
320 mtime = self.get_node_mtime(node)
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
321 if mtime > indexed_time:
2648
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
322 # The file has changed, delete it and add it to
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
323 # the list of files to reindex
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
324 log.debug(
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
325 'adding to reindex list %s mtime: %s vs %s' % (
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
326 indexed_path, mtime, indexed_time)
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
327 )
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
328 writer.delete_by_term('fileid', indexed_path)
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
329 writer_is_dirty = True
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
330
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
331 to_index.add(indexed_path)
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
332 except (ChangesetError, NodeDoesNotExistError):
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
333 # This file was deleted since it was indexed
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
334 log.debug('removing from index %s' % indexed_path)
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
335 writer.delete_by_term('path', indexed_path)
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
336 writer_is_dirty = True
631
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
337
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
338 # Loop over the files in the filesystem
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
339 # Assume we have a function that gathers the filenames of the
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
340 # documents to be indexed
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
341 ri_cnt_total = 0 # indexed
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
342 riwc_cnt_total = 0 # indexed with content
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
343 for repo_name, repo in self.repo_paths.items():
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
344 # skip indexing if there aren't any revisions
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
345 if len(repo) < 1:
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
346 continue
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
347 ri_cnt = 0 # indexed
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
348 riwc_cnt = 0 # indexed with content
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
349 for path in self.get_paths(repo):
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
350 path = safe_unicode(path)
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
351 if path in to_index or path not in indexed_paths:
631
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
352
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
353 # This is either a file that's changed, or a new file
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
354 # that wasn't indexed before. So index it!
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
355 i, iwc = self.add_doc(writer, path, repo, repo_name)
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
356 writer_is_dirty = True
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
357 log.debug('re indexing %s' % path)
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
358 ri_cnt += i
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
359 ri_cnt_total += 1
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
360 riwc_cnt += iwc
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
361 riwc_cnt_total += iwc
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
362 log.debug('added %s files %s with content for repo %s' % (
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
363 ri_cnt + riwc_cnt, riwc_cnt, repo.path)
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
364 )
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
365 log.debug('indexed %s files in total and %s with content' % (
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
366 ri_cnt_total, riwc_cnt_total)
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
367 )
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
368 finally:
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
369 if writer_is_dirty:
2840
c7c5825299fe fixed logging messages on whoosh indexer
Marcin Kuzminski <marcin@python-works.com>
parents: 2839
diff changeset
370 log.debug('>> COMMITING CHANGES TO FILE INDEX <<')
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
371 writer.commit(merge=True)
2840
c7c5825299fe fixed logging messages on whoosh indexer
Marcin Kuzminski <marcin@python-works.com>
parents: 2839
diff changeset
372 log.debug('>>> FINISHED REBUILDING FILE INDEX <<<')
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
373 else:
2840
c7c5825299fe fixed logging messages on whoosh indexer
Marcin Kuzminski <marcin@python-works.com>
parents: 2839
diff changeset
374 log.debug('>> NOTHING TO COMMIT TO FILE INDEX <<')
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
375 writer.cancel()
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
376
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
377 def build_indexes(self):
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
378 if os.path.exists(self.index_location):
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
379 log.debug('removing previous index')
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
380 rmtree(self.index_location)
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
381
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
382 if not os.path.exists(self.index_location):
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
383 os.mkdir(self.index_location)
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
384
2648
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
385 chgset_idx = create_in(self.index_location, CHGSETS_SCHEMA,
0911cf6940af little code cleanup
Marcin Kuzminski <marcin@python-works.com>
parents: 2643
diff changeset
386 indexname=CHGSET_IDX_NAME)
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
387 chgset_idx_writer = chgset_idx.writer()
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
388
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
389 file_idx = create_in(self.index_location, SCHEMA, indexname=IDX_NAME)
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
390 file_idx_writer = file_idx.writer()
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
391 log.debug('BUILDING INDEX FOR EXTENSIONS %s '
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
392 'AND REPOS %s' % (INDEX_EXTENSIONS, self.repo_paths.keys()))
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
393
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
394 for repo_name, repo in self.repo_paths.items():
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
395 # skip indexing if there aren't any revisions
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
396 if len(repo) < 1:
2373
1828eb7fa688 #469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents: 2372
diff changeset
397 continue
1828eb7fa688 #469 added --update-only option to whoosh to re-index only given list
Marcin Kuzminski <marcin@python-works.com>
parents: 2372
diff changeset
398
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
399 self.index_files(file_idx_writer, repo_name, repo)
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
400 self.index_changesets(chgset_idx_writer, repo_name, repo)
631
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
401
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
402 log.debug('>> COMMITING CHANGES <<')
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
403 file_idx_writer.commit(merge=True)
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
404 chgset_idx_writer.commit(merge=True)
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
405 log.debug('>>> FINISHED BUILDING INDEX <<<')
2388
a0ef98f2520b #453 added ID field in whoosh SCHEMA that solves the issue of reindexing modified files
Marcin Kuzminski <marcin@python-works.com>
parents: 2373
diff changeset
406
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
407 def update_indexes(self):
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
408 self.update_file_index()
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
409 self.update_changeset_index()
631
05528ad948c4 Hacking for git support,and new faster repo scan
Marcin Kuzminski <marcin@python-works.com>
parents: 629
diff changeset
410
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
411 def run(self, full_index=False):
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
412 """Run daemon"""
465
e01a85f9fc90 fixed initial whoosh indexer. Build full index on first run even with incremental flag
Marcin Kuzminski <marcin@python-works.com>
parents: 452
diff changeset
413 if full_index or self.initial:
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
414 self.build_indexes()
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
415 else:
2640
5f21a9dcb09d create an index for commit messages and the ability to search them and see results
Indra Talip <indra.talip@gmail.com>
parents: 2569
diff changeset
416 self.update_indexes()