Mercurial > kallithea

annotate rhodecode/controllers/search.py @ 1295:11548cc19c8f beta

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
fixes #183 decoding of paths in changes
author Marcin Kuzminski <marcin@python-works.com>
date Wed, 27 Apr 2011 20:16:03 +0200
parents 50e41777675d
children 9773b46e239f
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
861
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
1 # -*- coding: utf-8 -*-
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
2 """
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
3 rhodecode.controllers.search
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
5
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
6 Search controller for rhodecode
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 1045
diff changeset
7
861
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
8 :created_on: Aug 7, 2010
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
9 :author: marcink
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 1045
diff changeset
10 :copyright: (C) 2009-2011 Marcin Kuzminski <marcin@python-works.com>
861
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
11 :license: GPLv3, see COPYING for more details.
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
12 """
1206
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
13 # This program is free software: you can redistribute it and/or modify
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
14 # it under the terms of the GNU General Public License as published by
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
15 # the Free Software Foundation, either version 3 of the License, or
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
16 # (at your option) any later version.
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 1045
diff changeset
17 #
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
18 # This program is distributed in the hope that it will be useful,
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
19 # but WITHOUT ANY WARRANTY; without even the implied warranty of
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
20 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
21 # GNU General Public License for more details.
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 1045
diff changeset
22 #
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
23 # You should have received a copy of the GNU General Public License
1206
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
24 # along with this program. If not, see <http://www.gnu.org/licenses/>.
861
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
25 import logging
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
26 import traceback
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
27
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
28 from pylons.i18n.translation import _
1045
3fc9183e05dd another major codes rewrite:
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
29 from pylons import request, config, session, tmpl_context as c
861
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
30
547
1e757ac98988 renamed project to rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 525
diff changeset
31 from rhodecode.lib.auth import LoginRequired
1e757ac98988 renamed project to rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 525
diff changeset
32 from rhodecode.lib.base import BaseController, render
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
33 from rhodecode.lib.indexers import SCHEMA, IDX_NAME, ResultWrapper
861
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
34
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
35 from webhelpers.paginate import Page
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
36 from webhelpers.util import update_params
861
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
37
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
38 from whoosh.index import open_dir, EmptyIndexError
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
39 from whoosh.qparser import QueryParser, QueryParserError
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
40 from whoosh.query import Phrase
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
41
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
42 log = logging.getLogger(__name__)
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
43
1212
50e41777675d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
44
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
45 class SearchController(BaseController):
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
46
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
47 @LoginRequired()
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
48 def __before__(self):
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
49 super(SearchController, self).__before__()
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
50
525
87d80c84df09 added search in specific repository
Marcin Kuzminski <marcin@python-works.com>
parents: 478
diff changeset
51 def index(self, search_repo=None):
87d80c84df09 added search in specific repository
Marcin Kuzminski <marcin@python-works.com>
parents: 478
diff changeset
52 c.repo_name = search_repo
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
53 c.formated_results = []
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
54 c.runtime = ''
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
55 c.cur_query = request.GET.get('q', None)
556
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
56 c.cur_type = request.GET.get('type', 'source')
1212
50e41777675d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
57 c.cur_search = search_type = {'content': 'content',
50e41777675d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
58 'commit': 'content',
50e41777675d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
59 'path': 'path',
50e41777675d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
60 'repository': 'repository'}\
556
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
61 .get(c.cur_type, 'content')
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
62
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
63 if c.cur_query:
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
64 cur_query = c.cur_query.lower()
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
65
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
66 if c.cur_query:
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
67 p = int(request.params.get('page', 1))
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
68 highlight_items = set()
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
69 try:
1212
50e41777675d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
70 idx = open_dir(config['app_conf']['index_dir'],
50e41777675d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
71 indexname=IDX_NAME)
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
72 searcher = idx.searcher()
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
73
556
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
74 qp = QueryParser(search_type, schema=SCHEMA)
525
87d80c84df09 added search in specific repository
Marcin Kuzminski <marcin@python-works.com>
parents: 478
diff changeset
75 if c.repo_name:
87d80c84df09 added search in specific repository
Marcin Kuzminski <marcin@python-works.com>
parents: 478
diff changeset
76 cur_query = u'repository:%s %s' % (c.repo_name, cur_query)
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
77 try:
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
78 query = qp.parse(unicode(cur_query))
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
79
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
80 if isinstance(query, Phrase):
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
81 highlight_items.update(query.words)
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
82 else:
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
83 for i in query.all_terms():
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
84 if i[0] == 'content':
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
85 highlight_items.add(i[1])
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
86
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
87 matcher = query.matcher(searcher)
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
88
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
89 log.debug(query)
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
90 log.debug(highlight_items)
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
91 results = searcher.search(query)
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
92 res_ln = len(results)
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
93 c.runtime = '%s results (%.3f seconds)' \
556
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
94 % (res_ln, results.runtime)
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
95
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
96 def url_generator(**kw):
556
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
97 return update_params("?q=%s&type=%s" \
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
98 % (c.cur_query, c.cur_search), **kw)
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
99
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
100 c.formated_results = Page(
556
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
101 ResultWrapper(search_type, searcher, matcher,
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
102 highlight_items),
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
103 page=p, item_count=res_ln,
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
104 items_per_page=10, url=url_generator)
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
105
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
106 except QueryParserError:
410
9a7ae16ff53e fixes translations, style updates.
Marcin Kuzminski <marcin@python-works.com>
parents: 406
diff changeset
107 c.runtime = _('Invalid search query. Try quoting it.')
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
108 searcher.close()
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
109 except (EmptyIndexError, IOError):
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
110 log.error(traceback.format_exc())
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
111 log.error('Empty Index data')
861
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
112 c.runtime = _('There is no index to search in. '
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
113 'Please run whoosh indexer')
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
114
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
115 # Return a rendered template
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
116 return render('/search/search.html')