Mercurial > kallithea

annotate rhodecode/controllers/search.py @ 2031:82a88013a3fd

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
merge 1.3 into stable
author Marcin Kuzminski <marcin@python-works.com>
date Sun, 26 Feb 2012 17:25:09 +0200
parents bf263968da47 b6c902d88472
children 63e58ef80ef1
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
861
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
1 # -*- coding: utf-8 -*-
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
2 """
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
3 rhodecode.controllers.search
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
5
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
6 Search controller for rhodecode
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 1045
diff changeset
7
861
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
8 :created_on: Aug 7, 2010
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
9 :author: marcink
1824
89efedac4e6c 2012 copyrights
Marcin Kuzminski <marcin@python-works.com>
parents: 1303
diff changeset
10 :copyright: (C) 2010-2012 Marcin Kuzminski <marcin@python-works.com>
861
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
11 :license: GPLv3, see COPYING for more details.
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
12 """
1206
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
13 # This program is free software: you can redistribute it and/or modify
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
14 # it under the terms of the GNU General Public License as published by
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
15 # the Free Software Foundation, either version 3 of the License, or
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
16 # (at your option) any later version.
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 1045
diff changeset
17 #
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
18 # This program is distributed in the hope that it will be useful,
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
19 # but WITHOUT ANY WARRANTY; without even the implied warranty of
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
20 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
21 # GNU General Public License for more details.
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 1045
diff changeset
22 #
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
23 # You should have received a copy of the GNU General Public License
1206
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
24 # along with this program. If not, see <http://www.gnu.org/licenses/>.
861
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
25 import logging
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
26 import traceback
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
27
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
28 from pylons.i18n.translation import _
1995
b6c902d88472 bumbed whoosh to 2.3.X series
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
29 from pylons import request, config, tmpl_context as c
861
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
30
547
1e757ac98988 renamed project to rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 525
diff changeset
31 from rhodecode.lib.auth import LoginRequired
1e757ac98988 renamed project to rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 525
diff changeset
32 from rhodecode.lib.base import BaseController, render
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
33 from rhodecode.lib.indexers import SCHEMA, IDX_NAME, ResultWrapper
861
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
34
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
35 from webhelpers.paginate import Page
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
36 from webhelpers.util import update_params
861
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
37
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
38 from whoosh.index import open_dir, EmptyIndexError
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
39 from whoosh.qparser import QueryParser, QueryParserError
1303
9773b46e239f whoosh errors don't crash entire app now, just display an error and log exception instead
Marcin Kuzminski <marcin@python-works.com>
parents: 1212
diff changeset
40 from whoosh.query import Phrase, Wildcard, Term, Prefix
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
41
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
42 log = logging.getLogger(__name__)
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
43
1212
50e41777675d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
44
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
45 class SearchController(BaseController):
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
46
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
47 @LoginRequired()
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
48 def __before__(self):
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
49 super(SearchController, self).__before__()
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
50
525
87d80c84df09 added search in specific repository
Marcin Kuzminski <marcin@python-works.com>
parents: 478
diff changeset
51 def index(self, search_repo=None):
87d80c84df09 added search in specific repository
Marcin Kuzminski <marcin@python-works.com>
parents: 478
diff changeset
52 c.repo_name = search_repo
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
53 c.formated_results = []
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
54 c.runtime = ''
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
55 c.cur_query = request.GET.get('q', None)
556
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
56 c.cur_type = request.GET.get('type', 'source')
1212
50e41777675d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
57 c.cur_search = search_type = {'content': 'content',
50e41777675d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
58 'commit': 'content',
50e41777675d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
59 'path': 'path',
50e41777675d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
60 'repository': 'repository'}\
556
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
61 .get(c.cur_type, 'content')
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
62
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
63 if c.cur_query:
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
64 cur_query = c.cur_query.lower()
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
65
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
66 if c.cur_query:
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
67 p = int(request.params.get('page', 1))
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
68 highlight_items = set()
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
69 try:
1212
50e41777675d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
70 idx = open_dir(config['app_conf']['index_dir'],
50e41777675d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
71 indexname=IDX_NAME)
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
72 searcher = idx.searcher()
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
73
556
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
74 qp = QueryParser(search_type, schema=SCHEMA)
525
87d80c84df09 added search in specific repository
Marcin Kuzminski <marcin@python-works.com>
parents: 478
diff changeset
75 if c.repo_name:
87d80c84df09 added search in specific repository
Marcin Kuzminski <marcin@python-works.com>
parents: 478
diff changeset
76 cur_query = u'repository:%s %s' % (c.repo_name, cur_query)
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
77 try:
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
78 query = qp.parse(unicode(cur_query))
1995
b6c902d88472 bumbed whoosh to 2.3.X series
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
79 # extract words for highlight
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
80 if isinstance(query, Phrase):
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
81 highlight_items.update(query.words)
1303
9773b46e239f whoosh errors don't crash entire app now, just display an error and log exception instead
Marcin Kuzminski <marcin@python-works.com>
parents: 1212
diff changeset
82 elif isinstance(query, Prefix):
9773b46e239f whoosh errors don't crash entire app now, just display an error and log exception instead
Marcin Kuzminski <marcin@python-works.com>
parents: 1212
diff changeset
83 highlight_items.add(query.text)
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
84 else:
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
85 for i in query.all_terms():
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
86 if i[0] == 'content':
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
87 highlight_items.add(i[1])
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
88
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
89 matcher = query.matcher(searcher)
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
90
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
91 log.debug(query)
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
92 log.debug(highlight_items)
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
93 results = searcher.search(query)
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
94 res_ln = len(results)
1995
b6c902d88472 bumbed whoosh to 2.3.X series
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
95 c.runtime = '%s results (%.3f seconds)' % (
b6c902d88472 bumbed whoosh to 2.3.X series
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
96 res_ln, results.runtime
b6c902d88472 bumbed whoosh to 2.3.X series
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
97 )
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
98
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
99 def url_generator(**kw):
556
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
100 return update_params("?q=%s&type=%s" \
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
101 % (c.cur_query, c.cur_search), **kw)
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
102
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
103 c.formated_results = Page(
1995
b6c902d88472 bumbed whoosh to 2.3.X series
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
104 ResultWrapper(search_type, searcher, matcher,
b6c902d88472 bumbed whoosh to 2.3.X series
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
105 highlight_items),
b6c902d88472 bumbed whoosh to 2.3.X series
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
106 page=p,
b6c902d88472 bumbed whoosh to 2.3.X series
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
107 item_count=res_ln,
b6c902d88472 bumbed whoosh to 2.3.X series
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
108 items_per_page=10,
b6c902d88472 bumbed whoosh to 2.3.X series
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
109 url=url_generator
b6c902d88472 bumbed whoosh to 2.3.X series
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
110 )
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
111
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
112 except QueryParserError:
410
9a7ae16ff53e fixes translations, style updates.
Marcin Kuzminski <marcin@python-works.com>
parents: 406
diff changeset
113 c.runtime = _('Invalid search query. Try quoting it.')
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
114 searcher.close()
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
115 except (EmptyIndexError, IOError):
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
116 log.error(traceback.format_exc())
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
117 log.error('Empty Index data')
861
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
118 c.runtime = _('There is no index to search in. '
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
119 'Please run whoosh indexer')
1303
9773b46e239f whoosh errors don't crash entire app now, just display an error and log exception instead
Marcin Kuzminski <marcin@python-works.com>
parents: 1212
diff changeset
120 except (Exception):
9773b46e239f whoosh errors don't crash entire app now, just display an error and log exception instead
Marcin Kuzminski <marcin@python-works.com>
parents: 1212
diff changeset
121 log.error(traceback.format_exc())
9773b46e239f whoosh errors don't crash entire app now, just display an error and log exception instead
Marcin Kuzminski <marcin@python-works.com>
parents: 1212
diff changeset
122 c.runtime = _('An error occurred during this search operation')
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
123
1995
b6c902d88472 bumbed whoosh to 2.3.X series
Marcin Kuzminski <marcin@python-works.com>
parents: 1824
diff changeset
124
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
125 # Return a rendered template
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
126 return render('/search/search.html')