Mercurial > kallithea

annotate rhodecode/controllers/search.py @ 1345:3bce31f026b8 beta

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
#47 implemented Adding of new repo_groups+forms+validators. Fixed sorting of repo groups by main names in multiple locations. Removed some unneeded calls to self.sa for exchange to .query() methods. Added new db unique key for Group
author Marcin Kuzminski <marcin@python-works.com>
date Mon, 23 May 2011 02:22:00 +0200
parents 9773b46e239f
children bf263968da47 89efedac4e6c
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
861
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
1 # -*- coding: utf-8 -*-
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
2 """
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
3 rhodecode.controllers.search
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
5
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
6 Search controller for rhodecode
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 1045
diff changeset
7
861
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
8 :created_on: Aug 7, 2010
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
9 :author: marcink
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 1045
diff changeset
10 :copyright: (C) 2009-2011 Marcin Kuzminski <marcin@python-works.com>
861
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
11 :license: GPLv3, see COPYING for more details.
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
12 """
1206
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
13 # This program is free software: you can redistribute it and/or modify
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
14 # it under the terms of the GNU General Public License as published by
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
15 # the Free Software Foundation, either version 3 of the License, or
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
16 # (at your option) any later version.
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 1045
diff changeset
17 #
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
18 # This program is distributed in the hope that it will be useful,
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
19 # but WITHOUT ANY WARRANTY; without even the implied warranty of
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
20 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
21 # GNU General Public License for more details.
1203
6832ef664673 source code cleanup: remove trailing white space, normalize file endings
Marcin Kuzminski <marcin@python-works.com>
parents: 1045
diff changeset
22 #
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
23 # You should have received a copy of the GNU General Public License
1206
a671db5bdd58 fixed license issue #149
Marcin Kuzminski <marcin@python-works.com>
parents: 1203
diff changeset
24 # along with this program. If not, see <http://www.gnu.org/licenses/>.
861
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
25 import logging
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
26 import traceback
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
27
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
28 from pylons.i18n.translation import _
1045
3fc9183e05dd another major codes rewrite:
Marcin Kuzminski <marcin@python-works.com>
parents: 902
diff changeset
29 from pylons import request, config, session, tmpl_context as c
861
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
30
547
1e757ac98988 renamed project to rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 525
diff changeset
31 from rhodecode.lib.auth import LoginRequired
1e757ac98988 renamed project to rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 525
diff changeset
32 from rhodecode.lib.base import BaseController, render
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
33 from rhodecode.lib.indexers import SCHEMA, IDX_NAME, ResultWrapper
861
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
34
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
35 from webhelpers.paginate import Page
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
36 from webhelpers.util import update_params
861
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
37
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
38 from whoosh.index import open_dir, EmptyIndexError
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
39 from whoosh.qparser import QueryParser, QueryParserError
1303
9773b46e239f whoosh errors don't crash entire app now, just display an error and log exception instead
Marcin Kuzminski <marcin@python-works.com>
parents: 1212
diff changeset
40 from whoosh.query import Phrase, Wildcard, Term, Prefix
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
41
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
42 log = logging.getLogger(__name__)
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
43
1212
50e41777675d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
44
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
45 class SearchController(BaseController):
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
46
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
47 @LoginRequired()
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
48 def __before__(self):
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
49 super(SearchController, self).__before__()
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
50
525
87d80c84df09 added search in specific repository
Marcin Kuzminski <marcin@python-works.com>
parents: 478
diff changeset
51 def index(self, search_repo=None):
87d80c84df09 added search in specific repository
Marcin Kuzminski <marcin@python-works.com>
parents: 478
diff changeset
52 c.repo_name = search_repo
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
53 c.formated_results = []
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
54 c.runtime = ''
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
55 c.cur_query = request.GET.get('q', None)
556
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
56 c.cur_type = request.GET.get('type', 'source')
1212
50e41777675d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
57 c.cur_search = search_type = {'content': 'content',
50e41777675d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
58 'commit': 'content',
50e41777675d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
59 'path': 'path',
50e41777675d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
60 'repository': 'repository'}\
556
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
61 .get(c.cur_type, 'content')
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
62
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
63 if c.cur_query:
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
64 cur_query = c.cur_query.lower()
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
65
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
66 if c.cur_query:
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
67 p = int(request.params.get('page', 1))
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
68 highlight_items = set()
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
69 try:
1212
50e41777675d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
70 idx = open_dir(config['app_conf']['index_dir'],
50e41777675d pep8ify
Marcin Kuzminski <marcin@python-works.com>
parents: 1206
diff changeset
71 indexname=IDX_NAME)
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
72 searcher = idx.searcher()
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
73
556
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
74 qp = QueryParser(search_type, schema=SCHEMA)
525
87d80c84df09 added search in specific repository
Marcin Kuzminski <marcin@python-works.com>
parents: 478
diff changeset
75 if c.repo_name:
87d80c84df09 added search in specific repository
Marcin Kuzminski <marcin@python-works.com>
parents: 478
diff changeset
76 cur_query = u'repository:%s %s' % (c.repo_name, cur_query)
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
77 try:
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
78 query = qp.parse(unicode(cur_query))
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
79
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
80 if isinstance(query, Phrase):
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
81 highlight_items.update(query.words)
1303
9773b46e239f whoosh errors don't crash entire app now, just display an error and log exception instead
Marcin Kuzminski <marcin@python-works.com>
parents: 1212
diff changeset
82 elif isinstance(query, Prefix):
9773b46e239f whoosh errors don't crash entire app now, just display an error and log exception instead
Marcin Kuzminski <marcin@python-works.com>
parents: 1212
diff changeset
83 highlight_items.add(query.text)
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
84 else:
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
85 for i in query.all_terms():
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
86 if i[0] == 'content':
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
87 highlight_items.add(i[1])
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
88
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
89 matcher = query.matcher(searcher)
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
90
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
91 log.debug(query)
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
92 log.debug(highlight_items)
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
93 results = searcher.search(query)
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
94 res_ln = len(results)
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
95 c.runtime = '%s results (%.3f seconds)' \
556
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
96 % (res_ln, results.runtime)
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
97
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
98 def url_generator(**kw):
556
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
99 return update_params("?q=%s&type=%s" \
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
100 % (c.cur_query, c.cur_search), **kw)
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
101
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
102 c.formated_results = Page(
556
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
103 ResultWrapper(search_type, searcher, matcher,
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
104 highlight_items),
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
105 page=p, item_count=res_ln,
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
106 items_per_page=10, url=url_generator)
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
107
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
108 except QueryParserError:
410
9a7ae16ff53e fixes translations, style updates.
Marcin Kuzminski <marcin@python-works.com>
parents: 406
diff changeset
109 c.runtime = _('Invalid search query. Try quoting it.')
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
110 searcher.close()
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
111 except (EmptyIndexError, IOError):
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
112 log.error(traceback.format_exc())
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
113 log.error('Empty Index data')
861
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
114 c.runtime = _('There is no index to search in. '
fd2ea6ceadc8 updated docs on every controller
Marcin Kuzminski <marcin@python-works.com>
parents: 688
diff changeset
115 'Please run whoosh indexer')
1303
9773b46e239f whoosh errors don't crash entire app now, just display an error and log exception instead
Marcin Kuzminski <marcin@python-works.com>
parents: 1212
diff changeset
116 except (Exception):
9773b46e239f whoosh errors don't crash entire app now, just display an error and log exception instead
Marcin Kuzminski <marcin@python-works.com>
parents: 1212
diff changeset
117 log.error(traceback.format_exc())
9773b46e239f whoosh errors don't crash entire app now, just display an error and log exception instead
Marcin Kuzminski <marcin@python-works.com>
parents: 1212
diff changeset
118 c.runtime = _('An error occurred during this search operation')
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
119
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
120 # Return a rendered template
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
121 return render('/search/search.html')