Mercurial > kallithea

annotate rhodecode/controllers/search.py @ 763:0dad296d2a57 beta

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
extended trending languages to more entries, implemented new faster and "fancy" method of extensions recognition. Fixed small bug in indexer when data dir was empty
author Marcin Kuzminski <marcin@python-works.com>
date Fri, 26 Nov 2010 02:25:39 +0100
parents 8acbfa837180
children fd2ea6ceadc8
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
1 #!/usr/bin/env python
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
2 # encoding: utf-8
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
3 # search controller for pylons
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
4 # Copyright (C) 2009-2010 Marcin Kuzminski <marcin@python-works.com>
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
5 #
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
6 # This program is free software; you can redistribute it and/or
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
7 # modify it under the terms of the GNU General Public License
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
8 # as published by the Free Software Foundation; version 2
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
9 # of the License or (at your opinion) any later version of the license.
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
10 #
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
11 # This program is distributed in the hope that it will be useful,
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
12 # but WITHOUT ANY WARRANTY; without even the implied warranty of
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
13 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
14 # GNU General Public License for more details.
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
15 #
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
16 # You should have received a copy of the GNU General Public License
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
17 # along with this program; if not, write to the Free Software
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
18 # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
19 # MA 02110-1301, USA.
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
20 """
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
21 Created on Aug 7, 2010
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
22 search controller for pylons
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
23 @author: marcink
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
24 """
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
25 from pylons import request, response, config, session, tmpl_context as c, url
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
26 from pylons.controllers.util import abort, redirect
547
1e757ac98988 renamed project to rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 525
diff changeset
27 from rhodecode.lib.auth import LoginRequired
1e757ac98988 renamed project to rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 525
diff changeset
28 from rhodecode.lib.base import BaseController, render
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
29 from rhodecode.lib.indexers import SCHEMA, IDX_NAME, ResultWrapper
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
30 from webhelpers.paginate import Page
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
31 from webhelpers.util import update_params
410
9a7ae16ff53e fixes translations, style updates.
Marcin Kuzminski <marcin@python-works.com>
parents: 406
diff changeset
32 from pylons.i18n.translation import _
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
33 from whoosh.index import open_dir, EmptyIndexError
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
34 from whoosh.qparser import QueryParser, QueryParserError
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
35 from whoosh.query import Phrase
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
36 import logging
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
37 import traceback
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
38
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
39 log = logging.getLogger(__name__)
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
40
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
41 class SearchController(BaseController):
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
42
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
43 @LoginRequired()
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
44 def __before__(self):
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
45 super(SearchController, self).__before__()
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
46
525
87d80c84df09 added search in specific repository
Marcin Kuzminski <marcin@python-works.com>
parents: 478
diff changeset
47 def index(self, search_repo=None):
87d80c84df09 added search in specific repository
Marcin Kuzminski <marcin@python-works.com>
parents: 478
diff changeset
48 c.repo_name = search_repo
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
49 c.formated_results = []
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
50 c.runtime = ''
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
51 c.cur_query = request.GET.get('q', None)
556
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
52 c.cur_type = request.GET.get('type', 'source')
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
53 c.cur_search = search_type = {'content':'content',
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
54 'commit':'content',
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
55 'path':'path',
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
56 'repository':'repository'}\
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
57 .get(c.cur_type, 'content')
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
58
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
59
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
60 if c.cur_query:
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
61 cur_query = c.cur_query.lower()
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
62
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
63 if c.cur_query:
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
64 p = int(request.params.get('page', 1))
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
65 highlight_items = set()
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
66 try:
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
67 idx = open_dir(config['app_conf']['index_dir']
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
68 , indexname=IDX_NAME)
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
69 searcher = idx.searcher()
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
70
556
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
71 qp = QueryParser(search_type, schema=SCHEMA)
525
87d80c84df09 added search in specific repository
Marcin Kuzminski <marcin@python-works.com>
parents: 478
diff changeset
72 if c.repo_name:
87d80c84df09 added search in specific repository
Marcin Kuzminski <marcin@python-works.com>
parents: 478
diff changeset
73 cur_query = u'repository:%s %s' % (c.repo_name, cur_query)
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
74 try:
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
75 query = qp.parse(unicode(cur_query))
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
76
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
77 if isinstance(query, Phrase):
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
78 highlight_items.update(query.words)
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
79 else:
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
80 for i in query.all_terms():
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
81 if i[0] == 'content':
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
82 highlight_items.add(i[1])
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
83
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
84 matcher = query.matcher(searcher)
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
85
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
86 log.debug(query)
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
87 log.debug(highlight_items)
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
88 results = searcher.search(query)
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
89 res_ln = len(results)
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
90 c.runtime = '%s results (%.3f seconds)' \
556
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
91 % (res_ln, results.runtime)
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
92
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
93 def url_generator(**kw):
556
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
94 return update_params("?q=%s&type=%s" \
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
95 % (c.cur_query, c.cur_search), **kw)
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
96
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
97 c.formated_results = Page(
556
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
98 ResultWrapper(search_type, searcher, matcher,
65b2f150beb7 Added searching for file names within the repository in rhodecode
Marcin Kuzminski <marcin@python-works.com>
parents: 547
diff changeset
99 highlight_items),
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
100 page=p, item_count=res_ln,
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
101 items_per_page=10, url=url_generator)
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
102
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
103
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
104 except QueryParserError:
410
9a7ae16ff53e fixes translations, style updates.
Marcin Kuzminski <marcin@python-works.com>
parents: 406
diff changeset
105 c.runtime = _('Invalid search query. Try quoting it.')
478
7010af6efde5 Reimplemented searching for speed on large files and added paging for search results
Marcin Kuzminski <marcin@python-works.com>
parents: 410
diff changeset
106 searcher.close()
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
107 except (EmptyIndexError, IOError):
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
108 log.error(traceback.format_exc())
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
109 log.error('Empty Index data')
410
9a7ae16ff53e fixes translations, style updates.
Marcin Kuzminski <marcin@python-works.com>
parents: 406
diff changeset
110 c.runtime = _('There is no index to search in. Please run whoosh indexer')
688
8acbfa837180 Tests rewrite for 1.2 added some globals configs to make tests easier.
Marcin Kuzminski <marcin@python-works.com>
parents: 556
diff changeset
111
406
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
112 # Return a rendered template
b153a51b1d3b Implemented search using whoosh. Still as experimental option.
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
113 return render('/search/search.html')