Mercurial > kallithea

annotate rhodecode/tests/rhodecode_crawler.py @ 1818:cf51bbfb120e beta

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
auto white-space removal
author Marcin Kuzminski <marcin@python-works.com>
date Thu, 29 Dec 2011 07:35:51 +0200
parents 87ec80c280bb
children 89efedac4e6c
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1332
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
1 # -*- coding: utf-8 -*-
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
2 """
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
3 rhodecode.tests.test_crawer
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
5
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
6 Test for crawling a project for memory usage
1371
0ad5769fa78d Fixed desc on test_crawler
Marcin Kuzminski <marcin@python-works.com>
parents: 1341
diff changeset
7 This should be runned just as regular script together
0ad5769fa78d Fixed desc on test_crawler
Marcin Kuzminski <marcin@python-works.com>
parents: 1341
diff changeset
8 with a watch script that will show memory usage.
1818
cf51bbfb120e auto white-space removal
Marcin Kuzminski <marcin@python-works.com>
parents: 1514
diff changeset
9
1334
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
10 watch -n1 ./rhodecode/tests/mem_watch
1332
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
11
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
12 :created_on: Apr 21, 2010
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
13 :author: marcink
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
14 :copyright: (C) 2009-2011 Marcin Kuzminski <marcin@python-works.com>
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
15 :license: GPLv3, see COPYING for more details.
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
16 """
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
17 # This program is free software: you can redistribute it and/or modify
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
18 # it under the terms of the GNU General Public License as published by
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
19 # the Free Software Foundation, either version 3 of the License, or
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
20 # (at your option) any later version.
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
21 #
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
22 # This program is distributed in the hope that it will be useful,
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
23 # but WITHOUT ANY WARRANTY; without even the implied warranty of
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
24 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
25 # GNU General Public License for more details.
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
26 #
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
27 # You should have received a copy of the GNU General Public License
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
28 # along with this program. If not, see <http://www.gnu.org/licenses/>.
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
29
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
30
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
31 import cookielib
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
32 import urllib
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
33 import urllib2
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
34 import vcs
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
35 import time
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
36
1334
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
37 from os.path import join as jn
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
38
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
39
1332
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
40 BASE_URI = 'http://127.0.0.1:5000/%s'
1334
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
41 PROJECT = 'CPython'
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
42 PROJECT_PATH = jn('/', 'home', 'marcink', 'hg_repos')
1332
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
43
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
44
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
45 cj = cookielib.FileCookieJar('/tmp/rc_test_cookie.txt')
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
46 o = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
47 o.addheaders = [
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
48 ('User-agent', 'rhodecode-crawler'),
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
49 ('Accept-Language', 'en - us, en;q = 0.5')
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
50 ]
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
51
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
52 urllib2.install_opener(o)
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
53
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
54
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
55
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
56 def test_changelog_walk(pages=100):
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
57 total_time = 0
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
58 for i in range(1, pages):
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
59
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
60 page = '/'.join((PROJECT, 'changelog',))
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
61
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
62 full_uri = (BASE_URI % page) + '?' + urllib.urlencode({'page':i})
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
63 s = time.time()
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
64 f = o.open(full_uri)
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
65 size = len(f.read())
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
66 e = time.time() - s
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
67 total_time += e
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
68 print 'visited %s size:%s req:%s ms' % (full_uri, size, e)
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
69
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
70
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
71 print 'total_time', total_time
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
72 print 'average on req', total_time / float(pages)
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
73
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
74
1338
bbfc3f305c6b Updated test_crawler to scan paths in more sensible order using ordered tuple.
Marcin Kuzminski <marcin@python-works.com>
parents: 1334
diff changeset
75 def test_changeset_walk(limit=None):
bbfc3f305c6b Updated test_crawler to scan paths in more sensible order using ordered tuple.
Marcin Kuzminski <marcin@python-works.com>
parents: 1334
diff changeset
76 print 'processing', jn(PROJECT_PATH, PROJECT)
1334
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
77 total_time = 0
1332
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
78
1334
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
79 repo = vcs.get_repo(jn(PROJECT_PATH, PROJECT))
1338
bbfc3f305c6b Updated test_crawler to scan paths in more sensible order using ordered tuple.
Marcin Kuzminski <marcin@python-works.com>
parents: 1334
diff changeset
80 cnt = 0
1332
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
81 for i in repo:
1338
bbfc3f305c6b Updated test_crawler to scan paths in more sensible order using ordered tuple.
Marcin Kuzminski <marcin@python-works.com>
parents: 1334
diff changeset
82 cnt += 1
1332
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
83 raw_cs = '/'.join((PROJECT, 'changeset', i.raw_id))
1338
bbfc3f305c6b Updated test_crawler to scan paths in more sensible order using ordered tuple.
Marcin Kuzminski <marcin@python-works.com>
parents: 1334
diff changeset
84 if limit and limit == cnt:
bbfc3f305c6b Updated test_crawler to scan paths in more sensible order using ordered tuple.
Marcin Kuzminski <marcin@python-works.com>
parents: 1334
diff changeset
85 break
1332
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
86
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
87 full_uri = (BASE_URI % raw_cs)
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
88 s = time.time()
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
89 f = o.open(full_uri)
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
90 size = len(f.read())
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
91 e = time.time() - s
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
92 total_time += e
1341
1881b808a71d small fixes for test crawler
Marcin Kuzminski <marcin@python-works.com>
parents: 1338
diff changeset
93 print '%s visited %s\%s size:%s req:%s ms' % (cnt, full_uri, i, size, e)
1332
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
94
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
95 print 'total_time', total_time
1338
bbfc3f305c6b Updated test_crawler to scan paths in more sensible order using ordered tuple.
Marcin Kuzminski <marcin@python-works.com>
parents: 1334
diff changeset
96 print 'average on req', total_time / float(cnt)
bbfc3f305c6b Updated test_crawler to scan paths in more sensible order using ordered tuple.
Marcin Kuzminski <marcin@python-works.com>
parents: 1334
diff changeset
97
1332
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
98
1341
1881b808a71d small fixes for test crawler
Marcin Kuzminski <marcin@python-works.com>
parents: 1338
diff changeset
99 def test_files_walk(limit=100):
1338
bbfc3f305c6b Updated test_crawler to scan paths in more sensible order using ordered tuple.
Marcin Kuzminski <marcin@python-works.com>
parents: 1334
diff changeset
100 print 'processing', jn(PROJECT_PATH, PROJECT)
1334
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
101 total_time = 0
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
102
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
103 repo = vcs.get_repo(jn(PROJECT_PATH, PROJECT))
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
104
1514
87ec80c280bb fixed issues with python2.5
Marcin Kuzminski <marcin@python-works.com>
parents: 1374
diff changeset
105 from rhodecode.lib.compat import OrderedSet
1338
bbfc3f305c6b Updated test_crawler to scan paths in more sensible order using ordered tuple.
Marcin Kuzminski <marcin@python-works.com>
parents: 1334
diff changeset
106
bbfc3f305c6b Updated test_crawler to scan paths in more sensible order using ordered tuple.
Marcin Kuzminski <marcin@python-works.com>
parents: 1334
diff changeset
107 paths_ = OrderedSet([''])
1334
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
108 try:
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
109 tip = repo.get_changeset('tip')
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
110 for topnode, dirs, files in tip.walk('/'):
1338
bbfc3f305c6b Updated test_crawler to scan paths in more sensible order using ordered tuple.
Marcin Kuzminski <marcin@python-works.com>
parents: 1334
diff changeset
111
bbfc3f305c6b Updated test_crawler to scan paths in more sensible order using ordered tuple.
Marcin Kuzminski <marcin@python-works.com>
parents: 1334
diff changeset
112 for dir in dirs:
bbfc3f305c6b Updated test_crawler to scan paths in more sensible order using ordered tuple.
Marcin Kuzminski <marcin@python-works.com>
parents: 1334
diff changeset
113 paths_.add(dir.path)
bbfc3f305c6b Updated test_crawler to scan paths in more sensible order using ordered tuple.
Marcin Kuzminski <marcin@python-works.com>
parents: 1334
diff changeset
114 for f in dir:
bbfc3f305c6b Updated test_crawler to scan paths in more sensible order using ordered tuple.
Marcin Kuzminski <marcin@python-works.com>
parents: 1334
diff changeset
115 paths_.add(f.path)
bbfc3f305c6b Updated test_crawler to scan paths in more sensible order using ordered tuple.
Marcin Kuzminski <marcin@python-works.com>
parents: 1334
diff changeset
116
1334
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
117 for f in files:
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
118 paths_.add(f.path)
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
119
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
120 except vcs.exception.RepositoryError, e:
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
121 pass
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
122
1341
1881b808a71d small fixes for test crawler
Marcin Kuzminski <marcin@python-works.com>
parents: 1338
diff changeset
123 cnt = 0
1334
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
124 for f in paths_:
1341
1881b808a71d small fixes for test crawler
Marcin Kuzminski <marcin@python-works.com>
parents: 1338
diff changeset
125 cnt += 1
1881b808a71d small fixes for test crawler
Marcin Kuzminski <marcin@python-works.com>
parents: 1338
diff changeset
126 if limit and limit == cnt:
1881b808a71d small fixes for test crawler
Marcin Kuzminski <marcin@python-works.com>
parents: 1338
diff changeset
127 break
1881b808a71d small fixes for test crawler
Marcin Kuzminski <marcin@python-works.com>
parents: 1338
diff changeset
128
1334
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
129 file_path = '/'.join((PROJECT, 'files', 'tip', f))
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
130
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
131 full_uri = (BASE_URI % file_path)
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
132 s = time.time()
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
133 f = o.open(full_uri)
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
134 size = len(f.read())
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
135 e = time.time() - s
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
136 total_time += e
1341
1881b808a71d small fixes for test crawler
Marcin Kuzminski <marcin@python-works.com>
parents: 1338
diff changeset
137 print '%s visited %s size:%s req:%s ms' % (cnt, full_uri, size, e)
1334
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
138
08cd02374883 Added mem_watch script. Test can also walk on file tree. Fixed some path issues
Marcin Kuzminski <marcin@python-works.com>
parents: 1332
diff changeset
139 print 'total_time', total_time
1341
1881b808a71d small fixes for test crawler
Marcin Kuzminski <marcin@python-works.com>
parents: 1338
diff changeset
140 print 'average on req', total_time / float(cnt)
1881b808a71d small fixes for test crawler
Marcin Kuzminski <marcin@python-works.com>
parents: 1338
diff changeset
141
1332
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
142
3fdfecc52c32 added test for crawling and memory usage
Marcin Kuzminski <marcin@python-works.com>
parents:
diff changeset
143
1338
bbfc3f305c6b Updated test_crawler to scan paths in more sensible order using ordered tuple.
Marcin Kuzminski <marcin@python-works.com>
parents: 1334
diff changeset
144 test_changelog_walk(40)
1341
1881b808a71d small fixes for test crawler
Marcin Kuzminski <marcin@python-works.com>
parents: 1338
diff changeset
145 time.sleep(2)
1338
bbfc3f305c6b Updated test_crawler to scan paths in more sensible order using ordered tuple.
Marcin Kuzminski <marcin@python-works.com>
parents: 1334
diff changeset
146 test_changeset_walk(limit=100)
1341
1881b808a71d small fixes for test crawler
Marcin Kuzminski <marcin@python-works.com>
parents: 1338
diff changeset
147 time.sleep(2)
1881b808a71d small fixes for test crawler
Marcin Kuzminski <marcin@python-works.com>
parents: 1338
diff changeset
148 test_files_walk(100)