Mercurial > gemma
annotate pkg/controllers/diff.go @ 4686:1d6f28e45696
morphology: Doing ST_SimplifyPreserveTopology before ST_MakeValid when storing geometries to database speeds things up ... a lot.
author | Sascha L. Teichmann <sascha.teichmann@intevation.de> |
---|---|
date | Wed, 16 Oct 2019 11:51:53 +0200 |
parents | 7a9388943840 |
children | c91e759007da |
rev | line source |
---|---|
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
7 // Copyright (C) 2018, 2019 by via donau |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
13 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
14 package controllers |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 import ( |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
17 "context" |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
18 "database/sql" |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
19 "errors" |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
20 "fmt" |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
21 "log" |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
22 "net/http" |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
23 "runtime" |
2574
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
24 "sync" |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
25 "time" |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
26 |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
27 "gemma.intevation.de/gemma/pkg/auth" |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
28 "gemma.intevation.de/gemma/pkg/common" |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
29 "gemma.intevation.de/gemma/pkg/models" |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
30 "gemma.intevation.de/gemma/pkg/octree" |
4244
4394daeea96a
Moved JSONHandler into middleware package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4243
diff
changeset
|
31 |
4394daeea96a
Moved JSONHandler into middleware package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4243
diff
changeset
|
32 mw "gemma.intevation.de/gemma/pkg/middleware" |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
33 ) |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
34 |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
35 const ( |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
36 // maxNumberWorkers are processing diffs at a time. |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
37 maxNumberWorkers = 2 |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
38 // maxQueueLength are the number of diffs to |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
39 // be wait for when idle. |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
40 maxQueueLength = 5 |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
41 ) |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
42 |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
43 const ( |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
44 contourTolerance = 0.1 |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
45 contourStep = 0.1 |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
46 ) |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
47 |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
48 const ( |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
49 // isoCellSize is the side length of a raster cell when tracing |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
50 // iso areas. |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
51 isoCellSize = 0.5 |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
52 ) |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
53 |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
54 const ( |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
55 diffIDSQL = ` |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
56 SELECT sd.id FROM |
3656
2a079d0a71c1
Ensure sounding results are associated to matching bottleneck version
Tom Gottfried <tom@intevation.de>
parents:
2587
diff
changeset
|
57 caching.sounding_differences sd JOIN |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
58 waterway.sounding_results srm ON sd.minuend = srm.id JOIN |
2587
857bb070b9f1
Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2585
diff
changeset
|
59 waterway.sounding_results srs ON sd.subtrahend = srs.id |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
60 WHERE srm.bottleneck_id = srs.bottleneck_id AND |
2587
857bb070b9f1
Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2585
diff
changeset
|
61 srm.bottleneck_id = $1 AND |
857bb070b9f1
Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2585
diff
changeset
|
62 srm.date_info = $2::date AND |
857bb070b9f1
Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2585
diff
changeset
|
63 srs.date_info = $3::date |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
64 ` |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
65 insertDiffSQL = ` |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
66 WITH soundings AS ( |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
67 SELECT sr.id AS id, sr.date_info AS date_info FROM |
2587
857bb070b9f1
Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2585
diff
changeset
|
68 waterway.sounding_results sr |
857bb070b9f1
Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2585
diff
changeset
|
69 WHERE sr.bottleneck_id = $1 |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
70 ) |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
71 INSERT INTO caching.sounding_differences (minuend, subtrahend) |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
72 SELECT m.id, s.id FROM soundings m, soundings s |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
73 WHERE m.date_info = $2::date AND s.date_info = $3::date |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
74 RETURNING id |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
75 ` |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
76 insertDiffIsoAreasQL = ` |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
77 INSERT INTO caching.sounding_differences_iso_areas ( |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
78 sounding_differences_id, |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
79 height, |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
80 areas |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
81 ) |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
82 SELECT |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
83 $1, |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
84 $2, |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
85 ST_Transform( |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
86 ST_Multi( |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
87 ST_CollectionExtract( |
4686
1d6f28e45696
morphology: Doing ST_SimplifyPreserveTopology before ST_MakeValid when storing geometries to database speeds things up ... a lot.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4685
diff
changeset
|
88 ST_MakeValid( |
1d6f28e45696
morphology: Doing ST_SimplifyPreserveTopology before ST_MakeValid when storing geometries to database speeds things up ... a lot.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4685
diff
changeset
|
89 ST_Multi( |
1d6f28e45696
morphology: Doing ST_SimplifyPreserveTopology before ST_MakeValid when storing geometries to database speeds things up ... a lot.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4685
diff
changeset
|
90 ST_Collectionextract( |
1d6f28e45696
morphology: Doing ST_SimplifyPreserveTopology before ST_MakeValid when storing geometries to database speeds things up ... a lot.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4685
diff
changeset
|
91 ST_SimplifyPreserveTopology(ST_GeomFromWKB($4, $3::integer), $5), |
1d6f28e45696
morphology: Doing ST_SimplifyPreserveTopology before ST_MakeValid when storing geometries to database speeds things up ... a lot.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4685
diff
changeset
|
92 3 |
1d6f28e45696
morphology: Doing ST_SimplifyPreserveTopology before ST_MakeValid when storing geometries to database speeds things up ... a lot.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4685
diff
changeset
|
93 ) |
1d6f28e45696
morphology: Doing ST_SimplifyPreserveTopology before ST_MakeValid when storing geometries to database speeds things up ... a lot.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4685
diff
changeset
|
94 ) |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
95 ), |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
96 3 |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
97 ) |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
98 ), |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
99 4326 |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
100 ) |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
101 ` |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
102 ) |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
103 |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
104 type ( |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
105 diffResult struct { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
106 id int64 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
107 err error |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
108 } |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
109 |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
110 diffCalculationManager struct { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
111 waiting map[string][]chan diffResult |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
112 cmds chan func(*diffCalculationManager) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
113 jobs chan *models.DiffCalculationInput |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
114 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
115 ) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
116 |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
117 var errBufferFull = errors.New("buffer full") |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
118 |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
119 var diffCalcMng = startDiffCalculationManager() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
120 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
121 func diffCalcHash(dci *models.DiffCalculationInput) string { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
122 return dci.Bottleneck + "/" + |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
123 dci.Minuend.Format(common.DateFormat) + "/" + |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
124 dci.Subtrahend.Format(common.DateFormat) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
125 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
126 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
127 func startDiffCalculationManager() *diffCalculationManager { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
128 dcm := diffCalculationManager{ |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
129 waiting: make(map[string][]chan diffResult), |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
130 cmds: make(chan func(*diffCalculationManager)), |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
131 jobs: make(chan *models.DiffCalculationInput, maxQueueLength), |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
132 } |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
133 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
134 go dcm.run() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
135 return &dcm |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
136 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
137 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
138 func (dcm *diffCalculationManager) run() { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
139 n := runtime.NumCPU() |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
140 if n > maxNumberWorkers { |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
141 n = maxNumberWorkers |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
142 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
143 for i := 0; i < n; i++ { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
144 go dcm.calculate() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
145 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
146 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
147 for cmd := range dcm.cmds { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
148 cmd(dcm) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
149 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
150 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
151 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
152 func doDiff(ctx context.Context, conn *sql.Conn, dci *models.DiffCalculationInput) (int64, error) { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
153 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
154 start := time.Now() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
155 begin := start |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
156 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
157 minuendTree, err := octree.FromCache( |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
158 ctx, conn, |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
159 dci.Bottleneck, dci.Minuend.Time) |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
160 |
4658
4bbfe3dd2ab5
Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4641
diff
changeset
|
161 log.Printf("info: loading minuend mesh took %s\n", time.Since(start)) |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
162 if err != nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
163 return 0, err |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
164 } |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
165 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
166 if minuendTree == nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
167 return 0, fmt.Errorf("Cannot find survey for %s/%s.", |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
168 dci.Bottleneck, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
169 dci.Minuend.Format(common.DateFormat)) |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
170 } |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
171 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
172 start = time.Now() |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
173 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
174 subtrahendTree, err := octree.FromCache( |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
175 ctx, conn, |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
176 dci.Bottleneck, dci.Subtrahend.Time) |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
177 |
4658
4bbfe3dd2ab5
Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4641
diff
changeset
|
178 log.Printf("info: loading subtrahend mesh took %s\n", time.Since(start)) |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
179 if err != nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
180 return 0, err |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
181 } |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
182 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
183 if subtrahendTree == nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
184 return 0, fmt.Errorf("Cannot find survey for %s/%s.", |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
185 dci.Bottleneck, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
186 dci.Subtrahend.Format(common.DateFormat)) |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
187 } |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
188 |
2572
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
189 // We need a slow path implementation for this. |
4658
4bbfe3dd2ab5
Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4641
diff
changeset
|
190 epsg := minuendTree.EPSG() |
4bbfe3dd2ab5
Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4641
diff
changeset
|
191 if epsg != subtrahendTree.EPSG() { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
192 return 0, errors.New("Calculating differences between two different " + |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
193 "EPSG code meshes are not supported, yet.") |
2572
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
194 } |
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
195 |
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
196 start = time.Now() |
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
197 points := minuendTree.Diff(subtrahendTree) |
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
198 log.Printf("info: A - B took %v\n", time.Since(start)) |
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
199 |
4641
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
200 minuendTree, subtrahendTree = nil, nil |
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
201 |
2574
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
202 // The Triangulation and the loading of the clipping |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
203 // polygon can be done concurrently. |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
204 |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
205 jobs := make(chan func()) |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
206 |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
207 wg := new(sync.WaitGroup) |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
208 for i := 0; i < 2; i++ { |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
209 wg.Add(1) |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
210 go func() { |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
211 defer wg.Done() |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
212 for job := range jobs { |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
213 job() |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
214 } |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
215 }() |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
216 } |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
217 |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
218 var ( |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
219 tri *octree.Triangulation |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
220 triErr error |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
221 clip *octree.Polygon |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
222 clipErr error |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
223 ) |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
224 |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
225 jobs <- func() { |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
226 start := time.Now() |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
227 tri, triErr = points.Triangulate() |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
228 log.Printf("info: triangulation took %v\n", time.Since(start)) |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
229 } |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
230 |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
231 jobs <- func() { |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
232 start := time.Now() |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
233 clip, clipErr = octree.LoadClippingPolygon( |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
234 ctx, conn, |
4641
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
235 epsg, |
2574
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
236 dci.Bottleneck, |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
237 dci.Minuend.Time, |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
238 dci.Subtrahend.Time) |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
239 log.Printf("info: loading clipping polygon took %v\n", time.Since(start)) |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
240 } |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
241 close(jobs) |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
242 wg.Wait() |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
243 |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
244 switch { |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
245 case triErr != nil && clipErr != nil: |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
246 return 0, fmt.Errorf("%v %v", triErr, clipErr) |
2574
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
247 case triErr != nil: |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
248 return 0, triErr |
2574
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
249 case clipErr != nil: |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
250 return 0, clipErr |
2572
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
251 } |
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
252 |
2575
59e7a011d347
Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2574
diff
changeset
|
253 start = time.Now() |
59e7a011d347
Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2574
diff
changeset
|
254 tin := tri.Tin() |
59e7a011d347
Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2574
diff
changeset
|
255 removed := tin.Clip(clip) |
4641
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
256 clip = nil |
2575
59e7a011d347
Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2574
diff
changeset
|
257 log.Printf("info: clipping TIN took %v\n", time.Since(start)) |
59e7a011d347
Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2574
diff
changeset
|
258 |
4641
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
259 log.Printf("info: Number of triangles to clip: %d\n", len(removed)) |
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
260 |
2575
59e7a011d347
Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2574
diff
changeset
|
261 start = time.Now() |
4641
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
262 var tree octree.STRTree |
2575
59e7a011d347
Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2574
diff
changeset
|
263 |
4641
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
264 tree.BuildWithout(tin, removed) |
2576
647a58ee9ae9
Morphological differences: Centralized generation of height values for differences in octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2575
diff
changeset
|
265 |
4658
4bbfe3dd2ab5
Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4641
diff
changeset
|
266 log.Printf("info: Building final mesh took: %v\n", time.Since(start)) |
2576
647a58ee9ae9
Morphological differences: Centralized generation of height values for differences in octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2575
diff
changeset
|
267 |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
268 start = time.Now() |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
269 |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
270 // XXX: Maybe we should start this transaction earlier!? |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
271 var tx *sql.Tx |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
272 if tx, err = conn.BeginTx(ctx, nil); err != nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
273 return 0, err |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
274 } |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
275 defer tx.Rollback() |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
276 |
3883
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
277 var heights []float64 |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
278 |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
279 heights, err = octree.LoadClassBreaks( |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
280 ctx, tx, |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
281 "morphology_classbreaks_compare") |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
282 if err != nil { |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
283 log.Printf("warn: Loading class breaks failed: %v\n", err) |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
284 err = nil |
4641
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
285 heights = octree.SampleDiffHeights(tin.Min.Z, tin.Max.Z, contourStep) |
3883
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
286 } else { |
4641
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
287 heights = octree.ExtrapolateClassBreaks(heights, tin.Min.Z, tin.Max.Z) |
3954
cb4fda122321
Completing: Don't create extra inbetween class breaks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3883
diff
changeset
|
288 // heights = octree.InBetweenClassBreaks(heights, 0.05, 2) |
3883
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
289 } |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
290 |
4685
7a9388943840
morphology: Clip class breaks again Z min and max of the the height model.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4681
diff
changeset
|
291 log.Printf("info: z range: %.3f - %.3f\n", tin.Min.Z, tin.Max.Z) |
7a9388943840
morphology: Clip class breaks again Z min and max of the the height model.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4681
diff
changeset
|
292 |
3883
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
293 log.Printf("info: num heights: %d\n", len(heights)) |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
294 |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
295 var isoStmt *sql.Stmt |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
296 if isoStmt, err = tx.PrepareContext(ctx, insertDiffIsoAreasQL); err != nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
297 return 0, err |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
298 } |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
299 defer isoStmt.Close() |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
300 |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
301 var id int64 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
302 |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
303 if err = tx.QueryRowContext( |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
304 ctx, |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
305 insertDiffSQL, |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
306 dci.Bottleneck, |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
307 dci.Minuend.Time, |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
308 dci.Subtrahend.Time, |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
309 ).Scan(&id); err != nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
310 return 0, err |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
311 } |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
312 |
4256
4ca884dfc470
Prevent heights duplicates in iso line calculations.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4244
diff
changeset
|
313 heights = common.DedupFloat64s(heights) |
4ca884dfc470
Prevent heights duplicates in iso line calculations.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4244
diff
changeset
|
314 |
4641
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
315 areas := octree.TraceAreas(heights, isoCellSize, tin.Min, tin.Max, tree.Value) |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
316 |
4639
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
317 var size int |
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
318 |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
319 for i, a := range areas { |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
320 if len(a) == 0 { |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
321 continue |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
322 } |
4639
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
323 wkb := a.AsWKB() |
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
324 size += len(wkb) |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
325 if _, err = isoStmt.ExecContext( |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
326 ctx, |
4641
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
327 id, heights[i], epsg, |
4639
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
328 wkb, |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
329 contourTolerance, |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
330 ); err != nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
331 return 0, err |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
332 } |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
333 } |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
334 |
4639
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
335 log.Printf("info: Transferred WKB size: %.2fMB.\n", |
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
336 float64(size)/(1024*1024)) |
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
337 |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
338 log.Printf("info: calculating and storing iso lines took %v\n", |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
339 time.Since(start)) |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
340 |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
341 if err = tx.Commit(); err != nil { |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
342 log.Printf("info: difference calculation failed after %v\n", |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
343 time.Since(begin)) |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
344 return 0, err |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
345 } |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
346 log.Printf("info: difference calculation succeed after %v\n", |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
347 time.Since(begin)) |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
348 |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
349 return id, nil |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
350 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
351 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
352 func (dcm *diffCalculationManager) calculate() { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
353 for dci := range dcm.jobs { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
354 ctx := context.Background() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
355 var id int64 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
356 var err error |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
357 err = auth.RunAs(ctx, "sys_admin", func(conn *sql.Conn) error { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
358 id, err = doDiff(ctx, conn, dci) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
359 return err |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
360 }) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
361 dcm.cmds <- func(dcm *diffCalculationManager) { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
362 msg := diffResult{id: id, err: err} |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
363 key := diffCalcHash(dci) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
364 list := dcm.waiting[key] |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
365 delete(dcm.waiting, key) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
366 for _, ch := range list { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
367 ch <- msg |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
368 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
369 } |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
370 } |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
371 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
372 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
373 func (dcm *diffCalculationManager) enque(dci *models.DiffCalculationInput) chan diffResult { |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
374 // buffer to prevent blocking even if receiver is dead. |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
375 ch := make(chan diffResult, 1) |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
376 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
377 dcm.cmds <- func(dcm *diffCalculationManager) { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
378 // When already running add to list of waitings |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
379 key := diffCalcHash(dci) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
380 if list := dcm.waiting[key]; list != nil { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
381 dcm.waiting[key] = append(list, ch) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
382 return |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
383 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
384 // look if it is in database |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
385 var id int64 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
386 ctx := context.Background() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
387 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
388 err := auth.RunAs(ctx, "sys_admin", func(conn *sql.Conn) error { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
389 return conn.QueryRowContext( |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
390 ctx, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
391 diffIDSQL, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
392 dci.Bottleneck, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
393 dci.Minuend.Time, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
394 dci.Subtrahend.Time, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
395 ).Scan(&id) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
396 }) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
397 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
398 switch { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
399 case err == sql.ErrNoRows: |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
400 // We need to calculate it. |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
401 case err != nil: // error |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
402 ch <- diffResult{err: err} |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
403 return |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
404 default: // we have it |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
405 ch <- diffResult{id: id} |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
406 return |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
407 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
408 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
409 // add to waiting |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
410 dcm.waiting[key] = []chan diffResult{ch} |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
411 // feed to workers |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
412 select { |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
413 case dcm.jobs <- dci: |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
414 default: // buffer is full |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
415 delete(dcm.waiting, key) |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
416 ch <- diffResult{err: errBufferFull} |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
417 } |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
418 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
419 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
420 return ch |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
421 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
422 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
423 func (dcm *diffCalculationManager) unregister(dci *models.DiffCalculationInput, ch chan diffResult) { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
424 dcm.cmds <- func(dcm *diffCalculationManager) { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
425 key := diffCalcHash(dci) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
426 list := dcm.waiting[key] |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
427 for i, x := range list { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
428 if x == ch { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
429 if i != len(list)-1 { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
430 copy(list[i:], list[i+1:]) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
431 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
432 list[len(list)-1] = nil |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
433 list = list[:len(list)-1] |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
434 if len(list) == 0 { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
435 delete(dcm.waiting, key) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
436 } else { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
437 dcm.waiting[key] = list |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
438 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
439 return |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
440 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
441 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
442 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
443 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
444 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
445 func diffCalculation(req *http.Request) (jr mw.JSONResult, err error) { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
446 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
447 dci := mw.JSONInput(req).(*models.DiffCalculationInput) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
448 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
449 ctx := req.Context() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
450 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
451 resultCh := diffCalcMng.enque(dci) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
452 defer diffCalcMng.unregister(dci, resultCh) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
453 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
454 select { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
455 case res := <-resultCh: |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
456 switch { |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
457 case res.err == errBufferFull: |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
458 err = mw.JSONError{ |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
459 Code: http.StatusTooManyRequests, |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
460 Message: "Too many differences are waiting to be processed", |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
461 } |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
462 case res.err != nil: |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
463 err = res.err |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
464 default: |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
465 jr = mw.JSONResult{ |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
466 Result: map[string]int64{"id": res.id}, |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
467 } |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
468 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
469 case <-ctx.Done(): |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
470 log.Println("request canceled") |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
471 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
472 |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
473 return |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
474 } |