Mercurial > gemma
annotate pkg/controllers/diff.go @ 4931:e41d42be0e13 fairway-marks-import
One more callback to avoid code duplication more consequently
author | Tom Gottfried <tom@intevation.de> |
---|---|
date | Fri, 14 Feb 2020 19:34:08 +0100 |
parents | 4847ac70103a |
children | 34bc6041e61e |
rev | line source |
---|---|
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
7 // Copyright (C) 2018, 2019 by via donau |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
13 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
14 package controllers |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 import ( |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
17 "context" |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
18 "database/sql" |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
19 "errors" |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
20 "fmt" |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
21 "log" |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
22 "net/http" |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
23 "runtime" |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
24 "time" |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
25 |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
26 "gemma.intevation.de/gemma/pkg/auth" |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
27 "gemma.intevation.de/gemma/pkg/common" |
4827
f4abfd0ee8ad
Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4768
diff
changeset
|
28 "gemma.intevation.de/gemma/pkg/mesh" |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
29 "gemma.intevation.de/gemma/pkg/models" |
4244
4394daeea96a
Moved JSONHandler into middleware package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4243
diff
changeset
|
30 |
4394daeea96a
Moved JSONHandler into middleware package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4243
diff
changeset
|
31 mw "gemma.intevation.de/gemma/pkg/middleware" |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
32 ) |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
33 |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
34 const ( |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
35 // maxNumberWorkers are processing diffs at a time. |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
36 maxNumberWorkers = 2 |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
37 // maxQueueLength are the number of diffs to |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
38 // be wait for when idle. |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
39 maxQueueLength = 5 |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
40 ) |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
41 |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
42 const ( |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
43 contourTolerance = 0.1 |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
44 contourStep = 0.1 |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
45 ) |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
46 |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
47 const ( |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
48 // isoCellSize is the side length of a raster cell when tracing |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
49 // iso areas. |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
50 isoCellSize = 0.5 |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
51 ) |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
52 |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
53 const ( |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
54 diffIDSQL = ` |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
55 SELECT sd.id FROM |
3656
2a079d0a71c1
Ensure sounding results are associated to matching bottleneck version
Tom Gottfried <tom@intevation.de>
parents:
2587
diff
changeset
|
56 caching.sounding_differences sd JOIN |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
57 waterway.sounding_results srm ON sd.minuend = srm.id JOIN |
2587
857bb070b9f1
Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2585
diff
changeset
|
58 waterway.sounding_results srs ON sd.subtrahend = srs.id |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
59 WHERE srm.bottleneck_id = srs.bottleneck_id AND |
2587
857bb070b9f1
Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2585
diff
changeset
|
60 srm.bottleneck_id = $1 AND |
857bb070b9f1
Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2585
diff
changeset
|
61 srm.date_info = $2::date AND |
857bb070b9f1
Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2585
diff
changeset
|
62 srs.date_info = $3::date |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
63 ` |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
64 insertDiffSQL = ` |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
65 WITH soundings AS ( |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
66 SELECT sr.id AS id, sr.date_info AS date_info FROM |
2587
857bb070b9f1
Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2585
diff
changeset
|
67 waterway.sounding_results sr |
857bb070b9f1
Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2585
diff
changeset
|
68 WHERE sr.bottleneck_id = $1 |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
69 ) |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
70 INSERT INTO caching.sounding_differences (minuend, subtrahend) |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
71 SELECT m.id, s.id FROM soundings m, soundings s |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
72 WHERE m.date_info = $2::date AND s.date_info = $3::date |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
73 RETURNING id |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
74 ` |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
75 insertDiffIsoAreasQL = ` |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
76 INSERT INTO caching.sounding_differences_iso_areas ( |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
77 sounding_differences_id, |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
78 height, |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
79 areas |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
80 ) |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
81 SELECT |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
82 $1, |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
83 $2, |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
84 ST_Transform( |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
85 ST_Multi( |
4726
c91e759007da
Started with stacking shells and holes of generated polygons correctly.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4686
diff
changeset
|
86 ST_Collectionextract( |
c91e759007da
Started with stacking shells and holes of generated polygons correctly.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4686
diff
changeset
|
87 ST_SimplifyPreserveTopology(ST_GeomFromWKB($4, $3::integer), $5), |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
88 3 |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
89 ) |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
90 ), |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
91 4326 |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
92 ) |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
93 ` |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
94 commonDiffBBoxSQL = ` |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
95 WITH joined AS ( |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
96 SELECT |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
97 sr.area AS area, |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
98 sr.date_info AS date_info |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
99 FROM waterway.sounding_results sr |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
100 WHERE sr.bottleneck_id = $1 |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
101 ), |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
102 bbox AS ( |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
103 SELECT ST_Extent(ST_intersection( |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
104 (SELECT ST_Transform(area::geometry, $2::int) FROM joined WHERE date_info = $3::date), |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
105 (SELECT ST_Transform(area::geometry, $2::int) FROM joined WHERE date_info = $4::date) |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
106 )) AS area |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
107 ) |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
108 SELECT ST_XMin(area), ST_YMin(area), ST_XMax(area), ST_YMax(area) FROM bbox |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
109 ` |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
110 ) |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
111 |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
112 type ( |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
113 diffResult struct { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
114 id int64 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
115 err error |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
116 } |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
117 |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
118 diffCalculationManager struct { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
119 waiting map[string][]chan diffResult |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
120 cmds chan func(*diffCalculationManager) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
121 jobs chan *models.DiffCalculationInput |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
122 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
123 ) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
124 |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
125 var errBufferFull = errors.New("buffer full") |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
126 |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
127 var diffCalcMng = startDiffCalculationManager() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
128 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
129 func diffCalcHash(dci *models.DiffCalculationInput) string { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
130 return dci.Bottleneck + "/" + |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
131 dci.Minuend.Format(common.DateFormat) + "/" + |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
132 dci.Subtrahend.Format(common.DateFormat) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
133 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
134 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
135 func startDiffCalculationManager() *diffCalculationManager { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
136 dcm := diffCalculationManager{ |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
137 waiting: make(map[string][]chan diffResult), |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
138 cmds: make(chan func(*diffCalculationManager)), |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
139 jobs: make(chan *models.DiffCalculationInput, maxQueueLength), |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
140 } |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
141 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
142 go dcm.run() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
143 return &dcm |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
144 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
145 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
146 func (dcm *diffCalculationManager) run() { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
147 n := runtime.NumCPU() |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
148 if n > maxNumberWorkers { |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
149 n = maxNumberWorkers |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
150 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
151 for i := 0; i < n; i++ { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
152 go dcm.calculate() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
153 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
154 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
155 for cmd := range dcm.cmds { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
156 cmd(dcm) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
157 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
158 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
159 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
160 func doDiff(ctx context.Context, conn *sql.Conn, dci *models.DiffCalculationInput) (int64, error) { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
161 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
162 start := time.Now() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
163 begin := start |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
164 |
4827
f4abfd0ee8ad
Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4768
diff
changeset
|
165 minuendTree, err := mesh.FromCache( |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
166 ctx, conn, |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
167 dci.Bottleneck, dci.Minuend.Time) |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
168 |
4658
4bbfe3dd2ab5
Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4641
diff
changeset
|
169 log.Printf("info: loading minuend mesh took %s\n", time.Since(start)) |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
170 if err != nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
171 return 0, err |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
172 } |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
173 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
174 if minuendTree == nil { |
4847
4847ac70103a
Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4827
diff
changeset
|
175 return 0, fmt.Errorf("cannot find survey for %s/%s", |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
176 dci.Bottleneck, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
177 dci.Minuend.Format(common.DateFormat)) |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
178 } |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
179 |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
180 epsg := minuendTree.EPSG() |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
181 |
4827
f4abfd0ee8ad
Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4768
diff
changeset
|
182 var box mesh.Box2D |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
183 |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
184 switch err := conn.QueryRowContext( |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
185 ctx, |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
186 commonDiffBBoxSQL, |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
187 dci.Bottleneck, |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
188 epsg, |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
189 dci.Minuend.Time, |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
190 dci.Subtrahend.Time, |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
191 ).Scan(&box.X1, &box.Y1, &box.X2, &box.Y2); { |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
192 case err == sql.ErrNoRows: |
4847
4847ac70103a
Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4827
diff
changeset
|
193 return 0, errors.New("no such intersection") |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
194 case err != nil: |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
195 return 0, err |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
196 } |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
197 |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
198 if box.Empty() { |
4847
4847ac70103a
Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4827
diff
changeset
|
199 return 0, errors.New("intersection is empty") |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
200 } |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
201 |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
202 log.Printf("info: bbox of intersection: (%.2f, %.2f) - (%.2f, %.2f)\n", |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
203 box.X1, box.Y1, box.X2, box.Y2) |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
204 |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
205 start = time.Now() |
4827
f4abfd0ee8ad
Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4768
diff
changeset
|
206 raster := mesh.NewRaster(box, isoCellSize) |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
207 raster.Rasterize(minuendTree.Value) |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
208 log.Printf("info: rasterizing minuend took %v\n", time.Since(start)) |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
209 |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
210 minuendTree = nil |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
211 |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
212 start = time.Now() |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
213 |
4827
f4abfd0ee8ad
Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4768
diff
changeset
|
214 subtrahendTree, err := mesh.FromCache( |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
215 ctx, conn, |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
216 dci.Bottleneck, dci.Subtrahend.Time) |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
217 |
4658
4bbfe3dd2ab5
Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4641
diff
changeset
|
218 log.Printf("info: loading subtrahend mesh took %s\n", time.Since(start)) |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
219 if err != nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
220 return 0, err |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
221 } |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
222 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
223 if subtrahendTree == nil { |
4847
4847ac70103a
Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4827
diff
changeset
|
224 return 0, fmt.Errorf("cannot find survey for %s/%s", |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
225 dci.Bottleneck, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
226 dci.Subtrahend.Format(common.DateFormat)) |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
227 } |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
228 |
2572
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
229 // We need a slow path implementation for this. |
4658
4bbfe3dd2ab5
Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4641
diff
changeset
|
230 if epsg != subtrahendTree.EPSG() { |
4847
4847ac70103a
Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4827
diff
changeset
|
231 return 0, errors.New("calculating differences between two different " + |
4847ac70103a
Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4827
diff
changeset
|
232 "EPSG code meshes are not supported, yet") |
2572
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
233 } |
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
234 |
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
235 start = time.Now() |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
236 raster.Diff(subtrahendTree.Value) |
2572
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
237 log.Printf("info: A - B took %v\n", time.Since(start)) |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
238 subtrahendTree = nil |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
239 |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
240 // XXX: Maybe we should start this transaction earlier!? |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
241 var tx *sql.Tx |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
242 if tx, err = conn.BeginTx(ctx, nil); err != nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
243 return 0, err |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
244 } |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
245 defer tx.Rollback() |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
246 |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
247 zMin, zMax, ok := raster.ZExtent() |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
248 if !ok { |
4847
4847ac70103a
Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4827
diff
changeset
|
249 return 0, errors.New("scans do not have common points") |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
250 } |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
251 |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
252 log.Printf("info: z range: %.3f - %.3f\n", zMin, zMax) |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
253 |
3883
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
254 var heights []float64 |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
255 |
4827
f4abfd0ee8ad
Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4768
diff
changeset
|
256 heights, err = mesh.LoadClassBreaks( |
3883
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
257 ctx, tx, |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
258 "morphology_classbreaks_compare") |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
259 if err != nil { |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
260 log.Printf("warn: Loading class breaks failed: %v\n", err) |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
261 err = nil |
4827
f4abfd0ee8ad
Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4768
diff
changeset
|
262 heights = mesh.SampleDiffHeights(zMin, zMax, contourStep) |
3883
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
263 } else { |
4827
f4abfd0ee8ad
Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4768
diff
changeset
|
264 heights = mesh.ExtrapolateClassBreaks(heights, zMin, zMax) |
3883
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
265 } |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
266 |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
267 heights = common.DedupFloat64s(heights) |
4685
7a9388943840
morphology: Clip class breaks again Z min and max of the the height model.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4681
diff
changeset
|
268 |
3883
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
269 log.Printf("info: num heights: %d\n", len(heights)) |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
270 |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
271 var isoStmt *sql.Stmt |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
272 if isoStmt, err = tx.PrepareContext(ctx, insertDiffIsoAreasQL); err != nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
273 return 0, err |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
274 } |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
275 defer isoStmt.Close() |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
276 |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
277 var id int64 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
278 |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
279 if err = tx.QueryRowContext( |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
280 ctx, |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
281 insertDiffSQL, |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
282 dci.Bottleneck, |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
283 dci.Minuend.Time, |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
284 dci.Subtrahend.Time, |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
285 ).Scan(&id); err != nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
286 return 0, err |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
287 } |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
288 |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
289 areas := raster.Trace(heights) |
4256
4ca884dfc470
Prevent heights duplicates in iso line calculations.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4244
diff
changeset
|
290 |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
291 raster = nil |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
292 |
4639
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
293 var size int |
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
294 |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
295 for i, a := range areas { |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
296 if len(a) == 0 { |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
297 continue |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
298 } |
4639
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
299 wkb := a.AsWKB() |
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
300 size += len(wkb) |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
301 if _, err = isoStmt.ExecContext( |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
302 ctx, |
4641
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
303 id, heights[i], epsg, |
4639
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
304 wkb, |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
305 contourTolerance, |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
306 ); err != nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
307 return 0, err |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
308 } |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
309 } |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
310 |
4639
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
311 log.Printf("info: Transferred WKB size: %.2fMB.\n", |
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
312 float64(size)/(1024*1024)) |
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
313 |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
314 log.Printf("info: calculating and storing iso areas took %v\n", |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
315 time.Since(start)) |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
316 |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
317 if err = tx.Commit(); err != nil { |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
318 log.Printf("info: difference calculation failed after %v\n", |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
319 time.Since(begin)) |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
320 return 0, err |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
321 } |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
322 log.Printf("info: difference calculation succeed after %v\n", |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
323 time.Since(begin)) |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
324 |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
325 return id, nil |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
326 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
327 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
328 func (dcm *diffCalculationManager) calculate() { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
329 for dci := range dcm.jobs { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
330 ctx := context.Background() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
331 var id int64 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
332 var err error |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
333 err = auth.RunAs(ctx, "sys_admin", func(conn *sql.Conn) error { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
334 id, err = doDiff(ctx, conn, dci) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
335 return err |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
336 }) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
337 dcm.cmds <- func(dcm *diffCalculationManager) { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
338 msg := diffResult{id: id, err: err} |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
339 key := diffCalcHash(dci) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
340 list := dcm.waiting[key] |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
341 delete(dcm.waiting, key) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
342 for _, ch := range list { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
343 ch <- msg |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
344 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
345 } |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
346 } |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
347 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
348 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
349 func (dcm *diffCalculationManager) enque(dci *models.DiffCalculationInput) chan diffResult { |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
350 // buffer to prevent blocking even if receiver is dead. |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
351 ch := make(chan diffResult, 1) |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
352 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
353 dcm.cmds <- func(dcm *diffCalculationManager) { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
354 // When already running add to list of waitings |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
355 key := diffCalcHash(dci) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
356 if list := dcm.waiting[key]; list != nil { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
357 dcm.waiting[key] = append(list, ch) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
358 return |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
359 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
360 // look if it is in database |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
361 var id int64 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
362 ctx := context.Background() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
363 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
364 err := auth.RunAs(ctx, "sys_admin", func(conn *sql.Conn) error { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
365 return conn.QueryRowContext( |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
366 ctx, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
367 diffIDSQL, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
368 dci.Bottleneck, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
369 dci.Minuend.Time, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
370 dci.Subtrahend.Time, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
371 ).Scan(&id) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
372 }) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
373 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
374 switch { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
375 case err == sql.ErrNoRows: |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
376 // We need to calculate it. |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
377 case err != nil: // error |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
378 ch <- diffResult{err: err} |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
379 return |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
380 default: // we have it |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
381 ch <- diffResult{id: id} |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
382 return |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
383 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
384 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
385 // add to waiting |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
386 dcm.waiting[key] = []chan diffResult{ch} |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
387 // feed to workers |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
388 select { |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
389 case dcm.jobs <- dci: |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
390 default: // buffer is full |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
391 delete(dcm.waiting, key) |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
392 ch <- diffResult{err: errBufferFull} |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
393 } |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
394 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
395 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
396 return ch |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
397 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
398 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
399 func (dcm *diffCalculationManager) unregister(dci *models.DiffCalculationInput, ch chan diffResult) { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
400 dcm.cmds <- func(dcm *diffCalculationManager) { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
401 key := diffCalcHash(dci) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
402 list := dcm.waiting[key] |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
403 for i, x := range list { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
404 if x == ch { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
405 if i != len(list)-1 { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
406 copy(list[i:], list[i+1:]) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
407 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
408 list[len(list)-1] = nil |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
409 list = list[:len(list)-1] |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
410 if len(list) == 0 { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
411 delete(dcm.waiting, key) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
412 } else { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
413 dcm.waiting[key] = list |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
414 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
415 return |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
416 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
417 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
418 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
419 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
420 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
421 func diffCalculation(req *http.Request) (jr mw.JSONResult, err error) { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
422 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
423 dci := mw.JSONInput(req).(*models.DiffCalculationInput) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
424 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
425 ctx := req.Context() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
426 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
427 resultCh := diffCalcMng.enque(dci) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
428 defer diffCalcMng.unregister(dci, resultCh) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
429 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
430 select { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
431 case res := <-resultCh: |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
432 switch { |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
433 case res.err == errBufferFull: |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
434 err = mw.JSONError{ |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
435 Code: http.StatusTooManyRequests, |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
436 Message: "Too many differences are waiting to be processed", |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
437 } |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
438 case res.err != nil: |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
439 err = res.err |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
440 default: |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
441 jr = mw.JSONResult{ |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
442 Result: map[string]int64{"id": res.id}, |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
443 } |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
444 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
445 case <-ctx.Done(): |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
446 log.Println("request canceled") |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
447 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
448 |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
449 return |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
450 } |