Mercurial > gemma
annotate pkg/controllers/diff.go @ 5591:0011f50cf216 surveysperbottleneckid
Removed no longer used alternative api for surveys/ endpoint.
As bottlenecks in the summary for SR imports are now identified by
their id and no longer by the (not guarantied to be unique!) name,
there is no longer the need to request survey data by the name+date
tuple (which isn't reliable anyway). So the workaround was now
reversed.
author | Sascha Wilde <wilde@sha-bang.de> |
---|---|
date | Wed, 06 Apr 2022 13:30:29 +0200 |
parents | 5f47eeea988d |
children |
rev | line source |
---|---|
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
7 // Copyright (C) 2018, 2019 by via donau |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
13 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
14 package controllers |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 import ( |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
17 "context" |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
18 "database/sql" |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
19 "errors" |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
20 "fmt" |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
21 "net/http" |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
22 "runtime" |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
23 "time" |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
24 |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
25 "gemma.intevation.de/gemma/pkg/auth" |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
26 "gemma.intevation.de/gemma/pkg/common" |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5412
diff
changeset
|
27 "gemma.intevation.de/gemma/pkg/log" |
4827
f4abfd0ee8ad
Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4768
diff
changeset
|
28 "gemma.intevation.de/gemma/pkg/mesh" |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
29 "gemma.intevation.de/gemma/pkg/models" |
4244
4394daeea96a
Moved JSONHandler into middleware package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4243
diff
changeset
|
30 |
4394daeea96a
Moved JSONHandler into middleware package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4243
diff
changeset
|
31 mw "gemma.intevation.de/gemma/pkg/middleware" |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
32 ) |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
33 |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
34 const ( |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
35 // maxNumberWorkers are processing diffs at a time. |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
36 maxNumberWorkers = 2 |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
37 // maxQueueLength are the number of diffs to |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
38 // be wait for when idle. |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
39 maxQueueLength = 5 |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
40 ) |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
41 |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
42 const ( |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
43 contourTolerance = 0.1 |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
44 contourStep = 0.1 |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
45 ) |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
46 |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
47 const ( |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
48 // isoCellSize is the side length of a raster cell when tracing |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
49 // iso areas. |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
50 isoCellSize = 0.5 |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
51 ) |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
52 |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
53 const ( |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
54 diffIDSQL = ` |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
55 SELECT sd.id FROM |
3656
2a079d0a71c1
Ensure sounding results are associated to matching bottleneck version
Tom Gottfried <tom@intevation.de>
parents:
2587
diff
changeset
|
56 caching.sounding_differences sd JOIN |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
57 waterway.sounding_results srm ON sd.minuend = srm.id JOIN |
2587
857bb070b9f1
Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2585
diff
changeset
|
58 waterway.sounding_results srs ON sd.subtrahend = srs.id |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
59 WHERE srm.bottleneck_id = srs.bottleneck_id AND |
2587
857bb070b9f1
Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2585
diff
changeset
|
60 srm.bottleneck_id = $1 AND |
857bb070b9f1
Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2585
diff
changeset
|
61 srm.date_info = $2::date AND |
857bb070b9f1
Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2585
diff
changeset
|
62 srs.date_info = $3::date |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
63 ` |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
64 insertDiffSQL = ` |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
65 WITH soundings AS ( |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
66 SELECT sr.id AS id, sr.date_info AS date_info FROM |
2587
857bb070b9f1
Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2585
diff
changeset
|
67 waterway.sounding_results sr |
857bb070b9f1
Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2585
diff
changeset
|
68 WHERE sr.bottleneck_id = $1 |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
69 ) |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
70 INSERT INTO caching.sounding_differences (minuend, subtrahend) |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
71 SELECT m.id, s.id FROM soundings m, soundings s |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
72 WHERE m.date_info = $2::date AND s.date_info = $3::date |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
73 RETURNING id |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
74 ` |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
75 insertDiffIsoAreasQL = ` |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
76 INSERT INTO caching.sounding_differences_iso_areas ( |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
77 sounding_differences_id, |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
78 height, |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
79 areas |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
80 ) |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
81 SELECT |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
82 $1, |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
83 $2, |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
84 ST_Transform( |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
85 ST_Multi( |
4726
c91e759007da
Started with stacking shells and holes of generated polygons correctly.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4686
diff
changeset
|
86 ST_Collectionextract( |
c91e759007da
Started with stacking shells and holes of generated polygons correctly.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4686
diff
changeset
|
87 ST_SimplifyPreserveTopology(ST_GeomFromWKB($4, $3::integer), $5), |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
88 3 |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
89 ) |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
90 ), |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
91 4326 |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
92 ) |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
93 ` |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
94 commonDiffBBoxSQL = ` |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
95 WITH joined AS ( |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
96 SELECT |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
97 sr.area AS area, |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
98 sr.date_info AS date_info |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
99 FROM waterway.sounding_results sr |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
100 WHERE sr.bottleneck_id = $1 |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
101 ), |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
102 bbox AS ( |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
103 SELECT ST_Extent(ST_intersection( |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
104 (SELECT ST_Transform(area::geometry, $2::int) FROM joined WHERE date_info = $3::date), |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
105 (SELECT ST_Transform(area::geometry, $2::int) FROM joined WHERE date_info = $4::date) |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
106 )) AS area |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
107 ) |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
108 SELECT ST_XMin(area), ST_YMin(area), ST_XMax(area), ST_YMax(area) FROM bbox |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
109 ` |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
110 ) |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
111 |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
112 type ( |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
113 diffResult struct { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
114 id int64 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
115 err error |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
116 } |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
117 |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
118 diffCalculationManager struct { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
119 waiting map[string][]chan diffResult |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
120 cmds chan func(*diffCalculationManager) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
121 jobs chan *models.DiffCalculationInput |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
122 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
123 ) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
124 |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
125 var errBufferFull = errors.New("buffer full") |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
126 |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
127 var diffCalcMng = startDiffCalculationManager() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
128 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
129 func diffCalcHash(dci *models.DiffCalculationInput) string { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
130 return dci.Bottleneck + "/" + |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
131 dci.Minuend.Format(common.DateFormat) + "/" + |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
132 dci.Subtrahend.Format(common.DateFormat) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
133 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
134 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
135 func startDiffCalculationManager() *diffCalculationManager { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
136 dcm := diffCalculationManager{ |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
137 waiting: make(map[string][]chan diffResult), |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
138 cmds: make(chan func(*diffCalculationManager)), |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
139 jobs: make(chan *models.DiffCalculationInput, maxQueueLength), |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
140 } |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
141 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
142 go dcm.run() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
143 return &dcm |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
144 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
145 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
146 func (dcm *diffCalculationManager) run() { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
147 n := runtime.NumCPU() |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
148 if n > maxNumberWorkers { |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
149 n = maxNumberWorkers |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
150 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
151 for i := 0; i < n; i++ { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
152 go dcm.calculate() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
153 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
154 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
155 for cmd := range dcm.cmds { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
156 cmd(dcm) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
157 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
158 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
159 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
160 func doDiff(ctx context.Context, conn *sql.Conn, dci *models.DiffCalculationInput) (int64, error) { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
161 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
162 start := time.Now() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
163 begin := start |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
164 |
4827
f4abfd0ee8ad
Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4768
diff
changeset
|
165 minuendTree, err := mesh.FromCache( |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
166 ctx, conn, |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
167 dci.Bottleneck, dci.Minuend.Time) |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
168 |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5412
diff
changeset
|
169 log.Infof("loading minuend mesh took %s\n", time.Since(start)) |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
170 if err != nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
171 return 0, err |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
172 } |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
173 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
174 if minuendTree == nil { |
4847
4847ac70103a
Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4827
diff
changeset
|
175 return 0, fmt.Errorf("cannot find survey for %s/%s", |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
176 dci.Bottleneck, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
177 dci.Minuend.Format(common.DateFormat)) |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
178 } |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
179 |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
180 epsg := minuendTree.EPSG() |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
181 |
4827
f4abfd0ee8ad
Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4768
diff
changeset
|
182 var box mesh.Box2D |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
183 |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
184 switch err := conn.QueryRowContext( |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
185 ctx, |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
186 commonDiffBBoxSQL, |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
187 dci.Bottleneck, |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
188 epsg, |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
189 dci.Minuend.Time, |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
190 dci.Subtrahend.Time, |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
191 ).Scan(&box.X1, &box.Y1, &box.X2, &box.Y2); { |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
192 case err == sql.ErrNoRows: |
4847
4847ac70103a
Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4827
diff
changeset
|
193 return 0, errors.New("no such intersection") |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
194 case err != nil: |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
195 return 0, err |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
196 } |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
197 |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
198 if box.Empty() { |
4847
4847ac70103a
Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4827
diff
changeset
|
199 return 0, errors.New("intersection is empty") |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
200 } |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
201 |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5412
diff
changeset
|
202 log.Infof("bbox of intersection: (%.2f, %.2f) - (%.2f, %.2f)\n", |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
203 box.X1, box.Y1, box.X2, box.Y2) |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
204 |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
205 start = time.Now() |
4827
f4abfd0ee8ad
Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4768
diff
changeset
|
206 raster := mesh.NewRaster(box, isoCellSize) |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
207 raster.Rasterize(minuendTree.Value) |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5412
diff
changeset
|
208 log.Infof("rasterizing minuend took %v\n", time.Since(start)) |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
209 |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
210 minuendTree = nil |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
211 |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
212 start = time.Now() |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
213 |
4827
f4abfd0ee8ad
Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4768
diff
changeset
|
214 subtrahendTree, err := mesh.FromCache( |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
215 ctx, conn, |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
216 dci.Bottleneck, dci.Subtrahend.Time) |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
217 |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5412
diff
changeset
|
218 log.Infof("loading subtrahend mesh took %s\n", time.Since(start)) |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
219 if err != nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
220 return 0, err |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
221 } |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
222 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
223 if subtrahendTree == nil { |
4847
4847ac70103a
Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4827
diff
changeset
|
224 return 0, fmt.Errorf("cannot find survey for %s/%s", |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
225 dci.Bottleneck, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
226 dci.Subtrahend.Format(common.DateFormat)) |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
227 } |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
228 |
2572
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
229 // We need a slow path implementation for this. |
4658
4bbfe3dd2ab5
Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4641
diff
changeset
|
230 if epsg != subtrahendTree.EPSG() { |
4847
4847ac70103a
Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4827
diff
changeset
|
231 return 0, errors.New("calculating differences between two different " + |
4847ac70103a
Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4827
diff
changeset
|
232 "EPSG code meshes are not supported, yet") |
2572
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
233 } |
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
234 |
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
235 start = time.Now() |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
236 raster.Diff(subtrahendTree.Value) |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5412
diff
changeset
|
237 log.Infof("A - B took %v\n", time.Since(start)) |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
238 subtrahendTree = nil |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
239 |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
240 // XXX: Maybe we should start this transaction earlier!? |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
241 var tx *sql.Tx |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
242 if tx, err = conn.BeginTx(ctx, nil); err != nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
243 return 0, err |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
244 } |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
245 defer tx.Rollback() |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
246 |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
247 zMin, zMax, ok := raster.ZExtent() |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
248 if !ok { |
4847
4847ac70103a
Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4827
diff
changeset
|
249 return 0, errors.New("scans do not have common points") |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
250 } |
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
251 |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5412
diff
changeset
|
252 log.Infof("z range: %.3f - %.3f\n", zMin, zMax) |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
253 |
5412
34bc6041e61e
Added a type for class breaks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4847
diff
changeset
|
254 var heights mesh.ClassBreaks |
3883
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
255 |
4827
f4abfd0ee8ad
Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4768
diff
changeset
|
256 heights, err = mesh.LoadClassBreaks( |
3883
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
257 ctx, tx, |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
258 "morphology_classbreaks_compare") |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
259 if err != nil { |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5412
diff
changeset
|
260 log.Warnf("Loading class breaks failed: %v\n", err) |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
261 err = nil |
4827
f4abfd0ee8ad
Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4768
diff
changeset
|
262 heights = mesh.SampleDiffHeights(zMin, zMax, contourStep) |
3883
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
263 } else { |
5412
34bc6041e61e
Added a type for class breaks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4847
diff
changeset
|
264 heights = heights.ExtrapolateClassBreaks(zMin, zMax) |
3883
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
265 } |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
266 |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
267 heights = common.DedupFloat64s(heights) |
4685
7a9388943840
morphology: Clip class breaks again Z min and max of the the height model.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4681
diff
changeset
|
268 |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5412
diff
changeset
|
269 log.Infof("num heights: %d\n", len(heights)) |
3883
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
270 |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
271 var isoStmt *sql.Stmt |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
272 if isoStmt, err = tx.PrepareContext(ctx, insertDiffIsoAreasQL); err != nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
273 return 0, err |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
274 } |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
275 defer isoStmt.Close() |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
276 |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
277 var id int64 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
278 |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
279 if err = tx.QueryRowContext( |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
280 ctx, |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
281 insertDiffSQL, |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
282 dci.Bottleneck, |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
283 dci.Minuend.Time, |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
284 dci.Subtrahend.Time, |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
285 ).Scan(&id); err != nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
286 return 0, err |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
287 } |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
288 |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
289 areas := raster.Trace(heights) |
4256
4ca884dfc470
Prevent heights duplicates in iso line calculations.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4244
diff
changeset
|
290 |
4768
a2f16bbcc846
Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4762
diff
changeset
|
291 raster = nil |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
292 |
4639
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
293 var size int |
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
294 |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
295 for i, a := range areas { |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
296 if len(a) == 0 { |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
297 continue |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
298 } |
4639
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
299 wkb := a.AsWKB() |
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
300 size += len(wkb) |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
301 if _, err = isoStmt.ExecContext( |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
302 ctx, |
4641
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
303 id, heights[i], epsg, |
4639
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
304 wkb, |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
305 contourTolerance, |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
306 ); err != nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
307 return 0, err |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
308 } |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
309 } |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
310 |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5412
diff
changeset
|
311 log.Infof("Transferred WKB size: %.2fMB.\n", |
4639
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
312 float64(size)/(1024*1024)) |
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
313 |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5412
diff
changeset
|
314 log.Infof("calculating and storing iso areas took %v\n", |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
315 time.Since(start)) |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
316 |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
317 if err = tx.Commit(); err != nil { |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5412
diff
changeset
|
318 log.Infof("difference calculation failed after %v\n", |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
319 time.Since(begin)) |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
320 return 0, err |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
321 } |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5412
diff
changeset
|
322 log.Infof("difference calculation succeed after %v\n", |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
323 time.Since(begin)) |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
324 |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
325 return id, nil |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
326 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
327 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
328 func (dcm *diffCalculationManager) calculate() { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
329 for dci := range dcm.jobs { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
330 ctx := context.Background() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
331 var id int64 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
332 var err error |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
333 err = auth.RunAs(ctx, "sys_admin", func(conn *sql.Conn) error { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
334 id, err = doDiff(ctx, conn, dci) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
335 return err |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
336 }) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
337 dcm.cmds <- func(dcm *diffCalculationManager) { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
338 msg := diffResult{id: id, err: err} |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
339 key := diffCalcHash(dci) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
340 list := dcm.waiting[key] |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
341 delete(dcm.waiting, key) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
342 for _, ch := range list { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
343 ch <- msg |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
344 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
345 } |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
346 } |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
347 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
348 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
349 func (dcm *diffCalculationManager) enque(dci *models.DiffCalculationInput) chan diffResult { |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
350 // buffer to prevent blocking even if receiver is dead. |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
351 ch := make(chan diffResult, 1) |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
352 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
353 dcm.cmds <- func(dcm *diffCalculationManager) { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
354 // When already running add to list of waitings |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
355 key := diffCalcHash(dci) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
356 if list := dcm.waiting[key]; list != nil { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
357 dcm.waiting[key] = append(list, ch) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
358 return |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
359 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
360 // look if it is in database |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
361 var id int64 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
362 ctx := context.Background() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
363 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
364 err := auth.RunAs(ctx, "sys_admin", func(conn *sql.Conn) error { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
365 return conn.QueryRowContext( |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
366 ctx, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
367 diffIDSQL, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
368 dci.Bottleneck, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
369 dci.Minuend.Time, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
370 dci.Subtrahend.Time, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
371 ).Scan(&id) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
372 }) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
373 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
374 switch { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
375 case err == sql.ErrNoRows: |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
376 // We need to calculate it. |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
377 case err != nil: // error |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
378 ch <- diffResult{err: err} |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
379 return |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
380 default: // we have it |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
381 ch <- diffResult{id: id} |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
382 return |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
383 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
384 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
385 // add to waiting |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
386 dcm.waiting[key] = []chan diffResult{ch} |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
387 // feed to workers |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
388 select { |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
389 case dcm.jobs <- dci: |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
390 default: // buffer is full |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
391 delete(dcm.waiting, key) |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
392 ch <- diffResult{err: errBufferFull} |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
393 } |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
394 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
395 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
396 return ch |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
397 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
398 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
399 func (dcm *diffCalculationManager) unregister(dci *models.DiffCalculationInput, ch chan diffResult) { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
400 dcm.cmds <- func(dcm *diffCalculationManager) { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
401 key := diffCalcHash(dci) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
402 list := dcm.waiting[key] |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
403 for i, x := range list { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
404 if x == ch { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
405 if i != len(list)-1 { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
406 copy(list[i:], list[i+1:]) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
407 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
408 list[len(list)-1] = nil |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
409 list = list[:len(list)-1] |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
410 if len(list) == 0 { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
411 delete(dcm.waiting, key) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
412 } else { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
413 dcm.waiting[key] = list |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
414 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
415 return |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
416 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
417 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
418 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
419 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
420 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
421 func diffCalculation(req *http.Request) (jr mw.JSONResult, err error) { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
422 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
423 dci := mw.JSONInput(req).(*models.DiffCalculationInput) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
424 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
425 ctx := req.Context() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
426 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
427 resultCh := diffCalcMng.enque(dci) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
428 defer diffCalcMng.unregister(dci, resultCh) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
429 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
430 select { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
431 case res := <-resultCh: |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
432 switch { |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
433 case res.err == errBufferFull: |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
434 err = mw.JSONError{ |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
435 Code: http.StatusTooManyRequests, |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
436 Message: "Too many differences are waiting to be processed", |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
437 } |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
438 case res.err != nil: |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
439 err = res.err |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
440 default: |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
441 jr = mw.JSONResult{ |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
442 Result: map[string]int64{"id": res.id}, |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
443 } |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
444 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
445 case <-ctx.Done(): |
5490
5f47eeea988d
Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
5412
diff
changeset
|
446 log.Infoln("request canceled") |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
447 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
448 |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
449 return |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
450 } |