Mercurial > gemma

annotate pkg/controllers/diff.go @ 5591:0011f50cf216 surveysperbottleneckid

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Removed no longer used alternative api for surveys/ endpoint. As bottlenecks in the summary for SR imports are now identified by their id and no longer by the (not guarantied to be unique!) name, there is no longer the need to request survey data by the name+date tuple (which isn't reliable anyway). So the workaround was now reversed.
author Sascha Wilde <wilde@sha-bang.de>
date Wed, 06 Apr 2022 13:30:29 +0200
parents 5f47eeea988d
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
3 //
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
6 //
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
7 // Copyright (C) 2018, 2019 by via donau
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
10 //
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
11 // Author(s):
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
13
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
14 package controllers
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
15
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
16 import (
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
17 "context"
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
18 "database/sql"
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
19 "errors"
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
20 "fmt"
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
21 "net/http"
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
22 "runtime"
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
23 "time"
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
24
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
25 "gemma.intevation.de/gemma/pkg/auth"
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
26 "gemma.intevation.de/gemma/pkg/common"
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
27 "gemma.intevation.de/gemma/pkg/log"
4827
f4abfd0ee8ad Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4768
diff changeset
28 "gemma.intevation.de/gemma/pkg/mesh"
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
29 "gemma.intevation.de/gemma/pkg/models"
4244
4394daeea96a Moved JSONHandler into middleware package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4243
diff changeset
30
4394daeea96a Moved JSONHandler into middleware package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4243
diff changeset
31 mw "gemma.intevation.de/gemma/pkg/middleware"
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
32 )
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
33
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
34 const (
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
35 // maxNumberWorkers are processing diffs at a time.
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
36 maxNumberWorkers = 2
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
37 // maxQueueLength are the number of diffs to
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
38 // be wait for when idle.
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
39 maxQueueLength = 5
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
40 )
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
41
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
42 const (
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
43 contourTolerance = 0.1
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
44 contourStep = 0.1
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
45 )
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
46
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
47 const (
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
48 // isoCellSize is the side length of a raster cell when tracing
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
49 // iso areas.
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
50 isoCellSize = 0.5
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
51 )
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
52
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
53 const (
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
54 diffIDSQL = `
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
55 SELECT sd.id FROM
3656
2a079d0a71c1 Ensure sounding results are associated to matching bottleneck version
Tom Gottfried <tom@intevation.de>
parents: 2587
diff changeset
56 caching.sounding_differences sd JOIN
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
57 waterway.sounding_results srm ON sd.minuend = srm.id JOIN
2587
857bb070b9f1 Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2585
diff changeset
58 waterway.sounding_results srs ON sd.subtrahend = srs.id
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
59 WHERE srm.bottleneck_id = srs.bottleneck_id AND
2587
857bb070b9f1 Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2585
diff changeset
60 srm.bottleneck_id = $1 AND
857bb070b9f1 Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2585
diff changeset
61 srm.date_info = $2::date AND
857bb070b9f1 Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2585
diff changeset
62 srs.date_info = $3::date
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
63 `
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
64 insertDiffSQL = `
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
65 WITH soundings AS (
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
66 SELECT sr.id AS id, sr.date_info AS date_info FROM
2587
857bb070b9f1 Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2585
diff changeset
67 waterway.sounding_results sr
857bb070b9f1 Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2585
diff changeset
68 WHERE sr.bottleneck_id = $1
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
69 )
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
70 INSERT INTO caching.sounding_differences (minuend, subtrahend)
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
71 SELECT m.id, s.id FROM soundings m, soundings s
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
72 WHERE m.date_info = $2::date AND s.date_info = $3::date
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
73 RETURNING id
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
74 `
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
75 insertDiffIsoAreasQL = `
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
76 INSERT INTO caching.sounding_differences_iso_areas (
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
77 sounding_differences_id,
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
78 height,
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
79 areas
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
80 )
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
81 SELECT
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
82 $1,
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
83 $2,
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
84 ST_Transform(
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
85 ST_Multi(
4726
c91e759007da Started with stacking shells and holes of generated polygons correctly.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4686
diff changeset
86 ST_Collectionextract(
c91e759007da Started with stacking shells and holes of generated polygons correctly.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4686
diff changeset
87 ST_SimplifyPreserveTopology(ST_GeomFromWKB($4, $3::integer), $5),
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
88 3
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
89 )
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
90 ),
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
91 4326
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
92 )
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
93 `
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
94 commonDiffBBoxSQL = `
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
95 WITH joined AS (
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
96 SELECT
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
97 sr.area AS area,
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
98 sr.date_info AS date_info
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
99 FROM waterway.sounding_results sr
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
100 WHERE sr.bottleneck_id = $1
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
101 ),
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
102 bbox AS (
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
103 SELECT ST_Extent(ST_intersection(
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
104 (SELECT ST_Transform(area::geometry, $2::int) FROM joined WHERE date_info = $3::date),
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
105 (SELECT ST_Transform(area::geometry, $2::int) FROM joined WHERE date_info = $4::date)
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
106 )) AS area
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
107 )
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
108 SELECT ST_XMin(area), ST_YMin(area), ST_XMax(area), ST_YMax(area) FROM bbox
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
109 `
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
110 )
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
111
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
112 type (
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
113 diffResult struct {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
114 id int64
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
115 err error
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
116 }
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
117
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
118 diffCalculationManager struct {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
119 waiting map[string][]chan diffResult
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
120 cmds chan func(*diffCalculationManager)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
121 jobs chan *models.DiffCalculationInput
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
122 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
123 )
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
124
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
125 var errBufferFull = errors.New("buffer full")
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
126
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
127 var diffCalcMng = startDiffCalculationManager()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
128
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
129 func diffCalcHash(dci *models.DiffCalculationInput) string {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
130 return dci.Bottleneck + "/" +
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
131 dci.Minuend.Format(common.DateFormat) + "/" +
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
132 dci.Subtrahend.Format(common.DateFormat)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
133 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
134
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
135 func startDiffCalculationManager() *diffCalculationManager {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
136 dcm := diffCalculationManager{
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
137 waiting: make(map[string][]chan diffResult),
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
138 cmds: make(chan func(*diffCalculationManager)),
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
139 jobs: make(chan *models.DiffCalculationInput, maxQueueLength),
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
140 }
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
141
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
142 go dcm.run()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
143 return &dcm
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
144 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
145
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
146 func (dcm *diffCalculationManager) run() {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
147 n := runtime.NumCPU()
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
148 if n > maxNumberWorkers {
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
149 n = maxNumberWorkers
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
150 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
151 for i := 0; i < n; i++ {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
152 go dcm.calculate()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
153 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
154
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
155 for cmd := range dcm.cmds {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
156 cmd(dcm)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
157 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
158 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
159
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
160 func doDiff(ctx context.Context, conn *sql.Conn, dci *models.DiffCalculationInput) (int64, error) {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
161
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
162 start := time.Now()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
163 begin := start
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
164
4827
f4abfd0ee8ad Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4768
diff changeset
165 minuendTree, err := mesh.FromCache(
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
166 ctx, conn,
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
167 dci.Bottleneck, dci.Minuend.Time)
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
168
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
169 log.Infof("loading minuend mesh took %s\n", time.Since(start))
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
170 if err != nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
171 return 0, err
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
172 }
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
173
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
174 if minuendTree == nil {
4847
4847ac70103a Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4827
diff changeset
175 return 0, fmt.Errorf("cannot find survey for %s/%s",
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
176 dci.Bottleneck,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
177 dci.Minuend.Format(common.DateFormat))
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
178 }
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
179
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
180 epsg := minuendTree.EPSG()
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
181
4827
f4abfd0ee8ad Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4768
diff changeset
182 var box mesh.Box2D
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
183
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
184 switch err := conn.QueryRowContext(
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
185 ctx,
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
186 commonDiffBBoxSQL,
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
187 dci.Bottleneck,
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
188 epsg,
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
189 dci.Minuend.Time,
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
190 dci.Subtrahend.Time,
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
191 ).Scan(&box.X1, &box.Y1, &box.X2, &box.Y2); {
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
192 case err == sql.ErrNoRows:
4847
4847ac70103a Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4827
diff changeset
193 return 0, errors.New("no such intersection")
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
194 case err != nil:
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
195 return 0, err
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
196 }
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
197
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
198 if box.Empty() {
4847
4847ac70103a Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4827
diff changeset
199 return 0, errors.New("intersection is empty")
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
200 }
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
201
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
202 log.Infof("bbox of intersection: (%.2f, %.2f) - (%.2f, %.2f)\n",
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
203 box.X1, box.Y1, box.X2, box.Y2)
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
204
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
205 start = time.Now()
4827
f4abfd0ee8ad Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4768
diff changeset
206 raster := mesh.NewRaster(box, isoCellSize)
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
207 raster.Rasterize(minuendTree.Value)
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
208 log.Infof("rasterizing minuend took %v\n", time.Since(start))
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
209
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
210 minuendTree = nil
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
211
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
212 start = time.Now()
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
213
4827
f4abfd0ee8ad Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4768
diff changeset
214 subtrahendTree, err := mesh.FromCache(
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
215 ctx, conn,
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
216 dci.Bottleneck, dci.Subtrahend.Time)
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
217
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
218 log.Infof("loading subtrahend mesh took %s\n", time.Since(start))
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
219 if err != nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
220 return 0, err
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
221 }
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
222
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
223 if subtrahendTree == nil {
4847
4847ac70103a Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4827
diff changeset
224 return 0, fmt.Errorf("cannot find survey for %s/%s",
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
225 dci.Bottleneck,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
226 dci.Subtrahend.Format(common.DateFormat))
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
227 }
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
228
2572
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
229 // We need a slow path implementation for this.
4658
4bbfe3dd2ab5 Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4641
diff changeset
230 if epsg != subtrahendTree.EPSG() {
4847
4847ac70103a Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4827
diff changeset
231 return 0, errors.New("calculating differences between two different " +
4847ac70103a Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4827
diff changeset
232 "EPSG code meshes are not supported, yet")
2572
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
233 }
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
234
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
235 start = time.Now()
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
236 raster.Diff(subtrahendTree.Value)
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
237 log.Infof("A - B took %v\n", time.Since(start))
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
238 subtrahendTree = nil
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
239
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
240 // XXX: Maybe we should start this transaction earlier!?
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
241 var tx *sql.Tx
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
242 if tx, err = conn.BeginTx(ctx, nil); err != nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
243 return 0, err
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
244 }
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
245 defer tx.Rollback()
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
246
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
247 zMin, zMax, ok := raster.ZExtent()
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
248 if !ok {
4847
4847ac70103a Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4827
diff changeset
249 return 0, errors.New("scans do not have common points")
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
250 }
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
251
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
252 log.Infof("z range: %.3f - %.3f\n", zMin, zMax)
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
253
5412
34bc6041e61e Added a type for class breaks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4847
diff changeset
254 var heights mesh.ClassBreaks
3883
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
255
4827
f4abfd0ee8ad Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4768
diff changeset
256 heights, err = mesh.LoadClassBreaks(
3883
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
257 ctx, tx,
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
258 "morphology_classbreaks_compare")
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
259 if err != nil {
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
260 log.Warnf("Loading class breaks failed: %v\n", err)
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
261 err = nil
4827
f4abfd0ee8ad Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4768
diff changeset
262 heights = mesh.SampleDiffHeights(zMin, zMax, contourStep)
3883
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
263 } else {
5412
34bc6041e61e Added a type for class breaks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4847
diff changeset
264 heights = heights.ExtrapolateClassBreaks(zMin, zMax)
3883
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
265 }
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
266
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
267 heights = common.DedupFloat64s(heights)
4685
7a9388943840 morphology: Clip class breaks again Z min and max of the the height model.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4681
diff changeset
268
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
269 log.Infof("num heights: %d\n", len(heights))
3883
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
270
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
271 var isoStmt *sql.Stmt
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
272 if isoStmt, err = tx.PrepareContext(ctx, insertDiffIsoAreasQL); err != nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
273 return 0, err
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
274 }
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
275 defer isoStmt.Close()
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
276
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
277 var id int64
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
278
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
279 if err = tx.QueryRowContext(
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
280 ctx,
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
281 insertDiffSQL,
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
282 dci.Bottleneck,
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
283 dci.Minuend.Time,
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
284 dci.Subtrahend.Time,
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
285 ).Scan(&id); err != nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
286 return 0, err
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
287 }
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
288
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
289 areas := raster.Trace(heights)
4256
4ca884dfc470 Prevent heights duplicates in iso line calculations.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4244
diff changeset
290
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
291 raster = nil
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
292
4639
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
293 var size int
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
294
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
295 for i, a := range areas {
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
296 if len(a) == 0 {
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
297 continue
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
298 }
4639
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
299 wkb := a.AsWKB()
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
300 size += len(wkb)
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
301 if _, err = isoStmt.ExecContext(
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
302 ctx,
4641
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
303 id, heights[i], epsg,
4639
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
304 wkb,
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
305 contourTolerance,
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
306 ); err != nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
307 return 0, err
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
308 }
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
309 }
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
310
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
311 log.Infof("Transferred WKB size: %.2fMB.\n",
4639
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
312 float64(size)/(1024*1024))
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
313
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
314 log.Infof("calculating and storing iso areas took %v\n",
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
315 time.Since(start))
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
316
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
317 if err = tx.Commit(); err != nil {
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
318 log.Infof("difference calculation failed after %v\n",
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
319 time.Since(begin))
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
320 return 0, err
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
321 }
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
322 log.Infof("difference calculation succeed after %v\n",
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
323 time.Since(begin))
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
324
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
325 return id, nil
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
326 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
327
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
328 func (dcm *diffCalculationManager) calculate() {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
329 for dci := range dcm.jobs {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
330 ctx := context.Background()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
331 var id int64
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
332 var err error
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
333 err = auth.RunAs(ctx, "sys_admin", func(conn *sql.Conn) error {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
334 id, err = doDiff(ctx, conn, dci)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
335 return err
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
336 })
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
337 dcm.cmds <- func(dcm *diffCalculationManager) {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
338 msg := diffResult{id: id, err: err}
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
339 key := diffCalcHash(dci)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
340 list := dcm.waiting[key]
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
341 delete(dcm.waiting, key)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
342 for _, ch := range list {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
343 ch <- msg
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
344 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
345 }
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
346 }
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
347 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
348
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
349 func (dcm *diffCalculationManager) enque(dci *models.DiffCalculationInput) chan diffResult {
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
350 // buffer to prevent blocking even if receiver is dead.
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
351 ch := make(chan diffResult, 1)
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
352
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
353 dcm.cmds <- func(dcm *diffCalculationManager) {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
354 // When already running add to list of waitings
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
355 key := diffCalcHash(dci)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
356 if list := dcm.waiting[key]; list != nil {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
357 dcm.waiting[key] = append(list, ch)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
358 return
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
359 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
360 // look if it is in database
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
361 var id int64
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
362 ctx := context.Background()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
363
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
364 err := auth.RunAs(ctx, "sys_admin", func(conn *sql.Conn) error {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
365 return conn.QueryRowContext(
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
366 ctx,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
367 diffIDSQL,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
368 dci.Bottleneck,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
369 dci.Minuend.Time,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
370 dci.Subtrahend.Time,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
371 ).Scan(&id)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
372 })
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
373
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
374 switch {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
375 case err == sql.ErrNoRows:
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
376 // We need to calculate it.
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
377 case err != nil: // error
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
378 ch <- diffResult{err: err}
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
379 return
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
380 default: // we have it
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
381 ch <- diffResult{id: id}
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
382 return
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
383 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
384
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
385 // add to waiting
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
386 dcm.waiting[key] = []chan diffResult{ch}
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
387 // feed to workers
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
388 select {
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
389 case dcm.jobs <- dci:
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
390 default: // buffer is full
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
391 delete(dcm.waiting, key)
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
392 ch <- diffResult{err: errBufferFull}
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
393 }
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
394 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
395
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
396 return ch
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
397 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
398
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
399 func (dcm *diffCalculationManager) unregister(dci *models.DiffCalculationInput, ch chan diffResult) {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
400 dcm.cmds <- func(dcm *diffCalculationManager) {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
401 key := diffCalcHash(dci)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
402 list := dcm.waiting[key]
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
403 for i, x := range list {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
404 if x == ch {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
405 if i != len(list)-1 {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
406 copy(list[i:], list[i+1:])
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
407 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
408 list[len(list)-1] = nil
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
409 list = list[:len(list)-1]
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
410 if len(list) == 0 {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
411 delete(dcm.waiting, key)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
412 } else {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
413 dcm.waiting[key] = list
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
414 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
415 return
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
416 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
417 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
418 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
419 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
420
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
421 func diffCalculation(req *http.Request) (jr mw.JSONResult, err error) {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
422
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
423 dci := mw.JSONInput(req).(*models.DiffCalculationInput)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
424
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
425 ctx := req.Context()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
426
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
427 resultCh := diffCalcMng.enque(dci)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
428 defer diffCalcMng.unregister(dci, resultCh)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
429
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
430 select {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
431 case res := <-resultCh:
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
432 switch {
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
433 case res.err == errBufferFull:
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
434 err = mw.JSONError{
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
435 Code: http.StatusTooManyRequests,
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
436 Message: "Too many differences are waiting to be processed",
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
437 }
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
438 case res.err != nil:
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
439 err = res.err
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
440 default:
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
441 jr = mw.JSONResult{
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
442 Result: map[string]int64{"id": res.id},
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
443 }
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
444 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
445 case <-ctx.Done():
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
446 log.Infoln("request canceled")
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
447 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
448
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
449 return
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
450 }