Mercurial > gemma

annotate pkg/controllers/diff.go @ 5564:aaa9e658cabd

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Log export: Added marker interface to JobCreators that log messages should be loaded at export. Removes the hard coded SR export SQL logic.
author Sascha L. Teichmann <sascha.teichmann@intevation.de>
date Sat, 12 Feb 2022 18:56:55 +0100
parents 5f47eeea988d
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
3 //
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
6 //
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
7 // Copyright (C) 2018, 2019 by via donau
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
10 //
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
11 // Author(s):
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
13
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
14 package controllers
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
15
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
16 import (
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
17 "context"
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
18 "database/sql"
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
19 "errors"
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
20 "fmt"
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
21 "net/http"
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
22 "runtime"
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
23 "time"
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
24
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
25 "gemma.intevation.de/gemma/pkg/auth"
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
26 "gemma.intevation.de/gemma/pkg/common"
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
27 "gemma.intevation.de/gemma/pkg/log"
4827
f4abfd0ee8ad Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4768
diff changeset
28 "gemma.intevation.de/gemma/pkg/mesh"
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
29 "gemma.intevation.de/gemma/pkg/models"
4244
4394daeea96a Moved JSONHandler into middleware package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4243
diff changeset
30
4394daeea96a Moved JSONHandler into middleware package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4243
diff changeset
31 mw "gemma.intevation.de/gemma/pkg/middleware"
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
32 )
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
33
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
34 const (
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
35 // maxNumberWorkers are processing diffs at a time.
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
36 maxNumberWorkers = 2
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
37 // maxQueueLength are the number of diffs to
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
38 // be wait for when idle.
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
39 maxQueueLength = 5
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
40 )
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
41
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
42 const (
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
43 contourTolerance = 0.1
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
44 contourStep = 0.1
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
45 )
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
46
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
47 const (
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
48 // isoCellSize is the side length of a raster cell when tracing
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
49 // iso areas.
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
50 isoCellSize = 0.5
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
51 )
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
52
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
53 const (
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
54 diffIDSQL = `
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
55 SELECT sd.id FROM
3656
2a079d0a71c1 Ensure sounding results are associated to matching bottleneck version
Tom Gottfried <tom@intevation.de>
parents: 2587
diff changeset
56 caching.sounding_differences sd JOIN
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
57 waterway.sounding_results srm ON sd.minuend = srm.id JOIN
2587
857bb070b9f1 Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2585
diff changeset
58 waterway.sounding_results srs ON sd.subtrahend = srs.id
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
59 WHERE srm.bottleneck_id = srs.bottleneck_id AND
2587
857bb070b9f1 Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2585
diff changeset
60 srm.bottleneck_id = $1 AND
857bb070b9f1 Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2585
diff changeset
61 srm.date_info = $2::date AND
857bb070b9f1 Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2585
diff changeset
62 srs.date_info = $3::date
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
63 `
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
64 insertDiffSQL = `
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
65 WITH soundings AS (
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
66 SELECT sr.id AS id, sr.date_info AS date_info FROM
2587
857bb070b9f1 Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2585
diff changeset
67 waterway.sounding_results sr
857bb070b9f1 Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2585
diff changeset
68 WHERE sr.bottleneck_id = $1
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
69 )
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
70 INSERT INTO caching.sounding_differences (minuend, subtrahend)
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
71 SELECT m.id, s.id FROM soundings m, soundings s
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
72 WHERE m.date_info = $2::date AND s.date_info = $3::date
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
73 RETURNING id
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
74 `
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
75 insertDiffIsoAreasQL = `
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
76 INSERT INTO caching.sounding_differences_iso_areas (
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
77 sounding_differences_id,
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
78 height,
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
79 areas
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
80 )
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
81 SELECT
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
82 $1,
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
83 $2,
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
84 ST_Transform(
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
85 ST_Multi(
4726
c91e759007da Started with stacking shells and holes of generated polygons correctly.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4686
diff changeset
86 ST_Collectionextract(
c91e759007da Started with stacking shells and holes of generated polygons correctly.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4686
diff changeset
87 ST_SimplifyPreserveTopology(ST_GeomFromWKB($4, $3::integer), $5),
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
88 3
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
89 )
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
90 ),
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
91 4326
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
92 )
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
93 `
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
94 commonDiffBBoxSQL = `
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
95 WITH joined AS (
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
96 SELECT
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
97 sr.area AS area,
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
98 sr.date_info AS date_info
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
99 FROM waterway.sounding_results sr
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
100 WHERE sr.bottleneck_id = $1
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
101 ),
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
102 bbox AS (
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
103 SELECT ST_Extent(ST_intersection(
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
104 (SELECT ST_Transform(area::geometry, $2::int) FROM joined WHERE date_info = $3::date),
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
105 (SELECT ST_Transform(area::geometry, $2::int) FROM joined WHERE date_info = $4::date)
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
106 )) AS area
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
107 )
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
108 SELECT ST_XMin(area), ST_YMin(area), ST_XMax(area), ST_YMax(area) FROM bbox
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
109 `
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
110 )
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
111
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
112 type (
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
113 diffResult struct {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
114 id int64
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
115 err error
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
116 }
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
117
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
118 diffCalculationManager struct {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
119 waiting map[string][]chan diffResult
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
120 cmds chan func(*diffCalculationManager)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
121 jobs chan *models.DiffCalculationInput
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
122 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
123 )
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
124
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
125 var errBufferFull = errors.New("buffer full")
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
126
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
127 var diffCalcMng = startDiffCalculationManager()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
128
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
129 func diffCalcHash(dci *models.DiffCalculationInput) string {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
130 return dci.Bottleneck + "/" +
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
131 dci.Minuend.Format(common.DateFormat) + "/" +
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
132 dci.Subtrahend.Format(common.DateFormat)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
133 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
134
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
135 func startDiffCalculationManager() *diffCalculationManager {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
136 dcm := diffCalculationManager{
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
137 waiting: make(map[string][]chan diffResult),
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
138 cmds: make(chan func(*diffCalculationManager)),
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
139 jobs: make(chan *models.DiffCalculationInput, maxQueueLength),
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
140 }
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
141
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
142 go dcm.run()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
143 return &dcm
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
144 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
145
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
146 func (dcm *diffCalculationManager) run() {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
147 n := runtime.NumCPU()
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
148 if n > maxNumberWorkers {
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
149 n = maxNumberWorkers
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
150 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
151 for i := 0; i < n; i++ {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
152 go dcm.calculate()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
153 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
154
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
155 for cmd := range dcm.cmds {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
156 cmd(dcm)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
157 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
158 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
159
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
160 func doDiff(ctx context.Context, conn *sql.Conn, dci *models.DiffCalculationInput) (int64, error) {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
161
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
162 start := time.Now()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
163 begin := start
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
164
4827
f4abfd0ee8ad Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4768
diff changeset
165 minuendTree, err := mesh.FromCache(
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
166 ctx, conn,
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
167 dci.Bottleneck, dci.Minuend.Time)
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
168
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
169 log.Infof("loading minuend mesh took %s\n", time.Since(start))
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
170 if err != nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
171 return 0, err
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
172 }
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
173
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
174 if minuendTree == nil {
4847
4847ac70103a Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4827
diff changeset
175 return 0, fmt.Errorf("cannot find survey for %s/%s",
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
176 dci.Bottleneck,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
177 dci.Minuend.Format(common.DateFormat))
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
178 }
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
179
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
180 epsg := minuendTree.EPSG()
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
181
4827
f4abfd0ee8ad Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4768
diff changeset
182 var box mesh.Box2D
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
183
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
184 switch err := conn.QueryRowContext(
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
185 ctx,
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
186 commonDiffBBoxSQL,
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
187 dci.Bottleneck,
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
188 epsg,
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
189 dci.Minuend.Time,
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
190 dci.Subtrahend.Time,
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
191 ).Scan(&box.X1, &box.Y1, &box.X2, &box.Y2); {
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
192 case err == sql.ErrNoRows:
4847
4847ac70103a Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4827
diff changeset
193 return 0, errors.New("no such intersection")
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
194 case err != nil:
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
195 return 0, err
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
196 }
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
197
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
198 if box.Empty() {
4847
4847ac70103a Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4827
diff changeset
199 return 0, errors.New("intersection is empty")
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
200 }
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
201
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
202 log.Infof("bbox of intersection: (%.2f, %.2f) - (%.2f, %.2f)\n",
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
203 box.X1, box.Y1, box.X2, box.Y2)
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
204
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
205 start = time.Now()
4827
f4abfd0ee8ad Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4768
diff changeset
206 raster := mesh.NewRaster(box, isoCellSize)
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
207 raster.Rasterize(minuendTree.Value)
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
208 log.Infof("rasterizing minuend took %v\n", time.Since(start))
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
209
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
210 minuendTree = nil
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
211
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
212 start = time.Now()
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
213
4827
f4abfd0ee8ad Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4768
diff changeset
214 subtrahendTree, err := mesh.FromCache(
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
215 ctx, conn,
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
216 dci.Bottleneck, dci.Subtrahend.Time)
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
217
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
218 log.Infof("loading subtrahend mesh took %s\n", time.Since(start))
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
219 if err != nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
220 return 0, err
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
221 }
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
222
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
223 if subtrahendTree == nil {
4847
4847ac70103a Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4827
diff changeset
224 return 0, fmt.Errorf("cannot find survey for %s/%s",
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
225 dci.Bottleneck,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
226 dci.Subtrahend.Format(common.DateFormat))
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
227 }
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
228
2572
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
229 // We need a slow path implementation for this.
4658
4bbfe3dd2ab5 Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4641
diff changeset
230 if epsg != subtrahendTree.EPSG() {
4847
4847ac70103a Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4827
diff changeset
231 return 0, errors.New("calculating differences between two different " +
4847ac70103a Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4827
diff changeset
232 "EPSG code meshes are not supported, yet")
2572
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
233 }
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
234
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
235 start = time.Now()
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
236 raster.Diff(subtrahendTree.Value)
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
237 log.Infof("A - B took %v\n", time.Since(start))
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
238 subtrahendTree = nil
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
239
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
240 // XXX: Maybe we should start this transaction earlier!?
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
241 var tx *sql.Tx
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
242 if tx, err = conn.BeginTx(ctx, nil); err != nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
243 return 0, err
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
244 }
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
245 defer tx.Rollback()
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
246
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
247 zMin, zMax, ok := raster.ZExtent()
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
248 if !ok {
4847
4847ac70103a Made staticcheck happy.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4827
diff changeset
249 return 0, errors.New("scans do not have common points")
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
250 }
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
251
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
252 log.Infof("z range: %.3f - %.3f\n", zMin, zMax)
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
253
5412
34bc6041e61e Added a type for class breaks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4847
diff changeset
254 var heights mesh.ClassBreaks
3883
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
255
4827
f4abfd0ee8ad Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4768
diff changeset
256 heights, err = mesh.LoadClassBreaks(
3883
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
257 ctx, tx,
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
258 "morphology_classbreaks_compare")
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
259 if err != nil {
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
260 log.Warnf("Loading class breaks failed: %v\n", err)
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
261 err = nil
4827
f4abfd0ee8ad Renamed octree package to mesh as there is no octree any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4768
diff changeset
262 heights = mesh.SampleDiffHeights(zMin, zMax, contourStep)
3883
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
263 } else {
5412
34bc6041e61e Added a type for class breaks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4847
diff changeset
264 heights = heights.ExtrapolateClassBreaks(zMin, zMax)
3883
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
265 }
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
266
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
267 heights = common.DedupFloat64s(heights)
4685
7a9388943840 morphology: Clip class breaks again Z min and max of the the height model.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4681
diff changeset
268
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
269 log.Infof("num heights: %d\n", len(heights))
3883
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
270
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
271 var isoStmt *sql.Stmt
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
272 if isoStmt, err = tx.PrepareContext(ctx, insertDiffIsoAreasQL); err != nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
273 return 0, err
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
274 }
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
275 defer isoStmt.Close()
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
276
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
277 var id int64
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
278
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
279 if err = tx.QueryRowContext(
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
280 ctx,
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
281 insertDiffSQL,
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
282 dci.Bottleneck,
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
283 dci.Minuend.Time,
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
284 dci.Subtrahend.Time,
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
285 ).Scan(&id); err != nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
286 return 0, err
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
287 }
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
288
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
289 areas := raster.Trace(heights)
4256
4ca884dfc470 Prevent heights duplicates in iso line calculations.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4244
diff changeset
290
4768
a2f16bbcc846 Morph differences: Directly raster A and subtract B as a raster.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4762
diff changeset
291 raster = nil
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
292
4639
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
293 var size int
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
294
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
295 for i, a := range areas {
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
296 if len(a) == 0 {
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
297 continue
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
298 }
4639
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
299 wkb := a.AsWKB()
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
300 size += len(wkb)
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
301 if _, err = isoStmt.ExecContext(
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
302 ctx,
4641
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
303 id, heights[i], epsg,
4639
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
304 wkb,
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
305 contourTolerance,
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
306 ); err != nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
307 return 0, err
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
308 }
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
309 }
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
310
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
311 log.Infof("Transferred WKB size: %.2fMB.\n",
4639
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
312 float64(size)/(1024*1024))
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
313
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
314 log.Infof("calculating and storing iso areas took %v\n",
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
315 time.Since(start))
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
316
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
317 if err = tx.Commit(); err != nil {
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
318 log.Infof("difference calculation failed after %v\n",
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
319 time.Since(begin))
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
320 return 0, err
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
321 }
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
322 log.Infof("difference calculation succeed after %v\n",
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
323 time.Since(begin))
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
324
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
325 return id, nil
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
326 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
327
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
328 func (dcm *diffCalculationManager) calculate() {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
329 for dci := range dcm.jobs {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
330 ctx := context.Background()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
331 var id int64
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
332 var err error
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
333 err = auth.RunAs(ctx, "sys_admin", func(conn *sql.Conn) error {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
334 id, err = doDiff(ctx, conn, dci)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
335 return err
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
336 })
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
337 dcm.cmds <- func(dcm *diffCalculationManager) {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
338 msg := diffResult{id: id, err: err}
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
339 key := diffCalcHash(dci)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
340 list := dcm.waiting[key]
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
341 delete(dcm.waiting, key)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
342 for _, ch := range list {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
343 ch <- msg
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
344 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
345 }
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
346 }
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
347 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
348
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
349 func (dcm *diffCalculationManager) enque(dci *models.DiffCalculationInput) chan diffResult {
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
350 // buffer to prevent blocking even if receiver is dead.
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
351 ch := make(chan diffResult, 1)
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
352
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
353 dcm.cmds <- func(dcm *diffCalculationManager) {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
354 // When already running add to list of waitings
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
355 key := diffCalcHash(dci)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
356 if list := dcm.waiting[key]; list != nil {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
357 dcm.waiting[key] = append(list, ch)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
358 return
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
359 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
360 // look if it is in database
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
361 var id int64
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
362 ctx := context.Background()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
363
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
364 err := auth.RunAs(ctx, "sys_admin", func(conn *sql.Conn) error {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
365 return conn.QueryRowContext(
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
366 ctx,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
367 diffIDSQL,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
368 dci.Bottleneck,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
369 dci.Minuend.Time,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
370 dci.Subtrahend.Time,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
371 ).Scan(&id)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
372 })
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
373
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
374 switch {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
375 case err == sql.ErrNoRows:
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
376 // We need to calculate it.
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
377 case err != nil: // error
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
378 ch <- diffResult{err: err}
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
379 return
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
380 default: // we have it
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
381 ch <- diffResult{id: id}
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
382 return
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
383 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
384
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
385 // add to waiting
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
386 dcm.waiting[key] = []chan diffResult{ch}
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
387 // feed to workers
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
388 select {
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
389 case dcm.jobs <- dci:
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
390 default: // buffer is full
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
391 delete(dcm.waiting, key)
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
392 ch <- diffResult{err: errBufferFull}
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
393 }
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
394 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
395
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
396 return ch
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
397 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
398
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
399 func (dcm *diffCalculationManager) unregister(dci *models.DiffCalculationInput, ch chan diffResult) {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
400 dcm.cmds <- func(dcm *diffCalculationManager) {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
401 key := diffCalcHash(dci)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
402 list := dcm.waiting[key]
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
403 for i, x := range list {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
404 if x == ch {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
405 if i != len(list)-1 {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
406 copy(list[i:], list[i+1:])
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
407 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
408 list[len(list)-1] = nil
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
409 list = list[:len(list)-1]
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
410 if len(list) == 0 {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
411 delete(dcm.waiting, key)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
412 } else {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
413 dcm.waiting[key] = list
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
414 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
415 return
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
416 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
417 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
418 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
419 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
420
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
421 func diffCalculation(req *http.Request) (jr mw.JSONResult, err error) {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
422
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
423 dci := mw.JSONInput(req).(*models.DiffCalculationInput)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
424
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
425 ctx := req.Context()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
426
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
427 resultCh := diffCalcMng.enque(dci)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
428 defer diffCalcMng.unregister(dci, resultCh)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
429
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
430 select {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
431 case res := <-resultCh:
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
432 switch {
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
433 case res.err == errBufferFull:
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
434 err = mw.JSONError{
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
435 Code: http.StatusTooManyRequests,
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
436 Message: "Too many differences are waiting to be processed",
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
437 }
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
438 case res.err != nil:
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
439 err = res.err
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
440 default:
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
441 jr = mw.JSONResult{
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
442 Result: map[string]int64{"id": res.id},
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
443 }
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
444 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
445 case <-ctx.Done():
5490
5f47eeea988d Use own logging package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5412
diff changeset
446 log.Infoln("request canceled")
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
447 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
448
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
449 return
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
450 }