Mercurial > gemma

annotate pkg/controllers/diff.go @ 4762:c93c8a837af8

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Removed unused code that interplolated height levels between class breaks.
author Sascha L. Teichmann <teichmann@intevation.de>
date Sat, 19 Oct 2019 18:34:17 +0200
parents c91e759007da
children a2f16bbcc846
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
3 //
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
6 //
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
7 // Copyright (C) 2018, 2019 by via donau
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
10 //
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
11 // Author(s):
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
13
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
14 package controllers
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
15
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
16 import (
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
17 "context"
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
18 "database/sql"
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
19 "errors"
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
20 "fmt"
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
21 "log"
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
22 "net/http"
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
23 "runtime"
2574
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
24 "sync"
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
25 "time"
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
26
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
27 "gemma.intevation.de/gemma/pkg/auth"
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
28 "gemma.intevation.de/gemma/pkg/common"
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
29 "gemma.intevation.de/gemma/pkg/models"
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
30 "gemma.intevation.de/gemma/pkg/octree"
4244
4394daeea96a Moved JSONHandler into middleware package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4243
diff changeset
31
4394daeea96a Moved JSONHandler into middleware package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4243
diff changeset
32 mw "gemma.intevation.de/gemma/pkg/middleware"
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
33 )
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
34
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
35 const (
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
36 // maxNumberWorkers are processing diffs at a time.
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
37 maxNumberWorkers = 2
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
38 // maxQueueLength are the number of diffs to
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
39 // be wait for when idle.
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
40 maxQueueLength = 5
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
41 )
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
42
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
43 const (
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
44 contourTolerance = 0.1
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
45 contourStep = 0.1
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
46 )
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
47
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
48 const (
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
49 // isoCellSize is the side length of a raster cell when tracing
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
50 // iso areas.
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
51 isoCellSize = 0.5
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
52 )
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
53
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
54 const (
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
55 diffIDSQL = `
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
56 SELECT sd.id FROM
3656
2a079d0a71c1 Ensure sounding results are associated to matching bottleneck version
Tom Gottfried <tom@intevation.de>
parents: 2587
diff changeset
57 caching.sounding_differences sd JOIN
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
58 waterway.sounding_results srm ON sd.minuend = srm.id JOIN
2587
857bb070b9f1 Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2585
diff changeset
59 waterway.sounding_results srs ON sd.subtrahend = srs.id
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
60 WHERE srm.bottleneck_id = srs.bottleneck_id AND
2587
857bb070b9f1 Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2585
diff changeset
61 srm.bottleneck_id = $1 AND
857bb070b9f1 Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2585
diff changeset
62 srm.date_info = $2::date AND
857bb070b9f1 Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2585
diff changeset
63 srs.date_info = $3::date
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
64 `
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
65 insertDiffSQL = `
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
66 WITH soundings AS (
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
67 SELECT sr.id AS id, sr.date_info AS date_info FROM
2587
857bb070b9f1 Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2585
diff changeset
68 waterway.sounding_results sr
857bb070b9f1 Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2585
diff changeset
69 WHERE sr.bottleneck_id = $1
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
70 )
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
71 INSERT INTO caching.sounding_differences (minuend, subtrahend)
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
72 SELECT m.id, s.id FROM soundings m, soundings s
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
73 WHERE m.date_info = $2::date AND s.date_info = $3::date
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
74 RETURNING id
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
75 `
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
76 insertDiffIsoAreasQL = `
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
77 INSERT INTO caching.sounding_differences_iso_areas (
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
78 sounding_differences_id,
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
79 height,
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
80 areas
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
81 )
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
82 SELECT
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
83 $1,
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
84 $2,
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
85 ST_Transform(
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
86 ST_Multi(
4726
c91e759007da Started with stacking shells and holes of generated polygons correctly.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4686
diff changeset
87 ST_Collectionextract(
c91e759007da Started with stacking shells and holes of generated polygons correctly.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4686
diff changeset
88 ST_SimplifyPreserveTopology(ST_GeomFromWKB($4, $3::integer), $5),
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
89 3
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
90 )
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
91 ),
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
92 4326
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
93 )
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
94 `
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
95 )
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
96
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
97 type (
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
98 diffResult struct {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
99 id int64
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
100 err error
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
101 }
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
102
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
103 diffCalculationManager struct {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
104 waiting map[string][]chan diffResult
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
105 cmds chan func(*diffCalculationManager)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
106 jobs chan *models.DiffCalculationInput
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
107 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
108 )
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
109
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
110 var errBufferFull = errors.New("buffer full")
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
111
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
112 var diffCalcMng = startDiffCalculationManager()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
113
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
114 func diffCalcHash(dci *models.DiffCalculationInput) string {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
115 return dci.Bottleneck + "/" +
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
116 dci.Minuend.Format(common.DateFormat) + "/" +
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
117 dci.Subtrahend.Format(common.DateFormat)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
118 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
119
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
120 func startDiffCalculationManager() *diffCalculationManager {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
121 dcm := diffCalculationManager{
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
122 waiting: make(map[string][]chan diffResult),
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
123 cmds: make(chan func(*diffCalculationManager)),
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
124 jobs: make(chan *models.DiffCalculationInput, maxQueueLength),
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
125 }
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
126
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
127 go dcm.run()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
128 return &dcm
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
129 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
130
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
131 func (dcm *diffCalculationManager) run() {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
132 n := runtime.NumCPU()
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
133 if n > maxNumberWorkers {
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
134 n = maxNumberWorkers
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
135 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
136 for i := 0; i < n; i++ {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
137 go dcm.calculate()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
138 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
139
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
140 for cmd := range dcm.cmds {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
141 cmd(dcm)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
142 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
143 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
144
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
145 func doDiff(ctx context.Context, conn *sql.Conn, dci *models.DiffCalculationInput) (int64, error) {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
146
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
147 start := time.Now()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
148 begin := start
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
149
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
150 minuendTree, err := octree.FromCache(
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
151 ctx, conn,
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
152 dci.Bottleneck, dci.Minuend.Time)
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
153
4658
4bbfe3dd2ab5 Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4641
diff changeset
154 log.Printf("info: loading minuend mesh took %s\n", time.Since(start))
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
155 if err != nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
156 return 0, err
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
157 }
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
158
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
159 if minuendTree == nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
160 return 0, fmt.Errorf("Cannot find survey for %s/%s.",
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
161 dci.Bottleneck,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
162 dci.Minuend.Format(common.DateFormat))
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
163 }
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
164
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
165 start = time.Now()
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
166
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
167 subtrahendTree, err := octree.FromCache(
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
168 ctx, conn,
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
169 dci.Bottleneck, dci.Subtrahend.Time)
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
170
4658
4bbfe3dd2ab5 Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4641
diff changeset
171 log.Printf("info: loading subtrahend mesh took %s\n", time.Since(start))
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
172 if err != nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
173 return 0, err
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
174 }
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
175
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
176 if subtrahendTree == nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
177 return 0, fmt.Errorf("Cannot find survey for %s/%s.",
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
178 dci.Bottleneck,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
179 dci.Subtrahend.Format(common.DateFormat))
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
180 }
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
181
2572
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
182 // We need a slow path implementation for this.
4658
4bbfe3dd2ab5 Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4641
diff changeset
183 epsg := minuendTree.EPSG()
4bbfe3dd2ab5 Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4641
diff changeset
184 if epsg != subtrahendTree.EPSG() {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
185 return 0, errors.New("Calculating differences between two different " +
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
186 "EPSG code meshes are not supported, yet.")
2572
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
187 }
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
188
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
189 start = time.Now()
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
190 points := minuendTree.Diff(subtrahendTree)
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
191 log.Printf("info: A - B took %v\n", time.Since(start))
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
192
4641
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
193 minuendTree, subtrahendTree = nil, nil
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
194
2574
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
195 // The Triangulation and the loading of the clipping
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
196 // polygon can be done concurrently.
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
197
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
198 jobs := make(chan func())
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
199
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
200 wg := new(sync.WaitGroup)
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
201 for i := 0; i < 2; i++ {
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
202 wg.Add(1)
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
203 go func() {
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
204 defer wg.Done()
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
205 for job := range jobs {
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
206 job()
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
207 }
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
208 }()
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
209 }
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
210
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
211 var (
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
212 tri *octree.Triangulation
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
213 triErr error
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
214 clip *octree.Polygon
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
215 clipErr error
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
216 )
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
217
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
218 jobs <- func() {
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
219 start := time.Now()
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
220 tri, triErr = points.Triangulate()
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
221 log.Printf("info: triangulation took %v\n", time.Since(start))
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
222 }
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
223
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
224 jobs <- func() {
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
225 start := time.Now()
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
226 clip, clipErr = octree.LoadClippingPolygon(
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
227 ctx, conn,
4641
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
228 epsg,
2574
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
229 dci.Bottleneck,
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
230 dci.Minuend.Time,
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
231 dci.Subtrahend.Time)
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
232 log.Printf("info: loading clipping polygon took %v\n", time.Since(start))
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
233 }
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
234 close(jobs)
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
235 wg.Wait()
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
236
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
237 switch {
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
238 case triErr != nil && clipErr != nil:
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
239 return 0, fmt.Errorf("%v %v", triErr, clipErr)
2574
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
240 case triErr != nil:
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
241 return 0, triErr
2574
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
242 case clipErr != nil:
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
243 return 0, clipErr
2572
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
244 }
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
245
2575
59e7a011d347 Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2574
diff changeset
246 start = time.Now()
59e7a011d347 Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2574
diff changeset
247 tin := tri.Tin()
59e7a011d347 Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2574
diff changeset
248 removed := tin.Clip(clip)
4641
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
249 clip = nil
2575
59e7a011d347 Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2574
diff changeset
250 log.Printf("info: clipping TIN took %v\n", time.Since(start))
59e7a011d347 Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2574
diff changeset
251
4641
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
252 log.Printf("info: Number of triangles to clip: %d\n", len(removed))
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
253
2575
59e7a011d347 Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2574
diff changeset
254 start = time.Now()
4641
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
255 var tree octree.STRTree
2575
59e7a011d347 Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2574
diff changeset
256
4641
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
257 tree.BuildWithout(tin, removed)
2576
647a58ee9ae9 Morphological differences: Centralized generation of height values for differences in octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2575
diff changeset
258
4658
4bbfe3dd2ab5 Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4641
diff changeset
259 log.Printf("info: Building final mesh took: %v\n", time.Since(start))
2576
647a58ee9ae9 Morphological differences: Centralized generation of height values for differences in octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2575
diff changeset
260
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
261 start = time.Now()
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
262
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
263 // XXX: Maybe we should start this transaction earlier!?
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
264 var tx *sql.Tx
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
265 if tx, err = conn.BeginTx(ctx, nil); err != nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
266 return 0, err
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
267 }
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
268 defer tx.Rollback()
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
269
3883
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
270 var heights []float64
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
271
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
272 heights, err = octree.LoadClassBreaks(
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
273 ctx, tx,
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
274 "morphology_classbreaks_compare")
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
275 if err != nil {
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
276 log.Printf("warn: Loading class breaks failed: %v\n", err)
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
277 err = nil
4641
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
278 heights = octree.SampleDiffHeights(tin.Min.Z, tin.Max.Z, contourStep)
3883
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
279 } else {
4641
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
280 heights = octree.ExtrapolateClassBreaks(heights, tin.Min.Z, tin.Max.Z)
3883
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
281 }
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
282
4685
7a9388943840 morphology: Clip class breaks again Z min and max of the the height model.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4681
diff changeset
283 log.Printf("info: z range: %.3f - %.3f\n", tin.Min.Z, tin.Max.Z)
7a9388943840 morphology: Clip class breaks again Z min and max of the the height model.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4681
diff changeset
284
3883
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
285 log.Printf("info: num heights: %d\n", len(heights))
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
286
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
287 var isoStmt *sql.Stmt
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
288 if isoStmt, err = tx.PrepareContext(ctx, insertDiffIsoAreasQL); err != nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
289 return 0, err
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
290 }
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
291 defer isoStmt.Close()
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
292
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
293 var id int64
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
294
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
295 if err = tx.QueryRowContext(
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
296 ctx,
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
297 insertDiffSQL,
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
298 dci.Bottleneck,
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
299 dci.Minuend.Time,
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
300 dci.Subtrahend.Time,
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
301 ).Scan(&id); err != nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
302 return 0, err
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
303 }
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
304
4256
4ca884dfc470 Prevent heights duplicates in iso line calculations.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4244
diff changeset
305 heights = common.DedupFloat64s(heights)
4ca884dfc470 Prevent heights duplicates in iso line calculations.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4244
diff changeset
306
4641
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
307 areas := octree.TraceAreas(heights, isoCellSize, tin.Min, tin.Max, tree.Value)
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
308
4639
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
309 var size int
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
310
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
311 for i, a := range areas {
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
312 if len(a) == 0 {
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
313 continue
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
314 }
4639
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
315 wkb := a.AsWKB()
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
316 size += len(wkb)
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
317 if _, err = isoStmt.ExecContext(
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
318 ctx,
4641
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
319 id, heights[i], epsg,
4639
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
320 wkb,
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
321 contourTolerance,
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
322 ); err != nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
323 return 0, err
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
324 }
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
325 }
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
326
4639
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
327 log.Printf("info: Transferred WKB size: %.2fMB.\n",
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
328 float64(size)/(1024*1024))
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
329
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
330 log.Printf("info: calculating and storing iso lines took %v\n",
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
331 time.Since(start))
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
332
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
333 if err = tx.Commit(); err != nil {
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
334 log.Printf("info: difference calculation failed after %v\n",
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
335 time.Since(begin))
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
336 return 0, err
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
337 }
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
338 log.Printf("info: difference calculation succeed after %v\n",
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
339 time.Since(begin))
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
340
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
341 return id, nil
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
342 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
343
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
344 func (dcm *diffCalculationManager) calculate() {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
345 for dci := range dcm.jobs {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
346 ctx := context.Background()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
347 var id int64
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
348 var err error
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
349 err = auth.RunAs(ctx, "sys_admin", func(conn *sql.Conn) error {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
350 id, err = doDiff(ctx, conn, dci)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
351 return err
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
352 })
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
353 dcm.cmds <- func(dcm *diffCalculationManager) {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
354 msg := diffResult{id: id, err: err}
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
355 key := diffCalcHash(dci)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
356 list := dcm.waiting[key]
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
357 delete(dcm.waiting, key)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
358 for _, ch := range list {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
359 ch <- msg
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
360 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
361 }
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
362 }
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
363 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
364
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
365 func (dcm *diffCalculationManager) enque(dci *models.DiffCalculationInput) chan diffResult {
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
366 // buffer to prevent blocking even if receiver is dead.
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
367 ch := make(chan diffResult, 1)
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
368
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
369 dcm.cmds <- func(dcm *diffCalculationManager) {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
370 // When already running add to list of waitings
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
371 key := diffCalcHash(dci)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
372 if list := dcm.waiting[key]; list != nil {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
373 dcm.waiting[key] = append(list, ch)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
374 return
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
375 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
376 // look if it is in database
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
377 var id int64
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
378 ctx := context.Background()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
379
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
380 err := auth.RunAs(ctx, "sys_admin", func(conn *sql.Conn) error {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
381 return conn.QueryRowContext(
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
382 ctx,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
383 diffIDSQL,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
384 dci.Bottleneck,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
385 dci.Minuend.Time,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
386 dci.Subtrahend.Time,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
387 ).Scan(&id)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
388 })
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
389
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
390 switch {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
391 case err == sql.ErrNoRows:
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
392 // We need to calculate it.
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
393 case err != nil: // error
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
394 ch <- diffResult{err: err}
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
395 return
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
396 default: // we have it
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
397 ch <- diffResult{id: id}
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
398 return
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
399 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
400
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
401 // add to waiting
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
402 dcm.waiting[key] = []chan diffResult{ch}
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
403 // feed to workers
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
404 select {
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
405 case dcm.jobs <- dci:
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
406 default: // buffer is full
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
407 delete(dcm.waiting, key)
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
408 ch <- diffResult{err: errBufferFull}
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
409 }
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
410 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
411
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
412 return ch
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
413 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
414
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
415 func (dcm *diffCalculationManager) unregister(dci *models.DiffCalculationInput, ch chan diffResult) {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
416 dcm.cmds <- func(dcm *diffCalculationManager) {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
417 key := diffCalcHash(dci)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
418 list := dcm.waiting[key]
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
419 for i, x := range list {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
420 if x == ch {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
421 if i != len(list)-1 {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
422 copy(list[i:], list[i+1:])
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
423 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
424 list[len(list)-1] = nil
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
425 list = list[:len(list)-1]
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
426 if len(list) == 0 {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
427 delete(dcm.waiting, key)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
428 } else {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
429 dcm.waiting[key] = list
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
430 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
431 return
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
432 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
433 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
434 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
435 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
436
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
437 func diffCalculation(req *http.Request) (jr mw.JSONResult, err error) {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
438
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
439 dci := mw.JSONInput(req).(*models.DiffCalculationInput)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
440
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
441 ctx := req.Context()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
442
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
443 resultCh := diffCalcMng.enque(dci)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
444 defer diffCalcMng.unregister(dci, resultCh)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
445
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
446 select {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
447 case res := <-resultCh:
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
448 switch {
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
449 case res.err == errBufferFull:
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
450 err = mw.JSONError{
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
451 Code: http.StatusTooManyRequests,
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
452 Message: "Too many differences are waiting to be processed",
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
453 }
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
454 case res.err != nil:
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
455 err = res.err
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
456 default:
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
457 jr = mw.JSONResult{
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
458 Result: map[string]int64{"id": res.id},
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
459 }
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
460 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
461 case <-ctx.Done():
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
462 log.Println("request canceled")
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
463 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
464
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
465 return
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
466 }