Mercurial > gemma

annotate pkg/controllers/diff.go @ 4686:1d6f28e45696

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
morphology: Doing ST_SimplifyPreserveTopology before ST_MakeValid when storing geometries to database speeds things up ... a lot.
author Sascha L. Teichmann <sascha.teichmann@intevation.de>
date Wed, 16 Oct 2019 11:51:53 +0200
parents 7a9388943840
children c91e759007da
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
3 //
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
6 //
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
7 // Copyright (C) 2018, 2019 by via donau
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
10 //
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
11 // Author(s):
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
13
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
14 package controllers
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
15
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
16 import (
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
17 "context"
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
18 "database/sql"
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
19 "errors"
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
20 "fmt"
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
21 "log"
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
22 "net/http"
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
23 "runtime"
2574
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
24 "sync"
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
25 "time"
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
26
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
27 "gemma.intevation.de/gemma/pkg/auth"
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
28 "gemma.intevation.de/gemma/pkg/common"
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
29 "gemma.intevation.de/gemma/pkg/models"
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
30 "gemma.intevation.de/gemma/pkg/octree"
4244
4394daeea96a Moved JSONHandler into middleware package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4243
diff changeset
31
4394daeea96a Moved JSONHandler into middleware package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4243
diff changeset
32 mw "gemma.intevation.de/gemma/pkg/middleware"
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
33 )
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
34
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
35 const (
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
36 // maxNumberWorkers are processing diffs at a time.
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
37 maxNumberWorkers = 2
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
38 // maxQueueLength are the number of diffs to
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
39 // be wait for when idle.
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
40 maxQueueLength = 5
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
41 )
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
42
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
43 const (
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
44 contourTolerance = 0.1
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
45 contourStep = 0.1
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
46 )
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
47
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
48 const (
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
49 // isoCellSize is the side length of a raster cell when tracing
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
50 // iso areas.
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
51 isoCellSize = 0.5
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
52 )
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
53
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
54 const (
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
55 diffIDSQL = `
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
56 SELECT sd.id FROM
3656
2a079d0a71c1 Ensure sounding results are associated to matching bottleneck version
Tom Gottfried <tom@intevation.de>
parents: 2587
diff changeset
57 caching.sounding_differences sd JOIN
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
58 waterway.sounding_results srm ON sd.minuend = srm.id JOIN
2587
857bb070b9f1 Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2585
diff changeset
59 waterway.sounding_results srs ON sd.subtrahend = srs.id
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
60 WHERE srm.bottleneck_id = srs.bottleneck_id AND
2587
857bb070b9f1 Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2585
diff changeset
61 srm.bottleneck_id = $1 AND
857bb070b9f1 Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2585
diff changeset
62 srm.date_info = $2::date AND
857bb070b9f1 Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2585
diff changeset
63 srs.date_info = $3::date
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
64 `
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
65 insertDiffSQL = `
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
66 WITH soundings AS (
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
67 SELECT sr.id AS id, sr.date_info AS date_info FROM
2587
857bb070b9f1 Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2585
diff changeset
68 waterway.sounding_results sr
857bb070b9f1 Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2585
diff changeset
69 WHERE sr.bottleneck_id = $1
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
70 )
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
71 INSERT INTO caching.sounding_differences (minuend, subtrahend)
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
72 SELECT m.id, s.id FROM soundings m, soundings s
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
73 WHERE m.date_info = $2::date AND s.date_info = $3::date
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
74 RETURNING id
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
75 `
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
76 insertDiffIsoAreasQL = `
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
77 INSERT INTO caching.sounding_differences_iso_areas (
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
78 sounding_differences_id,
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
79 height,
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
80 areas
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
81 )
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
82 SELECT
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
83 $1,
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
84 $2,
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
85 ST_Transform(
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
86 ST_Multi(
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
87 ST_CollectionExtract(
4686
1d6f28e45696 morphology: Doing ST_SimplifyPreserveTopology before ST_MakeValid when storing geometries to database speeds things up ... a lot.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4685
diff changeset
88 ST_MakeValid(
1d6f28e45696 morphology: Doing ST_SimplifyPreserveTopology before ST_MakeValid when storing geometries to database speeds things up ... a lot.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4685
diff changeset
89 ST_Multi(
1d6f28e45696 morphology: Doing ST_SimplifyPreserveTopology before ST_MakeValid when storing geometries to database speeds things up ... a lot.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4685
diff changeset
90 ST_Collectionextract(
1d6f28e45696 morphology: Doing ST_SimplifyPreserveTopology before ST_MakeValid when storing geometries to database speeds things up ... a lot.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4685
diff changeset
91 ST_SimplifyPreserveTopology(ST_GeomFromWKB($4, $3::integer), $5),
1d6f28e45696 morphology: Doing ST_SimplifyPreserveTopology before ST_MakeValid when storing geometries to database speeds things up ... a lot.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4685
diff changeset
92 3
1d6f28e45696 morphology: Doing ST_SimplifyPreserveTopology before ST_MakeValid when storing geometries to database speeds things up ... a lot.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4685
diff changeset
93 )
1d6f28e45696 morphology: Doing ST_SimplifyPreserveTopology before ST_MakeValid when storing geometries to database speeds things up ... a lot.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4685
diff changeset
94 )
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
95 ),
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
96 3
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
97 )
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
98 ),
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
99 4326
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
100 )
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
101 `
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
102 )
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
103
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
104 type (
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
105 diffResult struct {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
106 id int64
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
107 err error
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
108 }
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
109
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
110 diffCalculationManager struct {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
111 waiting map[string][]chan diffResult
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
112 cmds chan func(*diffCalculationManager)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
113 jobs chan *models.DiffCalculationInput
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
114 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
115 )
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
116
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
117 var errBufferFull = errors.New("buffer full")
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
118
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
119 var diffCalcMng = startDiffCalculationManager()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
120
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
121 func diffCalcHash(dci *models.DiffCalculationInput) string {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
122 return dci.Bottleneck + "/" +
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
123 dci.Minuend.Format(common.DateFormat) + "/" +
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
124 dci.Subtrahend.Format(common.DateFormat)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
125 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
126
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
127 func startDiffCalculationManager() *diffCalculationManager {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
128 dcm := diffCalculationManager{
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
129 waiting: make(map[string][]chan diffResult),
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
130 cmds: make(chan func(*diffCalculationManager)),
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
131 jobs: make(chan *models.DiffCalculationInput, maxQueueLength),
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
132 }
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
133
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
134 go dcm.run()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
135 return &dcm
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
136 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
137
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
138 func (dcm *diffCalculationManager) run() {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
139 n := runtime.NumCPU()
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
140 if n > maxNumberWorkers {
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
141 n = maxNumberWorkers
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
142 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
143 for i := 0; i < n; i++ {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
144 go dcm.calculate()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
145 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
146
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
147 for cmd := range dcm.cmds {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
148 cmd(dcm)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
149 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
150 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
151
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
152 func doDiff(ctx context.Context, conn *sql.Conn, dci *models.DiffCalculationInput) (int64, error) {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
153
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
154 start := time.Now()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
155 begin := start
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
156
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
157 minuendTree, err := octree.FromCache(
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
158 ctx, conn,
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
159 dci.Bottleneck, dci.Minuend.Time)
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
160
4658
4bbfe3dd2ab5 Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4641
diff changeset
161 log.Printf("info: loading minuend mesh took %s\n", time.Since(start))
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
162 if err != nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
163 return 0, err
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
164 }
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
165
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
166 if minuendTree == nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
167 return 0, fmt.Errorf("Cannot find survey for %s/%s.",
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
168 dci.Bottleneck,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
169 dci.Minuend.Format(common.DateFormat))
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
170 }
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
171
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
172 start = time.Now()
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
173
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
174 subtrahendTree, err := octree.FromCache(
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
175 ctx, conn,
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
176 dci.Bottleneck, dci.Subtrahend.Time)
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
177
4658
4bbfe3dd2ab5 Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4641
diff changeset
178 log.Printf("info: loading subtrahend mesh took %s\n", time.Since(start))
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
179 if err != nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
180 return 0, err
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
181 }
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
182
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
183 if subtrahendTree == nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
184 return 0, fmt.Errorf("Cannot find survey for %s/%s.",
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
185 dci.Bottleneck,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
186 dci.Subtrahend.Format(common.DateFormat))
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
187 }
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
188
2572
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
189 // We need a slow path implementation for this.
4658
4bbfe3dd2ab5 Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4641
diff changeset
190 epsg := minuendTree.EPSG()
4bbfe3dd2ab5 Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4641
diff changeset
191 if epsg != subtrahendTree.EPSG() {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
192 return 0, errors.New("Calculating differences between two different " +
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
193 "EPSG code meshes are not supported, yet.")
2572
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
194 }
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
195
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
196 start = time.Now()
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
197 points := minuendTree.Diff(subtrahendTree)
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
198 log.Printf("info: A - B took %v\n", time.Since(start))
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
199
4641
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
200 minuendTree, subtrahendTree = nil, nil
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
201
2574
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
202 // The Triangulation and the loading of the clipping
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
203 // polygon can be done concurrently.
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
204
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
205 jobs := make(chan func())
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
206
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
207 wg := new(sync.WaitGroup)
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
208 for i := 0; i < 2; i++ {
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
209 wg.Add(1)
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
210 go func() {
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
211 defer wg.Done()
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
212 for job := range jobs {
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
213 job()
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
214 }
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
215 }()
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
216 }
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
217
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
218 var (
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
219 tri *octree.Triangulation
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
220 triErr error
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
221 clip *octree.Polygon
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
222 clipErr error
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
223 )
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
224
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
225 jobs <- func() {
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
226 start := time.Now()
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
227 tri, triErr = points.Triangulate()
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
228 log.Printf("info: triangulation took %v\n", time.Since(start))
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
229 }
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
230
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
231 jobs <- func() {
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
232 start := time.Now()
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
233 clip, clipErr = octree.LoadClippingPolygon(
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
234 ctx, conn,
4641
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
235 epsg,
2574
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
236 dci.Bottleneck,
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
237 dci.Minuend.Time,
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
238 dci.Subtrahend.Time)
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
239 log.Printf("info: loading clipping polygon took %v\n", time.Since(start))
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
240 }
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
241 close(jobs)
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
242 wg.Wait()
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
243
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
244 switch {
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
245 case triErr != nil && clipErr != nil:
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
246 return 0, fmt.Errorf("%v %v", triErr, clipErr)
2574
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
247 case triErr != nil:
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
248 return 0, triErr
2574
2833ff156cb2 Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2572
diff changeset
249 case clipErr != nil:
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
250 return 0, clipErr
2572
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
251 }
7686c7c23506 Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2570
diff changeset
252
2575
59e7a011d347 Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2574
diff changeset
253 start = time.Now()
59e7a011d347 Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2574
diff changeset
254 tin := tri.Tin()
59e7a011d347 Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2574
diff changeset
255 removed := tin.Clip(clip)
4641
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
256 clip = nil
2575
59e7a011d347 Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2574
diff changeset
257 log.Printf("info: clipping TIN took %v\n", time.Since(start))
59e7a011d347 Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2574
diff changeset
258
4641
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
259 log.Printf("info: Number of triangles to clip: %d\n", len(removed))
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
260
2575
59e7a011d347 Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2574
diff changeset
261 start = time.Now()
4641
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
262 var tree octree.STRTree
2575
59e7a011d347 Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2574
diff changeset
263
4641
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
264 tree.BuildWithout(tin, removed)
2576
647a58ee9ae9 Morphological differences: Centralized generation of height values for differences in octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2575
diff changeset
265
4658
4bbfe3dd2ab5 Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4641
diff changeset
266 log.Printf("info: Building final mesh took: %v\n", time.Since(start))
2576
647a58ee9ae9 Morphological differences: Centralized generation of height values for differences in octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2575
diff changeset
267
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
268 start = time.Now()
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
269
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
270 // XXX: Maybe we should start this transaction earlier!?
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
271 var tx *sql.Tx
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
272 if tx, err = conn.BeginTx(ctx, nil); err != nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
273 return 0, err
2580
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
274 }
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
275 defer tx.Rollback()
5125db802b79 Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2576
diff changeset
276
3883
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
277 var heights []float64
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
278
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
279 heights, err = octree.LoadClassBreaks(
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
280 ctx, tx,
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
281 "morphology_classbreaks_compare")
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
282 if err != nil {
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
283 log.Printf("warn: Loading class breaks failed: %v\n", err)
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
284 err = nil
4641
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
285 heights = octree.SampleDiffHeights(tin.Min.Z, tin.Max.Z, contourStep)
3883
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
286 } else {
4641
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
287 heights = octree.ExtrapolateClassBreaks(heights, tin.Min.Z, tin.Max.Z)
3954
cb4fda122321 Completing: Don't create extra inbetween class breaks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3883
diff changeset
288 // heights = octree.InBetweenClassBreaks(heights, 0.05, 2)
3883
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
289 }
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
290
4685
7a9388943840 morphology: Clip class breaks again Z min and max of the the height model.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4681
diff changeset
291 log.Printf("info: z range: %.3f - %.3f\n", tin.Min.Z, tin.Max.Z)
7a9388943840 morphology: Clip class breaks again Z min and max of the the height model.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4681
diff changeset
292
3883
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
293 log.Printf("info: num heights: %d\n", len(heights))
0b382767b5ab Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3656
diff changeset
294
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
295 var isoStmt *sql.Stmt
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
296 if isoStmt, err = tx.PrepareContext(ctx, insertDiffIsoAreasQL); err != nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
297 return 0, err
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
298 }
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
299 defer isoStmt.Close()
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
300
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
301 var id int64
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
302
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
303 if err = tx.QueryRowContext(
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
304 ctx,
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
305 insertDiffSQL,
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
306 dci.Bottleneck,
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
307 dci.Minuend.Time,
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
308 dci.Subtrahend.Time,
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
309 ).Scan(&id); err != nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
310 return 0, err
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
311 }
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
312
4256
4ca884dfc470 Prevent heights duplicates in iso line calculations.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4244
diff changeset
313 heights = common.DedupFloat64s(heights)
4ca884dfc470 Prevent heights duplicates in iso line calculations.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4244
diff changeset
314
4641
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
315 areas := octree.TraceAreas(heights, isoCellSize, tin.Min, tin.Max, tree.Value)
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
316
4639
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
317 var size int
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
318
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
319 for i, a := range areas {
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
320 if len(a) == 0 {
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
321 continue
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
322 }
4639
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
323 wkb := a.AsWKB()
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
324 size += len(wkb)
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
325 if _, err = isoStmt.ExecContext(
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
326 ctx,
4641
5ef04ae34872 Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4640
diff changeset
327 id, heights[i], epsg,
4639
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
328 wkb,
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
329 contourTolerance,
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
330 ); err != nil {
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
331 return 0, err
4574
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
332 }
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
333 }
daed8a92024a Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4256
diff changeset
334
4639
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
335 log.Printf("info: Transferred WKB size: %.2fMB.\n",
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
336 float64(size)/(1024*1024))
4380f5483c03 Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4637
diff changeset
337
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
338 log.Printf("info: calculating and storing iso lines took %v\n",
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
339 time.Since(start))
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
340
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
341 if err = tx.Commit(); err != nil {
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
342 log.Printf("info: difference calculation failed after %v\n",
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
343 time.Since(begin))
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
344 return 0, err
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
345 }
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
346 log.Printf("info: difference calculation succeed after %v\n",
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
347 time.Since(begin))
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
348
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
349 return id, nil
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
350 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
351
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
352 func (dcm *diffCalculationManager) calculate() {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
353 for dci := range dcm.jobs {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
354 ctx := context.Background()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
355 var id int64
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
356 var err error
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
357 err = auth.RunAs(ctx, "sys_admin", func(conn *sql.Conn) error {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
358 id, err = doDiff(ctx, conn, dci)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
359 return err
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
360 })
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
361 dcm.cmds <- func(dcm *diffCalculationManager) {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
362 msg := diffResult{id: id, err: err}
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
363 key := diffCalcHash(dci)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
364 list := dcm.waiting[key]
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
365 delete(dcm.waiting, key)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
366 for _, ch := range list {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
367 ch <- msg
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
368 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
369 }
2583
5c6a7e69b02d Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2580
diff changeset
370 }
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
371 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
372
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
373 func (dcm *diffCalculationManager) enque(dci *models.DiffCalculationInput) chan diffResult {
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
374 // buffer to prevent blocking even if receiver is dead.
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
375 ch := make(chan diffResult, 1)
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
376
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
377 dcm.cmds <- func(dcm *diffCalculationManager) {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
378 // When already running add to list of waitings
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
379 key := diffCalcHash(dci)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
380 if list := dcm.waiting[key]; list != nil {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
381 dcm.waiting[key] = append(list, ch)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
382 return
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
383 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
384 // look if it is in database
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
385 var id int64
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
386 ctx := context.Background()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
387
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
388 err := auth.RunAs(ctx, "sys_admin", func(conn *sql.Conn) error {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
389 return conn.QueryRowContext(
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
390 ctx,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
391 diffIDSQL,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
392 dci.Bottleneck,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
393 dci.Minuend.Time,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
394 dci.Subtrahend.Time,
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
395 ).Scan(&id)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
396 })
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
397
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
398 switch {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
399 case err == sql.ErrNoRows:
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
400 // We need to calculate it.
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
401 case err != nil: // error
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
402 ch <- diffResult{err: err}
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
403 return
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
404 default: // we have it
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
405 ch <- diffResult{id: id}
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
406 return
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
407 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
408
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
409 // add to waiting
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
410 dcm.waiting[key] = []chan diffResult{ch}
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
411 // feed to workers
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
412 select {
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
413 case dcm.jobs <- dci:
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
414 default: // buffer is full
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
415 delete(dcm.waiting, key)
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
416 ch <- diffResult{err: errBufferFull}
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
417 }
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
418 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
419
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
420 return ch
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
421 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
422
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
423 func (dcm *diffCalculationManager) unregister(dci *models.DiffCalculationInput, ch chan diffResult) {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
424 dcm.cmds <- func(dcm *diffCalculationManager) {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
425 key := diffCalcHash(dci)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
426 list := dcm.waiting[key]
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
427 for i, x := range list {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
428 if x == ch {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
429 if i != len(list)-1 {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
430 copy(list[i:], list[i+1:])
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
431 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
432 list[len(list)-1] = nil
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
433 list = list[:len(list)-1]
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
434 if len(list) == 0 {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
435 delete(dcm.waiting, key)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
436 } else {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
437 dcm.waiting[key] = list
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
438 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
439 return
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
440 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
441 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
442 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
443 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
444
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
445 func diffCalculation(req *http.Request) (jr mw.JSONResult, err error) {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
446
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
447 dci := mw.JSONInput(req).(*models.DiffCalculationInput)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
448
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
449 ctx := req.Context()
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
450
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
451 resultCh := diffCalcMng.enque(dci)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
452 defer diffCalcMng.unregister(dci, resultCh)
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
453
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
454 select {
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
455 case res := <-resultCh:
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
456 switch {
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
457 case res.err == errBufferFull:
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
458 err = mw.JSONError{
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
459 Code: http.StatusTooManyRequests,
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
460 Message: "Too many differences are waiting to be processed",
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
461 }
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
462 case res.err != nil:
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
463 err = res.err
4681
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
464 default:
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
465 jr = mw.JSONResult{
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
466 Result: map[string]int64{"id": res.id},
2690933e404c Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4680
diff changeset
467 }
4680
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
468 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
469 case <-ctx.Done():
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
470 log.Println("request canceled")
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
471 }
976aedc195e5 Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 4667
diff changeset
472
2570
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
473 return
c4242b9d59fe Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
474 }