Mercurial > gemma
annotate pkg/controllers/diff.go @ 4726:c91e759007da stack-polygons
Started with stacking shells and holes of generated polygons correctly.
author | Sascha L. Teichmann <sascha.teichmann@intevation.de> |
---|---|
date | Thu, 17 Oct 2019 18:33:46 +0200 |
parents | 1d6f28e45696 |
children | c93c8a837af8 |
rev | line source |
---|---|
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
7 // Copyright (C) 2018, 2019 by via donau |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
13 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
14 package controllers |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 import ( |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
17 "context" |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
18 "database/sql" |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
19 "errors" |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
20 "fmt" |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
21 "log" |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
22 "net/http" |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
23 "runtime" |
2574
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
24 "sync" |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
25 "time" |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
26 |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
27 "gemma.intevation.de/gemma/pkg/auth" |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
28 "gemma.intevation.de/gemma/pkg/common" |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
29 "gemma.intevation.de/gemma/pkg/models" |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
30 "gemma.intevation.de/gemma/pkg/octree" |
4244
4394daeea96a
Moved JSONHandler into middleware package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4243
diff
changeset
|
31 |
4394daeea96a
Moved JSONHandler into middleware package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4243
diff
changeset
|
32 mw "gemma.intevation.de/gemma/pkg/middleware" |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
33 ) |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
34 |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
35 const ( |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
36 // maxNumberWorkers are processing diffs at a time. |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
37 maxNumberWorkers = 2 |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
38 // maxQueueLength are the number of diffs to |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
39 // be wait for when idle. |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
40 maxQueueLength = 5 |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
41 ) |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
42 |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
43 const ( |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
44 contourTolerance = 0.1 |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
45 contourStep = 0.1 |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
46 ) |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
47 |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
48 const ( |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
49 // isoCellSize is the side length of a raster cell when tracing |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
50 // iso areas. |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
51 isoCellSize = 0.5 |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
52 ) |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
53 |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
54 const ( |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
55 diffIDSQL = ` |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
56 SELECT sd.id FROM |
3656
2a079d0a71c1
Ensure sounding results are associated to matching bottleneck version
Tom Gottfried <tom@intevation.de>
parents:
2587
diff
changeset
|
57 caching.sounding_differences sd JOIN |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
58 waterway.sounding_results srm ON sd.minuend = srm.id JOIN |
2587
857bb070b9f1
Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2585
diff
changeset
|
59 waterway.sounding_results srs ON sd.subtrahend = srs.id |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
60 WHERE srm.bottleneck_id = srs.bottleneck_id AND |
2587
857bb070b9f1
Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2585
diff
changeset
|
61 srm.bottleneck_id = $1 AND |
857bb070b9f1
Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2585
diff
changeset
|
62 srm.date_info = $2::date AND |
857bb070b9f1
Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2585
diff
changeset
|
63 srs.date_info = $3::date |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
64 ` |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
65 insertDiffSQL = ` |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
66 WITH soundings AS ( |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
67 SELECT sr.id AS id, sr.date_info AS date_info FROM |
2587
857bb070b9f1
Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2585
diff
changeset
|
68 waterway.sounding_results sr |
857bb070b9f1
Morphological differences: Bottleneck_id is an integer represented as a string.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2585
diff
changeset
|
69 WHERE sr.bottleneck_id = $1 |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
70 ) |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
71 INSERT INTO caching.sounding_differences (minuend, subtrahend) |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
72 SELECT m.id, s.id FROM soundings m, soundings s |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
73 WHERE m.date_info = $2::date AND s.date_info = $3::date |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
74 RETURNING id |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
75 ` |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
76 insertDiffIsoAreasQL = ` |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
77 INSERT INTO caching.sounding_differences_iso_areas ( |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
78 sounding_differences_id, |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
79 height, |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
80 areas |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
81 ) |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
82 SELECT |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
83 $1, |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
84 $2, |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
85 ST_Transform( |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
86 ST_Multi( |
4726
c91e759007da
Started with stacking shells and holes of generated polygons correctly.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4686
diff
changeset
|
87 ST_Collectionextract( |
c91e759007da
Started with stacking shells and holes of generated polygons correctly.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4686
diff
changeset
|
88 ST_SimplifyPreserveTopology(ST_GeomFromWKB($4, $3::integer), $5), |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
89 3 |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
90 ) |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
91 ), |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
92 4326 |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
93 ) |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
94 ` |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
95 ) |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
96 |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
97 type ( |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
98 diffResult struct { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
99 id int64 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
100 err error |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
101 } |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
102 |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
103 diffCalculationManager struct { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
104 waiting map[string][]chan diffResult |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
105 cmds chan func(*diffCalculationManager) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
106 jobs chan *models.DiffCalculationInput |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
107 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
108 ) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
109 |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
110 var errBufferFull = errors.New("buffer full") |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
111 |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
112 var diffCalcMng = startDiffCalculationManager() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
113 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
114 func diffCalcHash(dci *models.DiffCalculationInput) string { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
115 return dci.Bottleneck + "/" + |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
116 dci.Minuend.Format(common.DateFormat) + "/" + |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
117 dci.Subtrahend.Format(common.DateFormat) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
118 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
119 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
120 func startDiffCalculationManager() *diffCalculationManager { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
121 dcm := diffCalculationManager{ |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
122 waiting: make(map[string][]chan diffResult), |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
123 cmds: make(chan func(*diffCalculationManager)), |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
124 jobs: make(chan *models.DiffCalculationInput, maxQueueLength), |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
125 } |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
126 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
127 go dcm.run() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
128 return &dcm |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
129 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
130 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
131 func (dcm *diffCalculationManager) run() { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
132 n := runtime.NumCPU() |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
133 if n > maxNumberWorkers { |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
134 n = maxNumberWorkers |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
135 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
136 for i := 0; i < n; i++ { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
137 go dcm.calculate() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
138 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
139 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
140 for cmd := range dcm.cmds { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
141 cmd(dcm) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
142 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
143 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
144 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
145 func doDiff(ctx context.Context, conn *sql.Conn, dci *models.DiffCalculationInput) (int64, error) { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
146 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
147 start := time.Now() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
148 begin := start |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
149 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
150 minuendTree, err := octree.FromCache( |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
151 ctx, conn, |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
152 dci.Bottleneck, dci.Minuend.Time) |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
153 |
4658
4bbfe3dd2ab5
Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4641
diff
changeset
|
154 log.Printf("info: loading minuend mesh took %s\n", time.Since(start)) |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
155 if err != nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
156 return 0, err |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
157 } |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
158 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
159 if minuendTree == nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
160 return 0, fmt.Errorf("Cannot find survey for %s/%s.", |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
161 dci.Bottleneck, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
162 dci.Minuend.Format(common.DateFormat)) |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
163 } |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
164 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
165 start = time.Now() |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
166 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
167 subtrahendTree, err := octree.FromCache( |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
168 ctx, conn, |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
169 dci.Bottleneck, dci.Subtrahend.Time) |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
170 |
4658
4bbfe3dd2ab5
Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4641
diff
changeset
|
171 log.Printf("info: loading subtrahend mesh took %s\n", time.Since(start)) |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
172 if err != nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
173 return 0, err |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
174 } |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
175 |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
176 if subtrahendTree == nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
177 return 0, fmt.Errorf("Cannot find survey for %s/%s.", |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
178 dci.Bottleneck, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
179 dci.Subtrahend.Format(common.DateFormat)) |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
180 } |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
181 |
2572
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
182 // We need a slow path implementation for this. |
4658
4bbfe3dd2ab5
Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4641
diff
changeset
|
183 epsg := minuendTree.EPSG() |
4bbfe3dd2ab5
Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4641
diff
changeset
|
184 if epsg != subtrahendTree.EPSG() { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
185 return 0, errors.New("Calculating differences between two different " + |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
186 "EPSG code meshes are not supported, yet.") |
2572
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
187 } |
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
188 |
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
189 start = time.Now() |
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
190 points := minuendTree.Diff(subtrahendTree) |
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
191 log.Printf("info: A - B took %v\n", time.Since(start)) |
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
192 |
4641
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
193 minuendTree, subtrahendTree = nil, nil |
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
194 |
2574
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
195 // The Triangulation and the loading of the clipping |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
196 // polygon can be done concurrently. |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
197 |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
198 jobs := make(chan func()) |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
199 |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
200 wg := new(sync.WaitGroup) |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
201 for i := 0; i < 2; i++ { |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
202 wg.Add(1) |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
203 go func() { |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
204 defer wg.Done() |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
205 for job := range jobs { |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
206 job() |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
207 } |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
208 }() |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
209 } |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
210 |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
211 var ( |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
212 tri *octree.Triangulation |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
213 triErr error |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
214 clip *octree.Polygon |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
215 clipErr error |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
216 ) |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
217 |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
218 jobs <- func() { |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
219 start := time.Now() |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
220 tri, triErr = points.Triangulate() |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
221 log.Printf("info: triangulation took %v\n", time.Since(start)) |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
222 } |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
223 |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
224 jobs <- func() { |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
225 start := time.Now() |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
226 clip, clipErr = octree.LoadClippingPolygon( |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
227 ctx, conn, |
4641
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
228 epsg, |
2574
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
229 dci.Bottleneck, |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
230 dci.Minuend.Time, |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
231 dci.Subtrahend.Time) |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
232 log.Printf("info: loading clipping polygon took %v\n", time.Since(start)) |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
233 } |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
234 close(jobs) |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
235 wg.Wait() |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
236 |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
237 switch { |
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
238 case triErr != nil && clipErr != nil: |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
239 return 0, fmt.Errorf("%v %v", triErr, clipErr) |
2574
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
240 case triErr != nil: |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
241 return 0, triErr |
2574
2833ff156cb2
Morphological differences: Moved loading of clipping polygon into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2572
diff
changeset
|
242 case clipErr != nil: |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
243 return 0, clipErr |
2572
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
244 } |
7686c7c23506
Morphological differences: Moved some code into octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2570
diff
changeset
|
245 |
2575
59e7a011d347
Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2574
diff
changeset
|
246 start = time.Now() |
59e7a011d347
Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2574
diff
changeset
|
247 tin := tri.Tin() |
59e7a011d347
Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2574
diff
changeset
|
248 removed := tin.Clip(clip) |
4641
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
249 clip = nil |
2575
59e7a011d347
Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2574
diff
changeset
|
250 log.Printf("info: clipping TIN took %v\n", time.Since(start)) |
59e7a011d347
Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2574
diff
changeset
|
251 |
4641
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
252 log.Printf("info: Number of triangles to clip: %d\n", len(removed)) |
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
253 |
2575
59e7a011d347
Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2574
diff
changeset
|
254 start = time.Now() |
4641
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
255 var tree octree.STRTree |
2575
59e7a011d347
Morphological differences: Moved TIN clipping to octree package and hide the technical details there.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2574
diff
changeset
|
256 |
4641
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
257 tree.BuildWithout(tin, removed) |
2576
647a58ee9ae9
Morphological differences: Centralized generation of height values for differences in octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2575
diff
changeset
|
258 |
4658
4bbfe3dd2ab5
Completed usage of STRTrees.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4641
diff
changeset
|
259 log.Printf("info: Building final mesh took: %v\n", time.Since(start)) |
2576
647a58ee9ae9
Morphological differences: Centralized generation of height values for differences in octree package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2575
diff
changeset
|
260 |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
261 start = time.Now() |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
262 |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
263 // XXX: Maybe we should start this transaction earlier!? |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
264 var tx *sql.Tx |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
265 if tx, err = conn.BeginTx(ctx, nil); err != nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
266 return 0, err |
2580
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
267 } |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
268 defer tx.Rollback() |
5125db802b79
Morphological differences: Check if we already have a diff before we start to calculate it.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2576
diff
changeset
|
269 |
3883
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
270 var heights []float64 |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
271 |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
272 heights, err = octree.LoadClassBreaks( |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
273 ctx, tx, |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
274 "morphology_classbreaks_compare") |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
275 if err != nil { |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
276 log.Printf("warn: Loading class breaks failed: %v\n", err) |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
277 err = nil |
4641
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
278 heights = octree.SampleDiffHeights(tin.Min.Z, tin.Max.Z, contourStep) |
3883
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
279 } else { |
4641
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
280 heights = octree.ExtrapolateClassBreaks(heights, tin.Min.Z, tin.Max.Z) |
3954
cb4fda122321
Completing: Don't create extra inbetween class breaks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3883
diff
changeset
|
281 // heights = octree.InBetweenClassBreaks(heights, 0.05, 2) |
3883
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
282 } |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
283 |
4685
7a9388943840
morphology: Clip class breaks again Z min and max of the the height model.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4681
diff
changeset
|
284 log.Printf("info: z range: %.3f - %.3f\n", tin.Min.Z, tin.Max.Z) |
7a9388943840
morphology: Clip class breaks again Z min and max of the the height model.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4681
diff
changeset
|
285 |
3883
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
286 log.Printf("info: num heights: %d\n", len(heights)) |
0b382767b5ab
Use custom class breaks in morpho differences.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3656
diff
changeset
|
287 |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
288 var isoStmt *sql.Stmt |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
289 if isoStmt, err = tx.PrepareContext(ctx, insertDiffIsoAreasQL); err != nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
290 return 0, err |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
291 } |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
292 defer isoStmt.Close() |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
293 |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
294 var id int64 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
295 |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
296 if err = tx.QueryRowContext( |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
297 ctx, |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
298 insertDiffSQL, |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
299 dci.Bottleneck, |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
300 dci.Minuend.Time, |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
301 dci.Subtrahend.Time, |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
302 ).Scan(&id); err != nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
303 return 0, err |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
304 } |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
305 |
4256
4ca884dfc470
Prevent heights duplicates in iso line calculations.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4244
diff
changeset
|
306 heights = common.DedupFloat64s(heights) |
4ca884dfc470
Prevent heights duplicates in iso line calculations.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4244
diff
changeset
|
307 |
4641
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
308 areas := octree.TraceAreas(heights, isoCellSize, tin.Min, tin.Max, tree.Value) |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
309 |
4639
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
310 var size int |
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
311 |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
312 for i, a := range areas { |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
313 if len(a) == 0 { |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
314 continue |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
315 } |
4639
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
316 wkb := a.AsWKB() |
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
317 size += len(wkb) |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
318 if _, err = isoStmt.ExecContext( |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
319 ctx, |
4641
5ef04ae34872
Used STRTree for caluculating the diff.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4640
diff
changeset
|
320 id, heights[i], epsg, |
4639
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
321 wkb, |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
322 contourTolerance, |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
323 ); err != nil { |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
324 return 0, err |
4574
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
325 } |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
326 } |
daed8a92024a
Generate iso ares for sounding differences, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4256
diff
changeset
|
327 |
4639
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
328 log.Printf("info: Transferred WKB size: %.2fMB.\n", |
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
329 float64(size)/(1024*1024)) |
4380f5483c03
Log the size of the transferred WKB to database for differences, too.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
4637
diff
changeset
|
330 |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
331 log.Printf("info: calculating and storing iso lines took %v\n", |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
332 time.Since(start)) |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
333 |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
334 if err = tx.Commit(); err != nil { |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
335 log.Printf("info: difference calculation failed after %v\n", |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
336 time.Since(begin)) |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
337 return 0, err |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
338 } |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
339 log.Printf("info: difference calculation succeed after %v\n", |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
340 time.Since(begin)) |
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
341 |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
342 return id, nil |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
343 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
344 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
345 func (dcm *diffCalculationManager) calculate() { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
346 for dci := range dcm.jobs { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
347 ctx := context.Background() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
348 var id int64 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
349 var err error |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
350 err = auth.RunAs(ctx, "sys_admin", func(conn *sql.Conn) error { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
351 id, err = doDiff(ctx, conn, dci) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
352 return err |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
353 }) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
354 dcm.cmds <- func(dcm *diffCalculationManager) { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
355 msg := diffResult{id: id, err: err} |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
356 key := diffCalcHash(dci) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
357 list := dcm.waiting[key] |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
358 delete(dcm.waiting, key) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
359 for _, ch := range list { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
360 ch <- msg |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
361 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
362 } |
2583
5c6a7e69b02d
Morphological differences: Store calculated iso-lines in database.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2580
diff
changeset
|
363 } |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
364 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
365 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
366 func (dcm *diffCalculationManager) enque(dci *models.DiffCalculationInput) chan diffResult { |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
367 // buffer to prevent blocking even if receiver is dead. |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
368 ch := make(chan diffResult, 1) |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
369 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
370 dcm.cmds <- func(dcm *diffCalculationManager) { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
371 // When already running add to list of waitings |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
372 key := diffCalcHash(dci) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
373 if list := dcm.waiting[key]; list != nil { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
374 dcm.waiting[key] = append(list, ch) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
375 return |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
376 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
377 // look if it is in database |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
378 var id int64 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
379 ctx := context.Background() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
380 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
381 err := auth.RunAs(ctx, "sys_admin", func(conn *sql.Conn) error { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
382 return conn.QueryRowContext( |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
383 ctx, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
384 diffIDSQL, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
385 dci.Bottleneck, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
386 dci.Minuend.Time, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
387 dci.Subtrahend.Time, |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
388 ).Scan(&id) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
389 }) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
390 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
391 switch { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
392 case err == sql.ErrNoRows: |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
393 // We need to calculate it. |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
394 case err != nil: // error |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
395 ch <- diffResult{err: err} |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
396 return |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
397 default: // we have it |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
398 ch <- diffResult{id: id} |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
399 return |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
400 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
401 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
402 // add to waiting |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
403 dcm.waiting[key] = []chan diffResult{ch} |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
404 // feed to workers |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
405 select { |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
406 case dcm.jobs <- dci: |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
407 default: // buffer is full |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
408 delete(dcm.waiting, key) |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
409 ch <- diffResult{err: errBufferFull} |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
410 } |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
411 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
412 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
413 return ch |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
414 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
415 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
416 func (dcm *diffCalculationManager) unregister(dci *models.DiffCalculationInput, ch chan diffResult) { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
417 dcm.cmds <- func(dcm *diffCalculationManager) { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
418 key := diffCalcHash(dci) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
419 list := dcm.waiting[key] |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
420 for i, x := range list { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
421 if x == ch { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
422 if i != len(list)-1 { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
423 copy(list[i:], list[i+1:]) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
424 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
425 list[len(list)-1] = nil |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
426 list = list[:len(list)-1] |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
427 if len(list) == 0 { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
428 delete(dcm.waiting, key) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
429 } else { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
430 dcm.waiting[key] = list |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
431 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
432 return |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
433 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
434 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
435 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
436 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
437 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
438 func diffCalculation(req *http.Request) (jr mw.JSONResult, err error) { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
439 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
440 dci := mw.JSONInput(req).(*models.DiffCalculationInput) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
441 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
442 ctx := req.Context() |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
443 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
444 resultCh := diffCalcMng.enque(dci) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
445 defer diffCalcMng.unregister(dci, resultCh) |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
446 |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
447 select { |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
448 case res := <-resultCh: |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
449 switch { |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
450 case res.err == errBufferFull: |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
451 err = mw.JSONError{ |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
452 Code: http.StatusTooManyRequests, |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
453 Message: "Too many differences are waiting to be processed", |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
454 } |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
455 case res.err != nil: |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
456 err = res.err |
4681
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
457 default: |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
458 jr = mw.JSONResult{ |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
459 Result: map[string]int64{"id": res.id}, |
2690933e404c
Limit the number of waiting diff calculations to 5.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4680
diff
changeset
|
460 } |
4680
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
461 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
462 case <-ctx.Done(): |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
463 log.Println("request canceled") |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
464 } |
976aedc195e5
Be more clever when triggering diff calculations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4667
diff
changeset
|
465 |
2570
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
466 return |
c4242b9d59fe
Morphological differences: Added endpoint stub POST /api/diff
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
467 } |