Mercurial > gemma

annotate pkg/imports/wx.go @ 5560:f2204f91d286

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Join the log lines of imports to the log exports to recover data from them. Used in SR export to extract information that where in the meta json but now are only found in the log.
author Sascha L. Teichmann <sascha.teichmann@intevation.de>
date Wed, 09 Feb 2022 18:34:40 +0100
parents 6fe3662aafeb
children 6270951dda28
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1675
8fec3887c7e5 Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
8fec3887c7e5 Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
8fec3887c7e5 Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
3 //
8fec3887c7e5 Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
8fec3887c7e5 Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
8fec3887c7e5 Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
6 //
2649
9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping
Tom Gottfried <tom@intevation.de>
parents: 2622
diff changeset
7 // Copyright (C) 2018, 2019 by via donau
1675
8fec3887c7e5 Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
8fec3887c7e5 Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
8fec3887c7e5 Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
10 //
8fec3887c7e5 Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
11 // Author(s):
8fec3887c7e5 Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
2649
9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping
Tom Gottfried <tom@intevation.de>
parents: 2622
diff changeset
13 // * Tom Gottfried <tom.gottfried@intevation.de>
1675
8fec3887c7e5 Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
14
8fec3887c7e5 Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
15 package imports
8fec3887c7e5 Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
16
1696
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1683
diff changeset
17 // WXJobKind is the import queue type identifier.
1675
8fec3887c7e5 Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
18 const WXJobKind JobKind = "wx"
8fec3887c7e5 Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
19
8fec3887c7e5 Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
20 func init() {
5016
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
21 RegisterJobCreator(WXJobKind,
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
22 &WFSFeatureJobCreator{
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
23 description: "waterway axis",
5020
e4ab338e7ba9 Removed unnecessary creatings of empty slices in expressing read dependencies in imports.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5019
diff changeset
24 depends: [2][]string{{"waterway_axis"}},
5016
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
25 newConsumer: newSQLConsumer(
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
26 prepareStmnts(insertWaterwayAxisSQL),
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
27 consume,
5018
d6873fa95ed6 ... and adjust the name.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5017
diff changeset
28 axisInvalidation,
5016
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
29 newMultiLineFeature(func() interface{} {
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
30 return new(waterwayAxisProperties)
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
31 }),
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
32 ),
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
33 })
1675
8fec3887c7e5 Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
34 }
8fec3887c7e5 Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
35
1680
de8089944b19 Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1679
diff changeset
36 type waterwayAxisProperties struct {
de8089944b19 Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1679
diff changeset
37 ObjNam string `json:"hydro_objnam"`
de8089944b19 Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1679
diff changeset
38 NObjNnm *string `json:"hydro_nobjnm"`
de8089944b19 Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1679
diff changeset
39 }
de8089944b19 Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1679
diff changeset
40
1681
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1680
diff changeset
41 const (
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1680
diff changeset
42 insertWaterwayAxisSQL = `
1795
1333f96f18d0 Waterway axis: Clip features against responsibility area of importing user.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1785
diff changeset
43 WITH resp AS (
2912
93fa55bce126 Add utility function to get users area of responsibility
Tom Gottfried <tom@intevation.de>
parents: 2903
diff changeset
44 SELECT users.current_user_area_utm() AS a
5016
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
45 ),
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
46 g AS (
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
47 SELECT
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
48 ST_Multi(ST_Node(ST_CollectionExtract(ST_Transform(new_ax, 4326), 2)))
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
49 AS new_ax
2855
8a8a929182f9 Enable import of axis and area independently from areas of responsibility
Tom Gottfried <tom@intevation.de>
parents: 2833
diff changeset
50 FROM ST_GeomFromWKB($1, $2::integer) AS new_line (new_line),
5009
e8b2dc771f9e Store axis as MultiLinestring
Tom Gottfried <tom@intevation.de>
parents: 4879
diff changeset
51 LATERAL (SELECT
2649
9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping
Tom Gottfried <tom@intevation.de>
parents: 2622
diff changeset
52 CASE WHEN pg_has_role('sys_admin', 'MEMBER')
5019
dc3d648bdf4a Avoid needless geometry processing
Tom Gottfried <tom@intevation.de>
parents: 5018
diff changeset
53 OR ST_Covers((SELECT a FROM resp),
dc3d648bdf4a Avoid needless geometry processing
Tom Gottfried <tom@intevation.de>
parents: 5018
diff changeset
54 ST_Transform(new_line, (SELECT ST_SRID(a) FROM resp)))
5009
e8b2dc771f9e Store axis as MultiLinestring
Tom Gottfried <tom@intevation.de>
parents: 4879
diff changeset
55 THEN new_line
5013
7dff1015283d Add row level security policies for waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5009
diff changeset
56 ELSE ST_Intersection((SELECT ST_Buffer(a, -0.0001) FROM resp),
2912
93fa55bce126 Add utility function to get users area of responsibility
Tom Gottfried <tom@intevation.de>
parents: 2903
diff changeset
57 ST_Node(ST_Transform(new_line, (SELECT ST_SRID(a) FROM resp))))
5009
e8b2dc771f9e Store axis as MultiLinestring
Tom Gottfried <tom@intevation.de>
parents: 4879
diff changeset
58 END) AS new_ax (new_ax)
e8b2dc771f9e Store axis as MultiLinestring
Tom Gottfried <tom@intevation.de>
parents: 4879
diff changeset
59 -- Do nothing if intersection is empty:
e8b2dc771f9e Store axis as MultiLinestring
Tom Gottfried <tom@intevation.de>
parents: 4879
diff changeset
60 WHERE NOT ST_IsEmpty(new_ax)
5016
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
61 ),
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
62 t AS (
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
63 UPDATE waterway.waterway_axis SET last_found = current_timestamp
5021
6fe3662aafeb Add explaining comment
Tom Gottfried <tom@intevation.de>
parents: 5020
diff changeset
64 -- The first condition is just to help the PostgreSQL query planner
6fe3662aafeb Add explaining comment
Tom Gottfried <tom@intevation.de>
parents: 5020
diff changeset
65 -- to avoid evaluating more costly conditions including those
6fe3662aafeb Add explaining comment
Tom Gottfried <tom@intevation.de>
parents: 5020
diff changeset
66 -- introduced by row level security policies:
5016
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
67 WHERE (SELECT new_ax FROM g) IS NOT NULL
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
68 AND validity @> current_timestamp
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
69 AND (
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
70 wtwaxs, objnam, nobjnam
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
71 ) IS NOT DISTINCT FROM (
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
72 (SELECT new_ax FROM g), $3, $4)
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
73 RETURNING 1
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
74 )
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
75 INSERT INTO waterway.waterway_axis (wtwaxs, objnam, nobjnam)
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
76 SELECT new_ax, $3, $4
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
77 FROM g
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
78 WHERE NOT EXISTS(SELECT 1 FROM t)
2726
dff749c07ae0 Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents: 2723
diff changeset
79 RETURNING id
1795
1333f96f18d0 Waterway axis: Clip features against responsibility area of importing user.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1785
diff changeset
80 `
5016
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
81 invalidateAxisSQL = `
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
82 UPDATE waterway.waterway_axis
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
83 SET validity = tstzrange(lower(validity), current_timestamp)
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
84 WHERE validity @> current_timestamp
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
85 AND last_found < current_timestamp
cf25b23e3eec Keep historic data of waterway axis
Tom Gottfried <tom@intevation.de>
parents: 5013
diff changeset
86 `
1681
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1680
diff changeset
87 )
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1680
diff changeset
88
5018
d6873fa95ed6 ... and adjust the name.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5017
diff changeset
89 func axisInvalidation(spc *SQLGeometryConsumer) error {
5017
557afcd9a131 Curried functions all the way down. In this case a level too much. ;-)
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5016
diff changeset
90 res, err := spc.tx.ExecContext(spc.ctx, invalidateAxisSQL)
557afcd9a131 Curried functions all the way down. In this case a level too much. ;-)
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5016
diff changeset
91 if err != nil {
557afcd9a131 Curried functions all the way down. In this case a level too much. ;-)
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5016
diff changeset
92 return err
1679
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1677
diff changeset
93 }
5017
557afcd9a131 Curried functions all the way down. In this case a level too much. ;-)
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5016
diff changeset
94 old, err := res.RowsAffected()
557afcd9a131 Curried functions all the way down. In this case a level too much. ;-)
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5016
diff changeset
95 if err != nil {
557afcd9a131 Curried functions all the way down. In this case a level too much. ;-)
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5016
diff changeset
96 return err
557afcd9a131 Curried functions all the way down. In this case a level too much. ;-)
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5016
diff changeset
97 }
557afcd9a131 Curried functions all the way down. In this case a level too much. ;-)
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5016
diff changeset
98 if old == 0 {
557afcd9a131 Curried functions all the way down. In this case a level too much. ;-)
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5016
diff changeset
99 return ErrFeaturesUnmodified
557afcd9a131 Curried functions all the way down. In this case a level too much. ;-)
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5016
diff changeset
100 }
557afcd9a131 Curried functions all the way down. In this case a level too much. ;-)
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5016
diff changeset
101 spc.feedback.Info(
557afcd9a131 Curried functions all the way down. In this case a level too much. ;-)
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5016
diff changeset
102 "Number of features removed from data source: %d", old)
557afcd9a131 Curried functions all the way down. In this case a level too much. ;-)
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 5016
diff changeset
103 return nil
1675
8fec3887c7e5 Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
104 }