Mercurial > gemma
annotate pkg/imports/wx.go @ 4855:01b593ea2311
Added missing doc strings for the STR tree in the mesh package.
author | Sascha L. Teichmann <sascha.teichmann@intevation.de> |
---|---|
date | Tue, 19 Nov 2019 17:48:02 +0100 |
parents | 046a07a33b19 |
children | 783d0bec86d3 |
rev | line source |
---|---|
1675
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
2649
9b6a02923bb4
Allow sys_admin to import waterway axis and area without clipping
Tom Gottfried <tom@intevation.de>
parents:
2622
diff
changeset
|
7 // Copyright (C) 2018, 2019 by via donau |
1675
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
2649
9b6a02923bb4
Allow sys_admin to import waterway axis and area without clipping
Tom Gottfried <tom@intevation.de>
parents:
2622
diff
changeset
|
13 // * Tom Gottfried <tom.gottfried@intevation.de> |
1675
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
14 |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 package imports |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
17 import ( |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
18 "context" |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
19 "database/sql" |
1680
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
20 "encoding/json" |
1681
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
21 "errors" |
1679
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
22 "fmt" |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
23 "io" |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
24 "time" |
1675
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
25 |
4062
6c760abcff0e
Move handling of PostgreSQL errors to own package
Tom Gottfried <tom@intevation.de>
parents:
4058
diff
changeset
|
26 "gemma.intevation.de/gemma/pkg/pgxutils" |
1679
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
27 "gemma.intevation.de/gemma/pkg/wfs" |
1675
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
28 ) |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
29 |
1696
ad5e1cddaa09
Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1683
diff
changeset
|
30 // WaterwayAxis is an import job to import |
ad5e1cddaa09
Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1683
diff
changeset
|
31 // the waterway axes in form of line string geometries |
ad5e1cddaa09
Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1683
diff
changeset
|
32 // and attribute data from a WFS service. |
1675
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
33 type WaterwayAxis struct { |
1696
ad5e1cddaa09
Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1683
diff
changeset
|
34 // URL the GetCapabilities URL of the WFS service. |
ad5e1cddaa09
Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1683
diff
changeset
|
35 URL string `json:"url"` |
ad5e1cddaa09
Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1683
diff
changeset
|
36 // FeatureType selects the feature type of the WFS service. |
1677
53304db85888
Waterway axis import: Added route for manual import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1675
diff
changeset
|
37 FeatureType string `json:"feature-type"` |
1696
ad5e1cddaa09
Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1683
diff
changeset
|
38 // SortBy works around misconfigured services to |
ad5e1cddaa09
Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1683
diff
changeset
|
39 // establish a sort order to get the features. |
ad5e1cddaa09
Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1683
diff
changeset
|
40 SortBy string `json:"sort-by"` |
2723
a10022399e24
WFS downloads: Fetch user and password from config.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2721
diff
changeset
|
41 // User is an optional username for Basic Auth. |
a10022399e24
WFS downloads: Fetch user and password from config.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2721
diff
changeset
|
42 User string `json:"user,omitempty"` |
a10022399e24
WFS downloads: Fetch user and password from config.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2721
diff
changeset
|
43 // Password is an optional password for Basic Auth. |
a10022399e24
WFS downloads: Fetch user and password from config.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2721
diff
changeset
|
44 Password string `json:"password,omitempty"` |
1675
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
45 } |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
46 |
4852
046a07a33b19
Fixed the golint issues of the imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4832
diff
changeset
|
47 // Description gives a short info about relevant facts of this import. |
4798
ca6a5f722471
Added Description method to most imports.
Sascha Wilde <wilde@intevation.de>
parents:
4197
diff
changeset
|
48 func (wx *WaterwayAxis) Description() (string, error) { |
4799
f32d086b5dbf
Removed the mechanical touch of the last commit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4798
diff
changeset
|
49 return wx.URL + "|" + wx.FeatureType, nil |
4798
ca6a5f722471
Added Description method to most imports.
Sascha Wilde <wilde@intevation.de>
parents:
4197
diff
changeset
|
50 } |
ca6a5f722471
Added Description method to most imports.
Sascha Wilde <wilde@intevation.de>
parents:
4197
diff
changeset
|
51 |
1696
ad5e1cddaa09
Imports: Resolved the remaining golint issues with this package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1683
diff
changeset
|
52 // WXJobKind is the import queue type identifier. |
1675
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
53 const WXJobKind JobKind = "wx" |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
54 |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
55 type wxJobCreator struct{} |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
56 |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
57 func init() { |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
58 RegisterJobCreator(WXJobKind, wxJobCreator{}) |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
59 } |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
60 |
1754
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1706
diff
changeset
|
61 func (wxJobCreator) Description() string { return "waterway axis" } |
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1706
diff
changeset
|
62 |
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1706
diff
changeset
|
63 func (wxJobCreator) AutoAccept() bool { return true } |
1675
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
64 |
2187
7c83b5277c1c
Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
65 func (wxJobCreator) Create() Job { return new(WaterwayAxis) } |
1675
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
66 |
3219
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3037
diff
changeset
|
67 func (wxJobCreator) Depends() [2][]string { |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3037
diff
changeset
|
68 return [2][]string{ |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3037
diff
changeset
|
69 {"waterway_axis"}, |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3037
diff
changeset
|
70 {}, |
1675
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
71 } |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
72 } |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
73 |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
74 // StageDone is a NOP for waterway axis imports. |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
75 func (wxJobCreator) StageDone(context.Context, *sql.Tx, int64) error { |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
76 return nil |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
77 } |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
78 |
1785
614c6c766691
Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
1764
diff
changeset
|
79 // CleanUp for waterway axis imports is a NOP. |
1675
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
80 func (*WaterwayAxis) CleanUp() error { return nil } |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
81 |
1680
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
82 type waterwayAxisProperties struct { |
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
83 ObjNam string `json:"hydro_objnam"` |
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
84 NObjNnm *string `json:"hydro_nobjnm"` |
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
85 } |
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
86 |
1681
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
87 const ( |
1795
1333f96f18d0
Waterway axis: Clip features against responsibility area of importing user.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1785
diff
changeset
|
88 deleteWaterwayAxisSQL = ` |
1333f96f18d0
Waterway axis: Clip features against responsibility area of importing user.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1785
diff
changeset
|
89 WITH resp AS ( |
2912
93fa55bce126
Add utility function to get users area of responsibility
Tom Gottfried <tom@intevation.de>
parents:
2903
diff
changeset
|
90 SELECT users.current_user_area_utm() AS a |
1795
1333f96f18d0
Waterway axis: Clip features against responsibility area of importing user.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1785
diff
changeset
|
91 ) |
1333f96f18d0
Waterway axis: Clip features against responsibility area of importing user.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1785
diff
changeset
|
92 DELETE FROM waterway.waterway_axis |
2649
9b6a02923bb4
Allow sys_admin to import waterway axis and area without clipping
Tom Gottfried <tom@intevation.de>
parents:
2622
diff
changeset
|
93 WHERE pg_has_role('sys_admin', 'MEMBER') |
9b6a02923bb4
Allow sys_admin to import waterway axis and area without clipping
Tom Gottfried <tom@intevation.de>
parents:
2622
diff
changeset
|
94 OR ST_Covers((SELECT a FROM resp), |
2912
93fa55bce126
Add utility function to get users area of responsibility
Tom Gottfried <tom@intevation.de>
parents:
2903
diff
changeset
|
95 ST_Transform(wtwaxs::geometry, (SELECT ST_SRID(a) FROM resp))) |
1795
1333f96f18d0
Waterway axis: Clip features against responsibility area of importing user.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1785
diff
changeset
|
96 ` |
1333f96f18d0
Waterway axis: Clip features against responsibility area of importing user.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1785
diff
changeset
|
97 |
1681
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
98 insertWaterwayAxisSQL = ` |
1795
1333f96f18d0
Waterway axis: Clip features against responsibility area of importing user.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1785
diff
changeset
|
99 WITH resp AS ( |
2912
93fa55bce126
Add utility function to get users area of responsibility
Tom Gottfried <tom@intevation.de>
parents:
2903
diff
changeset
|
100 SELECT users.current_user_area_utm() AS a |
1795
1333f96f18d0
Waterway axis: Clip features against responsibility area of importing user.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1785
diff
changeset
|
101 ) |
1681
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
102 INSERT INTO waterway.waterway_axis (wtwaxs, objnam, nobjnam) |
2855
8a8a929182f9
Enable import of axis and area independently from areas of responsibility
Tom Gottfried <tom@intevation.de>
parents:
2833
diff
changeset
|
103 SELECT dmp.geom, $3, $4 |
8a8a929182f9
Enable import of axis and area independently from areas of responsibility
Tom Gottfried <tom@intevation.de>
parents:
2833
diff
changeset
|
104 FROM ST_GeomFromWKB($1, $2::integer) AS new_line (new_line), |
8a8a929182f9
Enable import of axis and area independently from areas of responsibility
Tom Gottfried <tom@intevation.de>
parents:
2833
diff
changeset
|
105 ST_Dump(ST_Transform(ST_CollectionExtract( |
2649
9b6a02923bb4
Allow sys_admin to import waterway axis and area without clipping
Tom Gottfried <tom@intevation.de>
parents:
2622
diff
changeset
|
106 CASE WHEN pg_has_role('sys_admin', 'MEMBER') |
2855
8a8a929182f9
Enable import of axis and area independently from areas of responsibility
Tom Gottfried <tom@intevation.de>
parents:
2833
diff
changeset
|
107 THEN ST_Node(ST_Transform(new_line, |
8a8a929182f9
Enable import of axis and area independently from areas of responsibility
Tom Gottfried <tom@intevation.de>
parents:
2833
diff
changeset
|
108 best_utm(ST_Transform(new_line, 4326)))) |
8a8a929182f9
Enable import of axis and area independently from areas of responsibility
Tom Gottfried <tom@intevation.de>
parents:
2833
diff
changeset
|
109 ELSE ST_Intersection((SELECT a FROM resp), |
2912
93fa55bce126
Add utility function to get users area of responsibility
Tom Gottfried <tom@intevation.de>
parents:
2903
diff
changeset
|
110 ST_Node(ST_Transform(new_line, (SELECT ST_SRID(a) FROM resp)))) |
2855
8a8a929182f9
Enable import of axis and area independently from areas of responsibility
Tom Gottfried <tom@intevation.de>
parents:
2833
diff
changeset
|
111 END, |
8a8a929182f9
Enable import of axis and area independently from areas of responsibility
Tom Gottfried <tom@intevation.de>
parents:
2833
diff
changeset
|
112 2), 4326)) AS dmp |
2726
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
113 RETURNING id |
1795
1333f96f18d0
Waterway axis: Clip features against responsibility area of importing user.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1785
diff
changeset
|
114 ` |
1681
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
115 ) |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
116 |
1785
614c6c766691
Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
1764
diff
changeset
|
117 // Do executes the actual waterway axis import. |
1675
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
118 func (wx *WaterwayAxis) Do( |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
119 ctx context.Context, |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
120 importID int64, |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
121 conn *sql.Conn, |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
122 feedback Feedback, |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
123 ) (interface{}, error) { |
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
124 |
1679
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
125 start := time.Now() |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
126 |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
127 feedback.Info("Import waterway axis") |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
128 |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
129 feedback.Info("Loading capabilities from %s", wx.URL) |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
130 caps, err := wfs.GetCapabilities(wx.URL) |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
131 if err != nil { |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
132 feedback.Error("Loading capabilities failed: %v", err) |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
133 return nil, err |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
134 } |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
135 |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
136 ft := caps.FindFeatureType(wx.FeatureType) |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
137 if ft == nil { |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4147
diff
changeset
|
138 return nil, fmt.Errorf("unknown feature type '%s'", wx.FeatureType) |
1679
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
139 } |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
140 |
1706
fb05027d93b6
Waterway axis: Be more verbose during import about feature type and sorting.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1696
diff
changeset
|
141 feedback.Info("Found feature type '%s", wx.FeatureType) |
fb05027d93b6
Waterway axis: Be more verbose during import about feature type and sorting.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1696
diff
changeset
|
142 |
1681
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
143 epsg, err := wfs.CRSToEPSG(ft.DefaultCRS) |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
144 if err != nil { |
4832
5062ccb2381d
Unified error message on "Unsupported CRS"
Sascha Wilde <wilde@intevation.de>
parents:
4799
diff
changeset
|
145 feedback.Error("Unsupported CRS: '%s'", ft.DefaultCRS) |
1681
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
146 return nil, err |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
147 } |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
148 |
1706
fb05027d93b6
Waterway axis: Be more verbose during import about feature type and sorting.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1696
diff
changeset
|
149 if wx.SortBy != "" { |
fb05027d93b6
Waterway axis: Be more verbose during import about feature type and sorting.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1696
diff
changeset
|
150 feedback.Info("Features will be sorted by '%s'", wx.SortBy) |
fb05027d93b6
Waterway axis: Be more verbose during import about feature type and sorting.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1696
diff
changeset
|
151 } |
fb05027d93b6
Waterway axis: Be more verbose during import about feature type and sorting.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1696
diff
changeset
|
152 |
2721
56c8ef515d93
WFS downloader: Use new GML/GeoJSON factory function in gemma.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2719
diff
changeset
|
153 dl, err := wfs.GetFeatures(caps, wx.FeatureType, wx.SortBy) |
1679
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
154 if err != nil { |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
155 feedback.Error("Cannot create GetFeature URLs. %v", err) |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
156 return nil, err |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
157 } |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
158 |
1681
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
159 tx, err := conn.BeginTx(ctx, nil) |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
160 if err != nil { |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
161 return nil, err |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
162 } |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
163 defer tx.Rollback() |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
164 |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
165 insertStmt, err := tx.PrepareContext(ctx, insertWaterwayAxisSQL) |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
166 if err != nil { |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
167 return nil, err |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
168 } |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
169 defer insertStmt.Close() |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
170 |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
171 // Delete the old features. |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
172 if _, err := tx.ExecContext(ctx, deleteWaterwayAxisSQL); err != nil { |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
173 return nil, err |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
174 } |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
175 |
1680
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
176 var ( |
1785
614c6c766691
Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
1764
diff
changeset
|
177 unsupported = stringCounter{} |
1680
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
178 missingProperties int |
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
179 badProperties int |
2937
0d79f5eec630
Fix logging in axis import
Tom Gottfried <tom@intevation.de>
parents:
2912
diff
changeset
|
180 outside int |
1681
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
181 features int |
1680
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
182 ) |
1679
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
183 |
2729
7cb027be277d
WFS downloader: Really use username and password from parameters.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2726
diff
changeset
|
184 if err := dl.Download(wx.User, wx.Password, func(url string, r io.Reader) error { |
2106
2b72f5e005aa
WFS imports: Write get GetFeature URLs into import log.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1828
diff
changeset
|
185 feedback.Info("Get features from: '%s'", url) |
1679
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
186 rfc, err := wfs.ParseRawFeatureCollection(r) |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
187 if err != nil { |
1764
bb4348ac52ab
Waterway axis import: Be more verbose when parsing the GetFeature document fails.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1754
diff
changeset
|
188 return fmt.Errorf("parsing GetFeature document failed: %v", err) |
1679
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
189 } |
1681
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
190 if rfc.CRS != nil { |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
191 crsName := rfc.CRS.Properties.Name |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
192 if epsg, err = wfs.CRSToEPSG(crsName); err != nil { |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
193 feedback.Error("Unsupported CRS: %d", crsName) |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
194 return err |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
195 } |
1679
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
196 } |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
197 |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
198 // No features -> ignore. |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
199 if rfc.Features == nil { |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
200 return nil |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
201 } |
1680
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
202 |
1683
509e8728e846
Fixed pre-calculated size of 2D WKB line strings.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1682
diff
changeset
|
203 feedback.Info("Using EPSG: %d", epsg) |
509e8728e846
Fixed pre-calculated size of 2D WKB line strings.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1682
diff
changeset
|
204 |
2780
5850ac0f9ab6
Axis import: do not fail if one item fails.
Tom Gottfried <tom@intevation.de>
parents:
2730
diff
changeset
|
205 savepoint := Savepoint(ctx, tx, "feature") |
5850ac0f9ab6
Axis import: do not fail if one item fails.
Tom Gottfried <tom@intevation.de>
parents:
2730
diff
changeset
|
206 |
1679
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
207 for _, feature := range rfc.Features { |
1681
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
208 if feature.Properties == nil || feature.Geometry.Coordinates == nil { |
1680
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
209 missingProperties++ |
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
210 continue |
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
211 } |
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
212 |
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
213 var props waterwayAxisProperties |
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
214 |
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
215 if err := json.Unmarshal(*feature.Properties, &props); err != nil { |
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
216 badProperties++ |
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
217 continue |
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
218 } |
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
219 |
1681
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
220 var nobjnam sql.NullString |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
221 if props.NObjNnm != nil { |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
222 nobjnam = sql.NullString{String: *props.NObjNnm, Valid: true} |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
223 } |
1680
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
224 |
1679
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
225 switch feature.Geometry.Type { |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
226 case "LineString": |
1785
614c6c766691
Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
1764
diff
changeset
|
227 var l lineSlice |
1681
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
228 if err := json.Unmarshal(*feature.Geometry.Coordinates, &l); err != nil { |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
229 return err |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
230 } |
2726
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
231 if err := storeLinestring( |
1681
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
232 ctx, |
2780
5850ac0f9ab6
Axis import: do not fail if one item fails.
Tom Gottfried <tom@intevation.de>
parents:
2730
diff
changeset
|
233 savepoint, |
2726
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
234 feedback, |
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
235 l, |
1681
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
236 epsg, |
2726
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
237 props, |
1681
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
238 nobjnam, |
2937
0d79f5eec630
Fix logging in axis import
Tom Gottfried <tom@intevation.de>
parents:
2912
diff
changeset
|
239 &outside, |
0d79f5eec630
Fix logging in axis import
Tom Gottfried <tom@intevation.de>
parents:
2912
diff
changeset
|
240 &features, |
2726
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
241 insertStmt); err != nil { |
1681
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
242 return err |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
243 } |
1679
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
244 case "MultiLineString": |
1785
614c6c766691
Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
1764
diff
changeset
|
245 var ls []lineSlice |
1681
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
246 if err := json.Unmarshal(*feature.Geometry.Coordinates, &ls); err != nil { |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
247 return err |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
248 } |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
249 for _, l := range ls { |
2726
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
250 if err := storeLinestring( |
1681
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
251 ctx, |
2780
5850ac0f9ab6
Axis import: do not fail if one item fails.
Tom Gottfried <tom@intevation.de>
parents:
2730
diff
changeset
|
252 savepoint, |
2726
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
253 feedback, |
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
254 l, |
1681
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
255 epsg, |
2726
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
256 props, |
1681
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
257 nobjnam, |
2937
0d79f5eec630
Fix logging in axis import
Tom Gottfried <tom@intevation.de>
parents:
2912
diff
changeset
|
258 &outside, |
0d79f5eec630
Fix logging in axis import
Tom Gottfried <tom@intevation.de>
parents:
2912
diff
changeset
|
259 &features, |
2726
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
260 insertStmt); err != nil { |
1681
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
261 return err |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
262 } |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
263 } |
1679
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
264 default: |
1785
614c6c766691
Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
1764
diff
changeset
|
265 unsupported[feature.Geometry.Type]++ |
1679
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
266 } |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
267 } |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
268 return nil |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
269 }); err != nil { |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
270 return nil, err |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
271 } |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
272 |
1680
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
273 if badProperties > 0 { |
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
274 feedback.Warn("Bad properties: %d", badProperties) |
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
275 } |
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
276 |
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
277 if missingProperties > 0 { |
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
278 feedback.Warn("Missing properties: %d", missingProperties) |
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
279 } |
de8089944b19
Waterway axis import: Started extracting properties from the features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1679
diff
changeset
|
280 |
1785
614c6c766691
Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
1764
diff
changeset
|
281 if len(unsupported) != 0 { |
614c6c766691
Waterway area import: Implemented.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
1764
diff
changeset
|
282 feedback.Warn("Unsupported types found: %s", unsupported) |
1679
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
283 } |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
284 |
2937
0d79f5eec630
Fix logging in axis import
Tom Gottfried <tom@intevation.de>
parents:
2912
diff
changeset
|
285 if outside > 0 { |
0d79f5eec630
Fix logging in axis import
Tom Gottfried <tom@intevation.de>
parents:
2912
diff
changeset
|
286 feedback.Info("Features outside responsibility area: %d", outside) |
0d79f5eec630
Fix logging in axis import
Tom Gottfried <tom@intevation.de>
parents:
2912
diff
changeset
|
287 } |
0d79f5eec630
Fix logging in axis import
Tom Gottfried <tom@intevation.de>
parents:
2912
diff
changeset
|
288 |
1828
1ecfcf46e4da
Waterway area/axis: Be more informative if features are bad.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1796
diff
changeset
|
289 if features == 0 { |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4147
diff
changeset
|
290 return nil, errors.New("no features found") |
1828
1ecfcf46e4da
Waterway area/axis: Be more informative if features are bad.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1796
diff
changeset
|
291 } |
1ecfcf46e4da
Waterway area/axis: Be more informative if features are bad.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1796
diff
changeset
|
292 |
1681
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
293 if err = tx.Commit(); err == nil { |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
294 feedback.Info("Storing %d features took %s", |
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
295 features, time.Since(start)) |
1679
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
296 } |
2dc7768be0e4
Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1677
diff
changeset
|
297 |
1681
4d6ce621379e
Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1680
diff
changeset
|
298 return nil, err |
1675
8fec3887c7e5
Waterway axis import: Added stub.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
299 } |
2726
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
300 |
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
301 func storeLinestring( |
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
302 ctx context.Context, |
2780
5850ac0f9ab6
Axis import: do not fail if one item fails.
Tom Gottfried <tom@intevation.de>
parents:
2730
diff
changeset
|
303 savepoint func(func() error) error, |
2726
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
304 feedback Feedback, |
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
305 l lineSlice, |
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
306 epsg int, |
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
307 props waterwayAxisProperties, |
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
308 nobjnam sql.NullString, |
2937
0d79f5eec630
Fix logging in axis import
Tom Gottfried <tom@intevation.de>
parents:
2912
diff
changeset
|
309 outside, features *int, |
3037
72451510d417
Remove expensive check with doubtful usefulness
Tom Gottfried <tom@intevation.de>
parents:
2937
diff
changeset
|
310 insertStmt *sql.Stmt, |
2726
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
311 ) error { |
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
312 var id int |
2937
0d79f5eec630
Fix logging in axis import
Tom Gottfried <tom@intevation.de>
parents:
2912
diff
changeset
|
313 err := savepoint(func() error { |
2780
5850ac0f9ab6
Axis import: do not fail if one item fails.
Tom Gottfried <tom@intevation.de>
parents:
2730
diff
changeset
|
314 err := insertStmt.QueryRowContext( |
5850ac0f9ab6
Axis import: do not fail if one item fails.
Tom Gottfried <tom@intevation.de>
parents:
2730
diff
changeset
|
315 ctx, |
5850ac0f9ab6
Axis import: do not fail if one item fails.
Tom Gottfried <tom@intevation.de>
parents:
2730
diff
changeset
|
316 l.asWKB(), |
5850ac0f9ab6
Axis import: do not fail if one item fails.
Tom Gottfried <tom@intevation.de>
parents:
2730
diff
changeset
|
317 epsg, |
5850ac0f9ab6
Axis import: do not fail if one item fails.
Tom Gottfried <tom@intevation.de>
parents:
2730
diff
changeset
|
318 props.ObjNam, |
5850ac0f9ab6
Axis import: do not fail if one item fails.
Tom Gottfried <tom@intevation.de>
parents:
2730
diff
changeset
|
319 nobjnam, |
5850ac0f9ab6
Axis import: do not fail if one item fails.
Tom Gottfried <tom@intevation.de>
parents:
2730
diff
changeset
|
320 ).Scan(&id) |
2726
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
321 return err |
2937
0d79f5eec630
Fix logging in axis import
Tom Gottfried <tom@intevation.de>
parents:
2912
diff
changeset
|
322 }) |
0d79f5eec630
Fix logging in axis import
Tom Gottfried <tom@intevation.de>
parents:
2912
diff
changeset
|
323 switch { |
0d79f5eec630
Fix logging in axis import
Tom Gottfried <tom@intevation.de>
parents:
2912
diff
changeset
|
324 case err == sql.ErrNoRows: |
0d79f5eec630
Fix logging in axis import
Tom Gottfried <tom@intevation.de>
parents:
2912
diff
changeset
|
325 *outside++ |
0d79f5eec630
Fix logging in axis import
Tom Gottfried <tom@intevation.de>
parents:
2912
diff
changeset
|
326 // ignore -> filtered by responsibility_areas |
0d79f5eec630
Fix logging in axis import
Tom Gottfried <tom@intevation.de>
parents:
2912
diff
changeset
|
327 return nil |
0d79f5eec630
Fix logging in axis import
Tom Gottfried <tom@intevation.de>
parents:
2912
diff
changeset
|
328 case err != nil: |
4197
5d7ce7f926eb
Log fatal errors as errors, not warnings, in imports
Tom Gottfried <tom@intevation.de>
parents:
4177
diff
changeset
|
329 feedback.Error(pgxutils.ReadableError{Err: err}.Error()) |
2937
0d79f5eec630
Fix logging in axis import
Tom Gottfried <tom@intevation.de>
parents:
2912
diff
changeset
|
330 default: |
0d79f5eec630
Fix logging in axis import
Tom Gottfried <tom@intevation.de>
parents:
2912
diff
changeset
|
331 *features++ |
2726
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
332 } |
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
333 return nil |
dff749c07ae0
Warn if imported axis linestrings cross each other
Tom Gottfried <tom@intevation.de>
parents:
2723
diff
changeset
|
334 } |