gemma: pkg/imports/wx.go annotate

annotate pkg/imports/wx.go @ 5013:7dff1015283d

Add row level security policies for waterway axis Enforcing the area of responsibility this way instead of leaving it up to the importer implementation will also reduce complexity of statements needed to implement keeping of historic axis data.

author	Tom Gottfried <tom@intevation.de>
date	Thu, 12 Mar 2020 14:49:19 +0100
parents	e8b2dc771f9e
children	cf25b23e3eec

rev	line source
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
2649 9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping Tom Gottfried <tom@intevation.de> parents: 2622 diff changeset	7 // Copyright (C) 2018, 2019 by via donau
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
2649 9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping Tom Gottfried <tom@intevation.de> parents: 2622 diff changeset	13 // * Tom Gottfried <tom.gottfried@intevation.de>
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	14
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15 package imports
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17 import (
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 "context"
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	19 "database/sql"
1680 de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	20 "encoding/json"
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	21 "errors"
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	22 "fmt"
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	23 "io"
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	24 "time"
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	25
4062 6c760abcff0e Move handling of PostgreSQL errors to own package Tom Gottfried <tom@intevation.de> parents: 4058 diff changeset	26 "gemma.intevation.de/gemma/pkg/pgxutils"
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	27 "gemma.intevation.de/gemma/pkg/wfs"
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	28 )
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	29
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1683 diff changeset	30 // WaterwayAxis is an import job to import
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1683 diff changeset	31 // the waterway axes in form of line string geometries
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1683 diff changeset	32 // and attribute data from a WFS service.
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	33 type WaterwayAxis struct {
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1683 diff changeset	34 // URL the GetCapabilities URL of the WFS service.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1683 diff changeset	35 URL string `json:"url"`
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1683 diff changeset	36 // FeatureType selects the feature type of the WFS service.
1677 53304db85888 Waterway axis import: Added route for manual import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1675 diff changeset	37 FeatureType string `json:"feature-type"`
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1683 diff changeset	38 // SortBy works around misconfigured services to
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1683 diff changeset	39 // establish a sort order to get the features.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1683 diff changeset	40 SortBy string `json:"sort-by"`
2723 a10022399e24 WFS downloads: Fetch user and password from config. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2721 diff changeset	41 // User is an optional username for Basic Auth.
a10022399e24 WFS downloads: Fetch user and password from config. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2721 diff changeset	42 User string `json:"user,omitempty"`
a10022399e24 WFS downloads: Fetch user and password from config. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2721 diff changeset	43 // Password is an optional password for Basic Auth.
a10022399e24 WFS downloads: Fetch user and password from config. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2721 diff changeset	44 Password string `json:"password,omitempty"`
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	45 }
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	46
4852 046a07a33b19 Fixed the golint issues of the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4832 diff changeset	47 // Description gives a short info about relevant facts of this import.
4798 ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4197 diff changeset	48 func (wx *WaterwayAxis) Description() (string, error) {
4799 f32d086b5dbf Removed the mechanical touch of the last commit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4798 diff changeset	49 return wx.URL + "\|" + wx.FeatureType, nil
4798 ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4197 diff changeset	50 }
ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4197 diff changeset	51
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1683 diff changeset	52 // WXJobKind is the import queue type identifier.
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	53 const WXJobKind JobKind = "wx"
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	54
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	55 type wxJobCreator struct{}
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	56
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	57 func init() {
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	58 RegisterJobCreator(WXJobKind, wxJobCreator{})
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	59 }
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	60
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1706 diff changeset	61 func (wxJobCreator) Description() string { return "waterway axis" }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1706 diff changeset	62
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1706 diff changeset	63 func (wxJobCreator) AutoAccept() bool { return true }
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	64
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	65 func (wxJobCreator) Create() Job { return new(WaterwayAxis) }
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	66
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3037 diff changeset	67 func (wxJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3037 diff changeset	68 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3037 diff changeset	69 {"waterway_axis"},
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3037 diff changeset	70 {},
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	71 }
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	72 }
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	73
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	74 // StageDone is a NOP for waterway axis imports.
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	75 func (wxJobCreator) StageDone(context.Context, *sql.Tx, int64) error {
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	76 return nil
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	77 }
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	78
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1764 diff changeset	79 // CleanUp for waterway axis imports is a NOP.
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	80 func (*WaterwayAxis) CleanUp() error { return nil }
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	81
1680 de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	82 type waterwayAxisProperties struct {
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	83 ObjNam string `json:"hydro_objnam"`
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	84 NObjNnm *string `json:"hydro_nobjnm"`
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	85 }
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	86
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	87 const (
1795 1333f96f18d0 Waterway axis: Clip features against responsibility area of importing user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1785 diff changeset	88 deleteWaterwayAxisSQL = `
1333f96f18d0 Waterway axis: Clip features against responsibility area of importing user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1785 diff changeset	89 DELETE FROM waterway.waterway_axis
1333f96f18d0 Waterway axis: Clip features against responsibility area of importing user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1785 diff changeset	90 `
1333f96f18d0 Waterway axis: Clip features against responsibility area of importing user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1785 diff changeset	91
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	92 insertWaterwayAxisSQL = `
1795 1333f96f18d0 Waterway axis: Clip features against responsibility area of importing user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1785 diff changeset	93 WITH resp AS (
2912 93fa55bce126 Add utility function to get users area of responsibility Tom Gottfried <tom@intevation.de> parents: 2903 diff changeset	94 SELECT users.current_user_area_utm() AS a
1795 1333f96f18d0 Waterway axis: Clip features against responsibility area of importing user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1785 diff changeset	95 )
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	96 INSERT INTO waterway.waterway_axis (wtwaxs, objnam, nobjnam)
5009 e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	97 SELECT
e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	98 ST_Multi(ST_Node(ST_CollectionExtract(ST_Transform(new_ax, 4326), 2))),
e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	99 $3, $4
2855 8a8a929182f9 Enable import of axis and area independently from areas of responsibility Tom Gottfried <tom@intevation.de> parents: 2833 diff changeset	100 FROM ST_GeomFromWKB($1, $2::integer) AS new_line (new_line),
5009 e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	101 LATERAL (SELECT
2649 9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping Tom Gottfried <tom@intevation.de> parents: 2622 diff changeset	102 CASE WHEN pg_has_role('sys_admin', 'MEMBER')
5009 e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	103 THEN new_line
5013 7dff1015283d Add row level security policies for waterway axis Tom Gottfried <tom@intevation.de> parents: 5009 diff changeset	104 ELSE ST_Intersection((SELECT ST_Buffer(a, -0.0001) FROM resp),
2912 93fa55bce126 Add utility function to get users area of responsibility Tom Gottfried <tom@intevation.de> parents: 2903 diff changeset	105 ST_Node(ST_Transform(new_line, (SELECT ST_SRID(a) FROM resp))))
5009 e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	106 END) AS new_ax (new_ax)
e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	107 -- Do nothing if intersection is empty:
e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	108 WHERE NOT ST_IsEmpty(new_ax)
2726 dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	109 RETURNING id
1795 1333f96f18d0 Waterway axis: Clip features against responsibility area of importing user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1785 diff changeset	110 `
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	111 )
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	112
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1764 diff changeset	113 // Do executes the actual waterway axis import.
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	114 func (wx *WaterwayAxis) Do(
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	115 ctx context.Context,
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	116 importID int64,
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	117 conn *sql.Conn,
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	118 feedback Feedback,
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	119 ) (interface{}, error) {
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	120
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	121 start := time.Now()
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	122
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	123 feedback.Info("Import waterway axis")
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	124
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	125 feedback.Info("Loading capabilities from %s", wx.URL)
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	126 caps, err := wfs.GetCapabilities(wx.URL)
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	127 if err != nil {
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	128 feedback.Error("Loading capabilities failed: %v", err)
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	129 return nil, err
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	130 }
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	131
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	132 ft := caps.FindFeatureType(wx.FeatureType)
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	133 if ft == nil {
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4147 diff changeset	134 return nil, fmt.Errorf("unknown feature type '%s'", wx.FeatureType)
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	135 }
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	136
4879 783d0bec86d3 Add missing quotation marks Tom Gottfried <tom@intevation.de> parents: 4852 diff changeset	137 feedback.Info("Found feature type '%s'", wx.FeatureType)
1706 fb05027d93b6 Waterway axis: Be more verbose during import about feature type and sorting. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	138
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	139 epsg, err := wfs.CRSToEPSG(ft.DefaultCRS)
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	140 if err != nil {
4832 5062ccb2381d Unified error message on "Unsupported CRS" Sascha Wilde <wilde@intevation.de> parents: 4799 diff changeset	141 feedback.Error("Unsupported CRS: '%s'", ft.DefaultCRS)
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	142 return nil, err
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	143 }
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	144
1706 fb05027d93b6 Waterway axis: Be more verbose during import about feature type and sorting. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	145 if wx.SortBy != "" {
fb05027d93b6 Waterway axis: Be more verbose during import about feature type and sorting. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	146 feedback.Info("Features will be sorted by '%s'", wx.SortBy)
fb05027d93b6 Waterway axis: Be more verbose during import about feature type and sorting. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	147 }
fb05027d93b6 Waterway axis: Be more verbose during import about feature type and sorting. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	148
2721 56c8ef515d93 WFS downloader: Use new GML/GeoJSON factory function in gemma. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2719 diff changeset	149 dl, err := wfs.GetFeatures(caps, wx.FeatureType, wx.SortBy)
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	150 if err != nil {
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	151 feedback.Error("Cannot create GetFeature URLs. %v", err)
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	152 return nil, err
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	153 }
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	154
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	155 tx, err := conn.BeginTx(ctx, nil)
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	156 if err != nil {
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	157 return nil, err
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	158 }
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	159 defer tx.Rollback()
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	160
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	161 insertStmt, err := tx.PrepareContext(ctx, insertWaterwayAxisSQL)
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	162 if err != nil {
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	163 return nil, err
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	164 }
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	165 defer insertStmt.Close()
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	166
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	167 // Delete the old features.
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	168 if _, err := tx.ExecContext(ctx, deleteWaterwayAxisSQL); err != nil {
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	169 return nil, err
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	170 }
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	171
1680 de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	172 var (
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1764 diff changeset	173 unsupported = stringCounter{}
1680 de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	174 missingProperties int
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	175 badProperties int
2937 0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	176 outside int
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	177 features int
1680 de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	178 )
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	179
2729 7cb027be277d WFS downloader: Really use username and password from parameters. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2726 diff changeset	180 if err := dl.Download(wx.User, wx.Password, func(url string, r io.Reader) error {
2106 2b72f5e005aa WFS imports: Write get GetFeature URLs into import log. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1828 diff changeset	181 feedback.Info("Get features from: '%s'", url)
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	182 rfc, err := wfs.ParseRawFeatureCollection(r)
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	183 if err != nil {
1764 bb4348ac52ab Waterway axis import: Be more verbose when parsing the GetFeature document fails. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1754 diff changeset	184 return fmt.Errorf("parsing GetFeature document failed: %v", err)
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	185 }
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	186 if rfc.CRS != nil {
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	187 crsName := rfc.CRS.Properties.Name
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	188 if epsg, err = wfs.CRSToEPSG(crsName); err != nil {
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	189 feedback.Error("Unsupported CRS: %d", crsName)
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	190 return err
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	191 }
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	192 }
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	193
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	194 // No features -> ignore.
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	195 if rfc.Features == nil {
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	196 return nil
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	197 }
1680 de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	198
1683 509e8728e846 Fixed pre-calculated size of 2D WKB line strings. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1682 diff changeset	199 feedback.Info("Using EPSG: %d", epsg)
509e8728e846 Fixed pre-calculated size of 2D WKB line strings. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1682 diff changeset	200
2780 5850ac0f9ab6 Axis import: do not fail if one item fails. Tom Gottfried <tom@intevation.de> parents: 2730 diff changeset	201 savepoint := Savepoint(ctx, tx, "feature")
5850ac0f9ab6 Axis import: do not fail if one item fails. Tom Gottfried <tom@intevation.de> parents: 2730 diff changeset	202
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	203 for _, feature := range rfc.Features {
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	204 if feature.Properties == nil \|\| feature.Geometry.Coordinates == nil {
1680 de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	205 missingProperties++
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	206 continue
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	207 }
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	208
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	209 var props waterwayAxisProperties
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	210
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	211 if err := json.Unmarshal(*feature.Properties, &props); err != nil {
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	212 badProperties++
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	213 continue
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	214 }
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	215
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	216 var nobjnam sql.NullString
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	217 if props.NObjNnm != nil {
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	218 nobjnam = sql.NullString{String: *props.NObjNnm, Valid: true}
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	219 }
1680 de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	220
5009 e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	221 var ls multiLineSlice
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	222 switch feature.Geometry.Type {
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	223 case "LineString":
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1764 diff changeset	224 var l lineSlice
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	225 if err := json.Unmarshal(*feature.Geometry.Coordinates, &l); err != nil {
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	226 return err
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	227 }
5009 e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	228 ls = append(ls, l)
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	229 case "MultiLineString":
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	230 if err := json.Unmarshal(*feature.Geometry.Coordinates, &ls); err != nil {
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	231 return err
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	232 }
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	233 default:
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1764 diff changeset	234 unsupported[feature.Geometry.Type]++
5009 e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	235 continue
e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	236 }
e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	237 if err := storeLinestring(
e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	238 ctx,
e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	239 savepoint,
e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	240 feedback,
e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	241 ls,
e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	242 epsg,
e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	243 props,
e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	244 nobjnam,
e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	245 &outside,
e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	246 &features,
e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	247 insertStmt); err != nil {
e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	248 return err
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	249 }
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	250 }
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	251 return nil
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	252 }); err != nil {
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	253 return nil, err
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	254 }
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	255
1680 de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	256 if badProperties > 0 {
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	257 feedback.Warn("Bad properties: %d", badProperties)
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	258 }
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	259
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	260 if missingProperties > 0 {
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	261 feedback.Warn("Missing properties: %d", missingProperties)
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	262 }
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	263
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1764 diff changeset	264 if len(unsupported) != 0 {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1764 diff changeset	265 feedback.Warn("Unsupported types found: %s", unsupported)
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	266 }
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	267
2937 0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	268 if outside > 0 {
0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	269 feedback.Info("Features outside responsibility area: %d", outside)
0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	270 }
0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	271
1828 1ecfcf46e4da Waterway area/axis: Be more informative if features are bad. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1796 diff changeset	272 if features == 0 {
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4147 diff changeset	273 return nil, errors.New("no features found")
1828 1ecfcf46e4da Waterway area/axis: Be more informative if features are bad. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1796 diff changeset	274 }
1ecfcf46e4da Waterway area/axis: Be more informative if features are bad. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1796 diff changeset	275
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	276 if err = tx.Commit(); err == nil {
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	277 feedback.Info("Storing %d features took %s",
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	278 features, time.Since(start))
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	279 }
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	280
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	281 return nil, err
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	282 }
2726 dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	283
dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	284 func storeLinestring(
dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	285 ctx context.Context,
2780 5850ac0f9ab6 Axis import: do not fail if one item fails. Tom Gottfried <tom@intevation.de> parents: 2730 diff changeset	286 savepoint func(func() error) error,
2726 dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	287 feedback Feedback,
5009 e8b2dc771f9e Store axis as MultiLinestring Tom Gottfried <tom@intevation.de> parents: 4879 diff changeset	288 l multiLineSlice,
2726 dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	289 epsg int,
dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	290 props waterwayAxisProperties,
dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	291 nobjnam sql.NullString,
2937 0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	292 outside, features *int,
3037 72451510d417 Remove expensive check with doubtful usefulness Tom Gottfried <tom@intevation.de> parents: 2937 diff changeset	293 insertStmt *sql.Stmt,
2726 dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	294 ) error {
dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	295 var id int
2937 0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	296 err := savepoint(func() error {
2780 5850ac0f9ab6 Axis import: do not fail if one item fails. Tom Gottfried <tom@intevation.de> parents: 2730 diff changeset	297 err := insertStmt.QueryRowContext(
5850ac0f9ab6 Axis import: do not fail if one item fails. Tom Gottfried <tom@intevation.de> parents: 2730 diff changeset	298 ctx,
5850ac0f9ab6 Axis import: do not fail if one item fails. Tom Gottfried <tom@intevation.de> parents: 2730 diff changeset	299 l.asWKB(),
5850ac0f9ab6 Axis import: do not fail if one item fails. Tom Gottfried <tom@intevation.de> parents: 2730 diff changeset	300 epsg,
5850ac0f9ab6 Axis import: do not fail if one item fails. Tom Gottfried <tom@intevation.de> parents: 2730 diff changeset	301 props.ObjNam,
5850ac0f9ab6 Axis import: do not fail if one item fails. Tom Gottfried <tom@intevation.de> parents: 2730 diff changeset	302 nobjnam,
5850ac0f9ab6 Axis import: do not fail if one item fails. Tom Gottfried <tom@intevation.de> parents: 2730 diff changeset	303 ).Scan(&id)
2726 dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	304 return err
2937 0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	305 })
0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	306 switch {
0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	307 case err == sql.ErrNoRows:
0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	308 *outside++
0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	309 // ignore -> filtered by responsibility_areas
0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	310 return nil
0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	311 case err != nil:
4197 5d7ce7f926eb Log fatal errors as errors, not warnings, in imports Tom Gottfried <tom@intevation.de> parents: 4177 diff changeset	312 feedback.Error(pgxutils.ReadableError{Err: err}.Error())
2937 0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	313 default:
0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	314 *features++
2726 dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	315 }
dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	316 return nil
dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	317 }

Mercurial > gemma

annotate pkg/imports/wx.go @ 5013:7dff1015283d