gemma: pkg/imports/wx.go annotate

annotate pkg/imports/wx.go @ 3277:232fc90e6ee2

Disentangle gauge measurements and predictions Representing both in one table has led to the necessity to make the distinction at many places such as statements, definitions of partial indexes and application code. At least in one place in the AGM import the distinction in application code was too late and measurements matching an approved measurement could have been missed.

author	Tom Gottfried <tom@intevation.de>
date	Wed, 15 May 2019 19:08:49 +0200
parents	4acbee65275d
children	6c5c15b2fb64

rev	line source
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
2649 9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping Tom Gottfried <tom@intevation.de> parents: 2622 diff changeset	7 // Copyright (C) 2018, 2019 by via donau
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
2649 9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping Tom Gottfried <tom@intevation.de> parents: 2622 diff changeset	13 // * Tom Gottfried <tom.gottfried@intevation.de>
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	14
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15 package imports
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17 import (
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 "context"
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	19 "database/sql"
1680 de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	20 "encoding/json"
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	21 "errors"
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	22 "fmt"
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	23 "io"
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	24 "time"
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	25
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	26 "gemma.intevation.de/gemma/pkg/wfs"
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	27 )
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	28
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1683 diff changeset	29 // WaterwayAxis is an import job to import
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1683 diff changeset	30 // the waterway axes in form of line string geometries
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1683 diff changeset	31 // and attribute data from a WFS service.
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	32 type WaterwayAxis struct {
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1683 diff changeset	33 // URL the GetCapabilities URL of the WFS service.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1683 diff changeset	34 URL string `json:"url"`
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1683 diff changeset	35 // FeatureType selects the feature type of the WFS service.
1677 53304db85888 Waterway axis import: Added route for manual import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1675 diff changeset	36 FeatureType string `json:"feature-type"`
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1683 diff changeset	37 // SortBy works around misconfigured services to
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1683 diff changeset	38 // establish a sort order to get the features.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1683 diff changeset	39 SortBy string `json:"sort-by"`
2723 a10022399e24 WFS downloads: Fetch user and password from config. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2721 diff changeset	40 // User is an optional username for Basic Auth.
a10022399e24 WFS downloads: Fetch user and password from config. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2721 diff changeset	41 User string `json:"user,omitempty"`
a10022399e24 WFS downloads: Fetch user and password from config. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2721 diff changeset	42 // Password is an optional password for Basic Auth.
a10022399e24 WFS downloads: Fetch user and password from config. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2721 diff changeset	43 Password string `json:"password,omitempty"`
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	44 }
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	45
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1683 diff changeset	46 // WXJobKind is the import queue type identifier.
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	47 const WXJobKind JobKind = "wx"
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	48
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	49 type wxJobCreator struct{}
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	50
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	51 func init() {
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	52 RegisterJobCreator(WXJobKind, wxJobCreator{})
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	53 }
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	54
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1706 diff changeset	55 func (wxJobCreator) Description() string { return "waterway axis" }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1706 diff changeset	56
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1706 diff changeset	57 func (wxJobCreator) AutoAccept() bool { return true }
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	58
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	59 func (wxJobCreator) Create() Job { return new(WaterwayAxis) }
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	60
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3037 diff changeset	61 func (wxJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3037 diff changeset	62 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3037 diff changeset	63 {"waterway_axis"},
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3037 diff changeset	64 {},
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	65 }
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	66 }
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	67
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	68 // StageDone is a NOP for waterway axis imports.
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	69 func (wxJobCreator) StageDone(context.Context, *sql.Tx, int64) error {
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	70 return nil
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	71 }
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	72
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1764 diff changeset	73 // CleanUp for waterway axis imports is a NOP.
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	74 func (*WaterwayAxis) CleanUp() error { return nil }
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	75
1680 de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	76 type waterwayAxisProperties struct {
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	77 ObjNam string `json:"hydro_objnam"`
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	78 NObjNnm *string `json:"hydro_nobjnm"`
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	79 }
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	80
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	81 const (
1795 1333f96f18d0 Waterway axis: Clip features against responsibility area of importing user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1785 diff changeset	82 deleteWaterwayAxisSQL = `
1333f96f18d0 Waterway axis: Clip features against responsibility area of importing user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1785 diff changeset	83 WITH resp AS (
2912 93fa55bce126 Add utility function to get users area of responsibility Tom Gottfried <tom@intevation.de> parents: 2903 diff changeset	84 SELECT users.current_user_area_utm() AS a
1795 1333f96f18d0 Waterway axis: Clip features against responsibility area of importing user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1785 diff changeset	85 )
1333f96f18d0 Waterway axis: Clip features against responsibility area of importing user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1785 diff changeset	86 DELETE FROM waterway.waterway_axis
2649 9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping Tom Gottfried <tom@intevation.de> parents: 2622 diff changeset	87 WHERE pg_has_role('sys_admin', 'MEMBER')
9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping Tom Gottfried <tom@intevation.de> parents: 2622 diff changeset	88 OR ST_Covers((SELECT a FROM resp),
2912 93fa55bce126 Add utility function to get users area of responsibility Tom Gottfried <tom@intevation.de> parents: 2903 diff changeset	89 ST_Transform(wtwaxs::geometry, (SELECT ST_SRID(a) FROM resp)))
1795 1333f96f18d0 Waterway axis: Clip features against responsibility area of importing user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1785 diff changeset	90 `
1333f96f18d0 Waterway axis: Clip features against responsibility area of importing user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1785 diff changeset	91
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	92 insertWaterwayAxisSQL = `
1795 1333f96f18d0 Waterway axis: Clip features against responsibility area of importing user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1785 diff changeset	93 WITH resp AS (
2912 93fa55bce126 Add utility function to get users area of responsibility Tom Gottfried <tom@intevation.de> parents: 2903 diff changeset	94 SELECT users.current_user_area_utm() AS a
1795 1333f96f18d0 Waterway axis: Clip features against responsibility area of importing user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1785 diff changeset	95 )
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	96 INSERT INTO waterway.waterway_axis (wtwaxs, objnam, nobjnam)
2855 8a8a929182f9 Enable import of axis and area independently from areas of responsibility Tom Gottfried <tom@intevation.de> parents: 2833 diff changeset	97 SELECT dmp.geom, $3, $4
8a8a929182f9 Enable import of axis and area independently from areas of responsibility Tom Gottfried <tom@intevation.de> parents: 2833 diff changeset	98 FROM ST_GeomFromWKB($1, $2::integer) AS new_line (new_line),
8a8a929182f9 Enable import of axis and area independently from areas of responsibility Tom Gottfried <tom@intevation.de> parents: 2833 diff changeset	99 ST_Dump(ST_Transform(ST_CollectionExtract(
2649 9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping Tom Gottfried <tom@intevation.de> parents: 2622 diff changeset	100 CASE WHEN pg_has_role('sys_admin', 'MEMBER')
2855 8a8a929182f9 Enable import of axis and area independently from areas of responsibility Tom Gottfried <tom@intevation.de> parents: 2833 diff changeset	101 THEN ST_Node(ST_Transform(new_line,
8a8a929182f9 Enable import of axis and area independently from areas of responsibility Tom Gottfried <tom@intevation.de> parents: 2833 diff changeset	102 best_utm(ST_Transform(new_line, 4326))))
8a8a929182f9 Enable import of axis and area independently from areas of responsibility Tom Gottfried <tom@intevation.de> parents: 2833 diff changeset	103 ELSE ST_Intersection((SELECT a FROM resp),
2912 93fa55bce126 Add utility function to get users area of responsibility Tom Gottfried <tom@intevation.de> parents: 2903 diff changeset	104 ST_Node(ST_Transform(new_line, (SELECT ST_SRID(a) FROM resp))))
2855 8a8a929182f9 Enable import of axis and area independently from areas of responsibility Tom Gottfried <tom@intevation.de> parents: 2833 diff changeset	105 END,
8a8a929182f9 Enable import of axis and area independently from areas of responsibility Tom Gottfried <tom@intevation.de> parents: 2833 diff changeset	106 2), 4326)) AS dmp
2726 dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	107 RETURNING id
1795 1333f96f18d0 Waterway axis: Clip features against responsibility area of importing user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1785 diff changeset	108 `
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	109 )
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	110
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1764 diff changeset	111 // Do executes the actual waterway axis import.
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	112 func (wx *WaterwayAxis) Do(
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	113 ctx context.Context,
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	114 importID int64,
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	115 conn *sql.Conn,
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	116 feedback Feedback,
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	117 ) (interface{}, error) {
8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	118
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	119 start := time.Now()
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	120
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	121 feedback.Info("Import waterway axis")
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	122
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	123 feedback.Info("Loading capabilities from %s", wx.URL)
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	124 caps, err := wfs.GetCapabilities(wx.URL)
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	125 if err != nil {
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	126 feedback.Error("Loading capabilities failed: %v", err)
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	127 return nil, err
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	128 }
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	129
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	130 ft := caps.FindFeatureType(wx.FeatureType)
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	131 if ft == nil {
1682 3c99d599503a Waterway axis import: small simplifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1681 diff changeset	132 return nil, fmt.Errorf("Unknown feature type '%s'", wx.FeatureType)
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	133 }
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	134
1706 fb05027d93b6 Waterway axis: Be more verbose during import about feature type and sorting. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	135 feedback.Info("Found feature type '%s", wx.FeatureType)
fb05027d93b6 Waterway axis: Be more verbose during import about feature type and sorting. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	136
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	137 epsg, err := wfs.CRSToEPSG(ft.DefaultCRS)
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	138 if err != nil {
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	139 feedback.Error("Unsupported CRS name '%s'", ft.DefaultCRS)
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	140 return nil, err
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	141 }
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	142
1706 fb05027d93b6 Waterway axis: Be more verbose during import about feature type and sorting. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	143 if wx.SortBy != "" {
fb05027d93b6 Waterway axis: Be more verbose during import about feature type and sorting. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	144 feedback.Info("Features will be sorted by '%s'", wx.SortBy)
fb05027d93b6 Waterway axis: Be more verbose during import about feature type and sorting. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	145 }
fb05027d93b6 Waterway axis: Be more verbose during import about feature type and sorting. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	146
2721 56c8ef515d93 WFS downloader: Use new GML/GeoJSON factory function in gemma. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2719 diff changeset	147 dl, err := wfs.GetFeatures(caps, wx.FeatureType, wx.SortBy)
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	148 if err != nil {
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	149 feedback.Error("Cannot create GetFeature URLs. %v", err)
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	150 return nil, err
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	151 }
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	152
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	153 tx, err := conn.BeginTx(ctx, nil)
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	154 if err != nil {
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	155 return nil, err
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	156 }
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	157 defer tx.Rollback()
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	158
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	159 insertStmt, err := tx.PrepareContext(ctx, insertWaterwayAxisSQL)
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	160 if err != nil {
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	161 return nil, err
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	162 }
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	163 defer insertStmt.Close()
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	164
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	165 // Delete the old features.
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	166 if _, err := tx.ExecContext(ctx, deleteWaterwayAxisSQL); err != nil {
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	167 return nil, err
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	168 }
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	169
1680 de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	170 var (
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1764 diff changeset	171 unsupported = stringCounter{}
1680 de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	172 missingProperties int
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	173 badProperties int
2937 0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	174 outside int
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	175 features int
1680 de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	176 )
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	177
2729 7cb027be277d WFS downloader: Really use username and password from parameters. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2726 diff changeset	178 if err := dl.Download(wx.User, wx.Password, func(url string, r io.Reader) error {
2106 2b72f5e005aa WFS imports: Write get GetFeature URLs into import log. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1828 diff changeset	179 feedback.Info("Get features from: '%s'", url)
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	180 rfc, err := wfs.ParseRawFeatureCollection(r)
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	181 if err != nil {
1764 bb4348ac52ab Waterway axis import: Be more verbose when parsing the GetFeature document fails. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1754 diff changeset	182 return fmt.Errorf("parsing GetFeature document failed: %v", err)
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	183 }
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	184 if rfc.CRS != nil {
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	185 crsName := rfc.CRS.Properties.Name
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	186 if epsg, err = wfs.CRSToEPSG(crsName); err != nil {
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	187 feedback.Error("Unsupported CRS: %d", crsName)
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	188 return err
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	189 }
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	190 }
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	191
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	192 // No features -> ignore.
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	193 if rfc.Features == nil {
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	194 return nil
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	195 }
1680 de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	196
1683 509e8728e846 Fixed pre-calculated size of 2D WKB line strings. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1682 diff changeset	197 feedback.Info("Using EPSG: %d", epsg)
509e8728e846 Fixed pre-calculated size of 2D WKB line strings. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1682 diff changeset	198
2780 5850ac0f9ab6 Axis import: do not fail if one item fails. Tom Gottfried <tom@intevation.de> parents: 2730 diff changeset	199 savepoint := Savepoint(ctx, tx, "feature")
5850ac0f9ab6 Axis import: do not fail if one item fails. Tom Gottfried <tom@intevation.de> parents: 2730 diff changeset	200
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	201 for _, feature := range rfc.Features {
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	202 if feature.Properties == nil \|\| feature.Geometry.Coordinates == nil {
1680 de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	203 missingProperties++
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	204 continue
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	205 }
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	206
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	207 var props waterwayAxisProperties
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	208
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	209 if err := json.Unmarshal(*feature.Properties, &props); err != nil {
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	210 badProperties++
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	211 continue
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	212 }
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	213
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	214 var nobjnam sql.NullString
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	215 if props.NObjNnm != nil {
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	216 nobjnam = sql.NullString{String: *props.NObjNnm, Valid: true}
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	217 }
1680 de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	218
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	219 switch feature.Geometry.Type {
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	220 case "LineString":
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1764 diff changeset	221 var l lineSlice
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	222 if err := json.Unmarshal(*feature.Geometry.Coordinates, &l); err != nil {
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	223 return err
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	224 }
2726 dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	225 if err := storeLinestring(
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	226 ctx,
2780 5850ac0f9ab6 Axis import: do not fail if one item fails. Tom Gottfried <tom@intevation.de> parents: 2730 diff changeset	227 savepoint,
2726 dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	228 feedback,
dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	229 l,
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	230 epsg,
2726 dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	231 props,
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	232 nobjnam,
2937 0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	233 &outside,
0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	234 &features,
2726 dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	235 insertStmt); err != nil {
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	236 return err
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	237 }
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	238 case "MultiLineString":
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1764 diff changeset	239 var ls []lineSlice
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	240 if err := json.Unmarshal(*feature.Geometry.Coordinates, &ls); err != nil {
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	241 return err
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	242 }
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	243 for _, l := range ls {
2726 dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	244 if err := storeLinestring(
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	245 ctx,
2780 5850ac0f9ab6 Axis import: do not fail if one item fails. Tom Gottfried <tom@intevation.de> parents: 2730 diff changeset	246 savepoint,
2726 dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	247 feedback,
dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	248 l,
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	249 epsg,
2726 dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	250 props,
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	251 nobjnam,
2937 0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	252 &outside,
0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	253 &features,
2726 dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	254 insertStmt); err != nil {
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	255 return err
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	256 }
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	257 }
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	258 default:
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1764 diff changeset	259 unsupported[feature.Geometry.Type]++
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	260 }
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	261 }
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	262 return nil
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	263 }); err != nil {
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	264 return nil, err
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	265 }
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	266
1680 de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	267 if badProperties > 0 {
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	268 feedback.Warn("Bad properties: %d", badProperties)
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	269 }
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	270
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	271 if missingProperties > 0 {
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	272 feedback.Warn("Missing properties: %d", missingProperties)
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	273 }
de8089944b19 Waterway axis import: Started extracting properties from the features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1679 diff changeset	274
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1764 diff changeset	275 if len(unsupported) != 0 {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1764 diff changeset	276 feedback.Warn("Unsupported types found: %s", unsupported)
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	277 }
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	278
2937 0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	279 if outside > 0 {
0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	280 feedback.Info("Features outside responsibility area: %d", outside)
0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	281 }
0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	282
1828 1ecfcf46e4da Waterway area/axis: Be more informative if features are bad. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1796 diff changeset	283 if features == 0 {
2903 813671873536 Waterway axis import: Don't log error twice if no feature was imported. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2855 diff changeset	284 return nil, errors.New("No features found")
1828 1ecfcf46e4da Waterway area/axis: Be more informative if features are bad. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1796 diff changeset	285 }
1ecfcf46e4da Waterway area/axis: Be more informative if features are bad. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1796 diff changeset	286
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	287 if err = tx.Commit(); err == nil {
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	288 feedback.Info("Storing %d features took %s",
4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	289 features, time.Since(start))
1679 2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	290 }
2dc7768be0e4 Waterway axis import: More on reading data from WFS. TODO: Parse to concrete features. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1677 diff changeset	291
1681 4d6ce621379e Waterway axis import: Completed, but the final commit unexpectedly resulted in rollback. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1680 diff changeset	292 return nil, err
1675 8fec3887c7e5 Waterway axis import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	293 }
2726 dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	294
dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	295 func storeLinestring(
dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	296 ctx context.Context,
2780 5850ac0f9ab6 Axis import: do not fail if one item fails. Tom Gottfried <tom@intevation.de> parents: 2730 diff changeset	297 savepoint func(func() error) error,
2726 dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	298 feedback Feedback,
dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	299 l lineSlice,
dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	300 epsg int,
dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	301 props waterwayAxisProperties,
dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	302 nobjnam sql.NullString,
2937 0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	303 outside, features *int,
3037 72451510d417 Remove expensive check with doubtful usefulness Tom Gottfried <tom@intevation.de> parents: 2937 diff changeset	304 insertStmt *sql.Stmt,
2726 dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	305 ) error {
dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	306 var id int
2937 0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	307 err := savepoint(func() error {
2780 5850ac0f9ab6 Axis import: do not fail if one item fails. Tom Gottfried <tom@intevation.de> parents: 2730 diff changeset	308 err := insertStmt.QueryRowContext(
5850ac0f9ab6 Axis import: do not fail if one item fails. Tom Gottfried <tom@intevation.de> parents: 2730 diff changeset	309 ctx,
5850ac0f9ab6 Axis import: do not fail if one item fails. Tom Gottfried <tom@intevation.de> parents: 2730 diff changeset	310 l.asWKB(),
5850ac0f9ab6 Axis import: do not fail if one item fails. Tom Gottfried <tom@intevation.de> parents: 2730 diff changeset	311 epsg,
5850ac0f9ab6 Axis import: do not fail if one item fails. Tom Gottfried <tom@intevation.de> parents: 2730 diff changeset	312 props.ObjNam,
5850ac0f9ab6 Axis import: do not fail if one item fails. Tom Gottfried <tom@intevation.de> parents: 2730 diff changeset	313 nobjnam,
5850ac0f9ab6 Axis import: do not fail if one item fails. Tom Gottfried <tom@intevation.de> parents: 2730 diff changeset	314 ).Scan(&id)
2726 dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	315 return err
2937 0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	316 })
0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	317 switch {
0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	318 case err == sql.ErrNoRows:
0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	319 *outside++
0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	320 // ignore -> filtered by responsibility_areas
0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	321 return nil
0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	322 case err != nil:
2780 5850ac0f9ab6 Axis import: do not fail if one item fails. Tom Gottfried <tom@intevation.de> parents: 2730 diff changeset	323 feedback.Warn(handleError(err).Error())
2937 0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	324 default:
0d79f5eec630 Fix logging in axis import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	325 *features++
2726 dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	326 }
dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	327 return nil
dff749c07ae0 Warn if imported axis linestrings cross each other Tom Gottfried <tom@intevation.de> parents: 2723 diff changeset	328 }

Mercurial > gemma

annotate pkg/imports/wx.go @ 3277:232fc90e6ee2