gemma: pkg/imports/wa.go annotate

annotate pkg/imports/wa.go @ 5591:0011f50cf216 surveysperbottleneckid

Removed no longer used alternative api for surveys/ endpoint. As bottlenecks in the summary for SR imports are now identified by their id and no longer by the (not guarantied to be unique!) name, there is no longer the need to request survey data by the name+date tuple (which isn't reliable anyway). So the workaround was now reversed.

author	Sascha Wilde <wilde@sha-bang.de>
date	Wed, 06 Apr 2022 13:30:29 +0200
parents	ade07a3f2cfd
children	2dd155cc95ec

rev	line source
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	3 //
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	6 //
2649 9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping Tom Gottfried <tom@intevation.de> parents: 2622 diff changeset	7 // Copyright (C) 2018, 2019 by via donau
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	10 //
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	11 // Author(s):
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
2649 9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping Tom Gottfried <tom@intevation.de> parents: 2622 diff changeset	13 // * Tom Gottfried <tom.gottfried@intevation.de>
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	14
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	15 package imports
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	16
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	17 import (
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	18 "context"
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	19 "database/sql"
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	20 "encoding/json"
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	21 "errors"
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	22 "fmt"
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	23 "io"
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	24 "strconv"
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	25 "time"
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	26
4062 6c760abcff0e Move handling of PostgreSQL errors to own package Tom Gottfried <tom@intevation.de> parents: 4058 diff changeset	27 "gemma.intevation.de/gemma/pkg/pgxutils"
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	28 "gemma.intevation.de/gemma/pkg/wfs"
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	29 )
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	30
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	31 // WaterwayArea is an import job to import
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	32 // the waterway area in form of polygon geometries
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	33 // and attribute data from a WFS service.
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	34 type WaterwayArea struct {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	35 // URL the GetCapabilities URL of the WFS service.
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	36 URL string `json:"url"`
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	37 // FeatureType selects the feature type of the WFS service.
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	38 FeatureType string `json:"feature-type"`
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	39 // SortBy works around misconfigured services to
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	40 // establish a sort order to get the features.
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	41 SortBy string `json:"sort-by"`
2723 a10022399e24 WFS downloads: Fetch user and password from config. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2721 diff changeset	42 // User is an optional username for Basic Auth.
a10022399e24 WFS downloads: Fetch user and password from config. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2721 diff changeset	43 User string `json:"user,omitempty"`
a10022399e24 WFS downloads: Fetch user and password from config. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2721 diff changeset	44 // Password is an optional password for Basic Auth.
a10022399e24 WFS downloads: Fetch user and password from config. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2721 diff changeset	45 Password string `json:"password,omitempty"`
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	46 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	47
4852 046a07a33b19 Fixed the golint issues of the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4844 diff changeset	48 // Description gives a short info about relevant facts of this import.
5565 ade07a3f2cfd Forget to change signature of some imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5034 diff changeset	49 func (wa *WaterwayArea) Description([]string) (string, error) {
4799 f32d086b5dbf Removed the mechanical touch of the last commit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4798 diff changeset	50 return wa.URL + "\|" + wa.FeatureType, nil
4798 ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4197 diff changeset	51 }
ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4197 diff changeset	52
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	53 // WAJobKind is the import queue type identifier.
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	54 const WAJobKind JobKind = "wa"
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	55
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	56 type waJobCreator struct{}
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	57
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	58 func init() {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	59 RegisterJobCreator(WAJobKind, waJobCreator{})
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	60 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	61
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	62 func (waJobCreator) Description() string { return "waterway area" }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	63
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	64 func (waJobCreator) AutoAccept() bool { return true }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	65
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	66 func (waJobCreator) Create() Job { return new(WaterwayArea) }
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	67
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2938 diff changeset	68 func (waJobCreator) Depends() [2][]string {
5020 e4ab338e7ba9 Removed unnecessary creatings of empty slices in expressing read dependencies in imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4879 diff changeset	69 return [2][]string{{"waterway_area"}}
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	70 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	71
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	72 // StageDone is a NOP for waterway area imports.
5034 59a99655f34d Added feedback support for StageDone. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5020 diff changeset	73 func (waJobCreator) StageDone(context.Context, *sql.Tx, int64, Feedback) error {
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	74 return nil
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	75 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	76
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	77 // CleanUp for waterway area imports is a NOP.
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	78 func (*WaterwayArea) CleanUp() error { return nil }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	79
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	80 type waterwayAreaProperties struct {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	81 Catccl *string `json:"ienc_catccl"`
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	82 Dirimp *string `json:"ienc_dirimp"`
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	83 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	84
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	85 const (
1796 904c7e853adc Waterway area import: Clip features against responsibility area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	86 deleteWaterwayAreaSQL = `
904c7e853adc Waterway area import: Clip features against responsibility area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	87 WITH resp AS (
2912 93fa55bce126 Add utility function to get users area of responsibility Tom Gottfried <tom@intevation.de> parents: 2855 diff changeset	88 SELECT users.current_user_area_utm() AS a
1796 904c7e853adc Waterway area import: Clip features against responsibility area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	89 )
904c7e853adc Waterway area import: Clip features against responsibility area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	90 DELETE FROM waterway.waterway_area
2649 9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping Tom Gottfried <tom@intevation.de> parents: 2622 diff changeset	91 WHERE pg_has_role('sys_admin', 'MEMBER')
9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping Tom Gottfried <tom@intevation.de> parents: 2622 diff changeset	92 OR ST_Covers((SELECT a FROM resp),
2912 93fa55bce126 Add utility function to get users area of responsibility Tom Gottfried <tom@intevation.de> parents: 2855 diff changeset	93 ST_Transform(area::geometry, (SELECT ST_SRID(a) FROM resp)))
1796 904c7e853adc Waterway area import: Clip features against responsibility area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	94 `
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	95 insertWaterwayAreaSQL = `
1796 904c7e853adc Waterway area import: Clip features against responsibility area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	96 WITH resp AS (
2912 93fa55bce126 Add utility function to get users area of responsibility Tom Gottfried <tom@intevation.de> parents: 2855 diff changeset	97 SELECT users.current_user_area_utm() AS a
1796 904c7e853adc Waterway area import: Clip features against responsibility area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	98 )
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	99 INSERT INTO waterway.waterway_area (area, catccl, dirimp)
2855 8a8a929182f9 Enable import of axis and area independently from areas of responsibility Tom Gottfried <tom@intevation.de> parents: 2833 diff changeset	100 SELECT dmp.geom, $3, $4
8a8a929182f9 Enable import of axis and area independently from areas of responsibility Tom Gottfried <tom@intevation.de> parents: 2833 diff changeset	101 FROM ST_GeomFromWKB($1, $2::integer) AS new_area (new_area),
4844 58642179e896 Fix possibly invalid geometry output from transformation Tom Gottfried <tom@intevation.de> parents: 4832 diff changeset	102 ST_Dump(ST_CollectionExtract(ST_MakeValid(ST_Transform(
2649 9b6a02923bb4 Allow sys_admin to import waterway axis and area without clipping Tom Gottfried <tom@intevation.de> parents: 2622 diff changeset	103 CASE WHEN pg_has_role('sys_admin', 'MEMBER')
4844 58642179e896 Fix possibly invalid geometry output from transformation Tom Gottfried <tom@intevation.de> parents: 4832 diff changeset	104 THEN new_area
2855 8a8a929182f9 Enable import of axis and area independently from areas of responsibility Tom Gottfried <tom@intevation.de> parents: 2833 diff changeset	105 ELSE ST_Intersection((SELECT a FROM resp),
2912 93fa55bce126 Add utility function to get users area of responsibility Tom Gottfried <tom@intevation.de> parents: 2855 diff changeset	106 ST_MakeValid(ST_Transform(new_area, (SELECT ST_SRID(a) FROM resp))))
2855 8a8a929182f9 Enable import of axis and area independently from areas of responsibility Tom Gottfried <tom@intevation.de> parents: 2833 diff changeset	107 END,
4844 58642179e896 Fix possibly invalid geometry output from transformation Tom Gottfried <tom@intevation.de> parents: 4832 diff changeset	108 4326)), 3)) AS dmp
2938 ecb1e34157a3 Fix logging in area import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	109 RETURNING id
1796 904c7e853adc Waterway area import: Clip features against responsibility area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	110 `
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	111 )
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	112
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	113 // Do executes the actual waterway axis import.
4852 046a07a33b19 Fixed the golint issues of the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4844 diff changeset	114 func (wa *WaterwayArea) Do(
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	115 ctx context.Context,
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	116 importID int64,
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	117 conn *sql.Conn,
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	118 feedback Feedback,
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	119 ) (interface{}, error) {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	120
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	121 start := time.Now()
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	122
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	123 feedback.Info("Import waterway area")
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	124
4852 046a07a33b19 Fixed the golint issues of the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4844 diff changeset	125 feedback.Info("Loading capabilities from %s", wa.URL)
046a07a33b19 Fixed the golint issues of the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4844 diff changeset	126 caps, err := wfs.GetCapabilities(wa.URL)
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	127 if err != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	128 feedback.Error("Loading capabilities failed: %v", err)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	129 return nil, err
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	130 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	131
4852 046a07a33b19 Fixed the golint issues of the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4844 diff changeset	132 ft := caps.FindFeatureType(wa.FeatureType)
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	133 if ft == nil {
4852 046a07a33b19 Fixed the golint issues of the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4844 diff changeset	134 return nil, fmt.Errorf("unknown feature type '%s'", wa.FeatureType)
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	135 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	136
4879 783d0bec86d3 Add missing quotation marks Tom Gottfried <tom@intevation.de> parents: 4852 diff changeset	137 feedback.Info("Found feature type '%s'", wa.FeatureType)
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	138
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	139 epsg, err := wfs.CRSToEPSG(ft.DefaultCRS)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	140 if err != nil {
4832 5062ccb2381d Unified error message on "Unsupported CRS" Sascha Wilde <wilde@intevation.de> parents: 4799 diff changeset	141 feedback.Error("Unsupported CRS: '%s'", ft.DefaultCRS)
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	142 return nil, err
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	143 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	144
4852 046a07a33b19 Fixed the golint issues of the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4844 diff changeset	145 if wa.SortBy != "" {
046a07a33b19 Fixed the golint issues of the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4844 diff changeset	146 feedback.Info("Features will be sorted by '%s'", wa.SortBy)
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	147 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	148
4852 046a07a33b19 Fixed the golint issues of the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4844 diff changeset	149 dl, err := wfs.GetFeatures(caps, wa.FeatureType, wa.SortBy)
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	150 if err != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	151 feedback.Error("Cannot create GetFeature URLs. %v", err)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	152 return nil, err
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	153 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	154
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	155 tx, err := conn.BeginTx(ctx, nil)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	156 if err != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	157 return nil, err
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	158 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	159 defer tx.Rollback()
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	160
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	161 insertStmt, err := tx.PrepareContext(ctx, insertWaterwayAreaSQL)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	162 if err != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	163 return nil, err
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	164 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	165 defer insertStmt.Close()
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	166
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	167 // Delete the old features.
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	168 if _, err := tx.ExecContext(ctx, deleteWaterwayAreaSQL); err != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	169 return nil, err
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	170 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	171
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	172 var (
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	173 unsupported = stringCounter{}
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	174 missingProperties int
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	175 badProperties int
2938 ecb1e34157a3 Fix logging in area import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	176 outside int
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	177 features int
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	178 )
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	179
4852 046a07a33b19 Fixed the golint issues of the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4844 diff changeset	180 if err := dl.Download(wa.User, wa.Password, func(url string, r io.Reader) error {
2106 2b72f5e005aa WFS imports: Write get GetFeature URLs into import log. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1899 diff changeset	181 feedback.Info("Get features from: '%s'", url)
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	182 rfc, err := wfs.ParseRawFeatureCollection(r)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	183 if err != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	184 return fmt.Errorf("parsing GetFeature document failed: %v", err)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	185 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	186 if rfc.CRS != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	187 crsName := rfc.CRS.Properties.Name
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	188 if epsg, err = wfs.CRSToEPSG(crsName); err != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	189 feedback.Error("Unsupported CRS: %d", crsName)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	190 return err
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	191 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	192 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	193
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	194 // No features -> ignore.
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	195 if rfc.Features == nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	196 return nil
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	197 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	198
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	199 feedback.Info("Using EPSG: %d", epsg)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	200
2758 a996f2ca9fa5 Simplified savepoint handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2729 diff changeset	201 savepoint := Savepoint(ctx, tx, "feature")
2659 00b0a7b2225a Waterway area import: do not fail if INSERT of one item fails Tom Gottfried <tom@intevation.de> parents: 2649 diff changeset	202
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	203 for _, feature := range rfc.Features {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	204 if feature.Properties == nil \|\| feature.Geometry.Coordinates == nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	205 missingProperties++
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	206 continue
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	207 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	208
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	209 var props waterwayAreaProperties
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	210
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	211 if err := json.Unmarshal(*feature.Properties, &props); err != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	212 badProperties++
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	213 continue
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	214 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	215
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	216 var catccl sql.NullInt64
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	217 if props.Catccl != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	218 if value, err := strconv.ParseInt(*props.Catccl, 10, 64); err == nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	219 catccl = sql.NullInt64{Int64: value, Valid: true}
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	220 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	221 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	222 var dirimp sql.NullInt64
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	223 if props.Dirimp != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	224 if value, err := strconv.ParseInt(*props.Dirimp, 10, 64); err == nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	225 dirimp = sql.NullInt64{Int64: value, Valid: true}
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	226 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	227 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	228
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	229 switch feature.Geometry.Type {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	230 case "Polygon":
1786 09349ca27dd7 Imports: Removed duplicated code path to store WKB polygons into database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1785 diff changeset	231 var p polygonSlice
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	232 if err := json.Unmarshal(*feature.Geometry.Coordinates, &p); err != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	233 return err
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	234 }
2938 ecb1e34157a3 Fix logging in area import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	235 var waid int64
ecb1e34157a3 Fix logging in area import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	236 err := savepoint(func() error {
ecb1e34157a3 Fix logging in area import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	237 err := insertStmt.QueryRowContext(
2758 a996f2ca9fa5 Simplified savepoint handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2729 diff changeset	238 ctx,
a996f2ca9fa5 Simplified savepoint handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2729 diff changeset	239 p.asWKB(),
a996f2ca9fa5 Simplified savepoint handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2729 diff changeset	240 epsg,
a996f2ca9fa5 Simplified savepoint handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2729 diff changeset	241 catccl,
a996f2ca9fa5 Simplified savepoint handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2729 diff changeset	242 dirimp,
2938 ecb1e34157a3 Fix logging in area import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	243 ).Scan(&waid)
2659 00b0a7b2225a Waterway area import: do not fail if INSERT of one item fails Tom Gottfried <tom@intevation.de> parents: 2649 diff changeset	244 return err
2938 ecb1e34157a3 Fix logging in area import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	245 })
ecb1e34157a3 Fix logging in area import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	246 switch {
ecb1e34157a3 Fix logging in area import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	247 case err == sql.ErrNoRows:
ecb1e34157a3 Fix logging in area import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	248 outside++
ecb1e34157a3 Fix logging in area import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	249 // ignore -> filtered by responsibility_areas
ecb1e34157a3 Fix logging in area import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	250 case err != nil:
4197 5d7ce7f926eb Log fatal errors as errors, not warnings, in imports Tom Gottfried <tom@intevation.de> parents: 4177 diff changeset	251 feedback.Error(pgxutils.ReadableError{Err: err}.Error())
2938 ecb1e34157a3 Fix logging in area import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	252 default:
2659 00b0a7b2225a Waterway area import: do not fail if INSERT of one item fails Tom Gottfried <tom@intevation.de> parents: 2649 diff changeset	253 features++
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	254 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	255 default:
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	256 unsupported[feature.Geometry.Type]++
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	257 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	258 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	259 return nil
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	260 }); err != nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	261 return nil, err
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	262 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	263
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	264 if badProperties > 0 {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	265 feedback.Warn("Bad properties: %d", badProperties)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	266 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	267
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	268 if missingProperties > 0 {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	269 feedback.Warn("Missing properties: %d", missingProperties)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	270 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	271
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	272 if len(unsupported) != 0 {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	273 feedback.Warn("Unsupported types found: %s", unsupported)
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	274 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	275
2938 ecb1e34157a3 Fix logging in area import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	276 if outside > 0 {
ecb1e34157a3 Fix logging in area import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	277 feedback.Info("Features outside responsibility area: %d", outside)
ecb1e34157a3 Fix logging in area import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	278 }
ecb1e34157a3 Fix logging in area import Tom Gottfried <tom@intevation.de> parents: 2912 diff changeset	279
1828 1ecfcf46e4da Waterway area/axis: Be more informative if features are bad. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1796 diff changeset	280 if features == 0 {
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4147 diff changeset	281 err := errors.New("no features found")
1828 1ecfcf46e4da Waterway area/axis: Be more informative if features are bad. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1796 diff changeset	282 feedback.Error("%v", err)
1ecfcf46e4da Waterway area/axis: Be more informative if features are bad. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1796 diff changeset	283 return nil, err
1ecfcf46e4da Waterway area/axis: Be more informative if features are bad. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1796 diff changeset	284 }
1ecfcf46e4da Waterway area/axis: Be more informative if features are bad. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1796 diff changeset	285
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	286 if err = tx.Commit(); err == nil {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	287 feedback.Info("Storing %d features took %s",
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	288 features, time.Since(start))
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	289 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	290
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	291 return nil, err
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: diff changeset	292 }

Mercurial > gemma

annotate pkg/imports/wa.go @ 5591:0011f50cf216 surveysperbottleneckid