gemma: pkg/imports/dmv.go annotate

annotate pkg/imports/dmv.go @ 5670:b75d0b303328

Various fixes and improvements of gauges import: - Allow update of erased data (and thereby set erased to false) - Fix source_organization to work with ERDMS2 - Give ISRS of new and updated gauges in summary - Fixed reference of null pointers if revlevels are missing - Fixed reference of null pointer on update errors - Added ISRS to reference_code warning

author	Sascha Wilde <wilde@sha-bang.de>
date	Fri, 08 Dec 2023 17:29:56 +0100
parents	a4ecd66b5940
children	4bfcbb747768

rev	line source
1855 bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	7 // Copyright (C) 2018 by via donau
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	13
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	14 package imports
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16 import (
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17 "context"
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 "database/sql"
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	19 "errors"
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	20 "time"
1855 bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	21
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	22 "gemma.intevation.de/gemma/pkg/models"
1855 bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	23 )
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	24
4179 a376351d2774 Another round of doc strings to make 'golint' a little bit more happy with the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4177 diff changeset	25 // DistanceMarksVirtual is a Job to import virtual distance marks
a376351d2774 Another round of doc strings to make 'golint' a little bit more happy with the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4177 diff changeset	26 // from an external SOAP service into the database.
1855 bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	27 type DistanceMarksVirtual struct {
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	28 // URL is the URL of the SOAP service.
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	29 URL string `json:"url"`
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	30 // Username is the username used to authenticate.
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	31 Username string `json:"username"`
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	32 // Passwort is the password to authenticate.
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	33 Password string `json:"password"`
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	34 // Insecure indicates if HTTPS traffic
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	35 // should validate certificates or not.
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	36 Insecure bool `json:"insecure"`
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	37 }
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	38
4852 046a07a33b19 Fixed the golint issues of the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4799 diff changeset	39 // Description gives a short info about relevant facts of this import.
5560 f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5034 diff changeset	40 func (dmv *DistanceMarksVirtual) Description([]string) (string, error) {
4799 f32d086b5dbf Removed the mechanical touch of the last commit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4798 diff changeset	41 return dmv.URL, nil
4798 ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4179 diff changeset	42 }
ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4179 diff changeset	43
4179 a376351d2774 Another round of doc strings to make 'golint' a little bit more happy with the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4177 diff changeset	44 // DMVJobKind is the import queue type identifier.
1855 bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	45 const DMVJobKind JobKind = "dmv"
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	46
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	47 type dmvJobCreator struct{}
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	48
4179 a376351d2774 Another round of doc strings to make 'golint' a little bit more happy with the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4177 diff changeset	49 func init() { RegisterJobCreator(DMVJobKind, dmvJobCreator{}) }
1855 bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	50
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	51 func (dmvJobCreator) Description() string { return "distance marks virtual" }
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	52
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	53 func (dmvJobCreator) AutoAccept() bool { return true }
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	54
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	55 func (dmvJobCreator) Create() Job { return new(DistanceMarksVirtual) }
1855 bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	56
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3163 diff changeset	57 func (dmvJobCreator) Depends() [2][]string {
5020 e4ab338e7ba9 Removed unnecessary creatings of empty slices in expressing read dependencies in imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4852 diff changeset	58 return [2][]string{{"distance_marks_virtual"}}
1855 bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	59 }
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	60
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	61 // StageDone does nothing as there is no staging for distance marks virtual.
5034 59a99655f34d Added feedback support for StageDone. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5020 diff changeset	62 func (dmvJobCreator) StageDone(context.Context, *sql.Tx, int64, Feedback) error { return nil }
1855 bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	63
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	64 // CleanUp does nothing as there is nothing to cleanup with distance marks virtual.
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	65 func (*DistanceMarksVirtual) CleanUp() error { return nil }
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	66
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	67 const (
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	68 insertDistanceMarksVirtualSQL = `
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	69 INSERT INTO waterway.distance_marks_virtual (
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	70 location_code,
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	71 geom,
3778 1b11ff97c1d1 Add wwname to import of virtual distance marks. Sascha Wilde <wilde@intevation.de> parents: 3551 diff changeset	72 related_enc,
1b11ff97c1d1 Add wwname to import of virtual distance marks. Sascha Wilde <wilde@intevation.de> parents: 3551 diff changeset	73 wwname
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	74 )
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	75 VALUES (
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	76 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int),
1918 62b5f6e48e82 Distance marks virtual/Waterway gauge imports: Better fix for swapped Lat/Lon fields. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1917 diff changeset	77 ST_SetSRID(ST_MakePoint($6, $7), 4326)::geography,
3778 1b11ff97c1d1 Add wwname to import of virtual distance marks. Sascha Wilde <wilde@intevation.de> parents: 3551 diff changeset	78 $8,
1b11ff97c1d1 Add wwname to import of virtual distance marks. Sascha Wilde <wilde@intevation.de> parents: 3551 diff changeset	79 $9
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	80 ) ON CONFLICT (location_code) DO UPDATE SET
1918 62b5f6e48e82 Distance marks virtual/Waterway gauge imports: Better fix for swapped Lat/Lon fields. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1917 diff changeset	81 geom = ST_SetSRID(ST_MakePoint($6, $7), 4326)::geography,
3778 1b11ff97c1d1 Add wwname to import of virtual distance marks. Sascha Wilde <wilde@intevation.de> parents: 3551 diff changeset	82 related_enc = $8,
1b11ff97c1d1 Add wwname to import of virtual distance marks. Sascha Wilde <wilde@intevation.de> parents: 3551 diff changeset	83 wwname = $9
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	84 `
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	85 )
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	86
4179 a376351d2774 Another round of doc strings to make 'golint' a little bit more happy with the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4177 diff changeset	87 // Do executes the actual downloading and importing
a376351d2774 Another round of doc strings to make 'golint' a little bit more happy with the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4177 diff changeset	88 // of the virtual distance marks.
1855 bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	89 func (dmv *DistanceMarksVirtual) Do(
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	90 ctx context.Context,
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	91 importID int64,
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	92 conn *sql.Conn,
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	93 feedback Feedback,
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	94 ) (interface{}, error) {
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	95
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	96 start := time.Now()
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	97
3551 a5448426e4e2 Only erase gauges that have been requested from ERDMS Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	98 responseData, _, err := getRisData(
2633 eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	99 ctx,
3163 d9903cb34842 Handle failing INSERTs gracefully during gauges import Tom Gottfried <tom@intevation.de> parents: 2633 diff changeset	100 conn,
2633 eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	101 feedback,
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	102 dmv.Username,
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	103 dmv.Password,
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	104 dmv.URL,
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	105 dmv.Insecure,
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	106 "dismar")
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	107 if err != nil {
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	108 return nil, err
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	109 }
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	110
3163 d9903cb34842 Handle failing INSERTs gracefully during gauges import Tom Gottfried <tom@intevation.de> parents: 2633 diff changeset	111 tx, err := conn.BeginTx(ctx, nil)
d9903cb34842 Handle failing INSERTs gracefully during gauges import Tom Gottfried <tom@intevation.de> parents: 2633 diff changeset	112 if err != nil {
d9903cb34842 Handle failing INSERTs gracefully during gauges import Tom Gottfried <tom@intevation.de> parents: 2633 diff changeset	113 return nil, err
d9903cb34842 Handle failing INSERTs gracefully during gauges import Tom Gottfried <tom@intevation.de> parents: 2633 diff changeset	114 }
d9903cb34842 Handle failing INSERTs gracefully during gauges import Tom Gottfried <tom@intevation.de> parents: 2633 diff changeset	115 defer tx.Rollback()
d9903cb34842 Handle failing INSERTs gracefully during gauges import Tom Gottfried <tom@intevation.de> parents: 2633 diff changeset	116
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	117 insertStmt, err := tx.PrepareContext(ctx, insertDistanceMarksVirtualSQL)
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	118 if err != nil {
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	119 return nil, err
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	120 }
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	121 defer insertStmt.Close()
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	122
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	123 var ignored, features int
1855 bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	124
2633 eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	125 for _, data := range responseData {
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	126 for _, dr := range data.RisdataReturn {
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	127 if dr.RisidxCode == nil {
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	128 ignored++
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	129 continue
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	130 }
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	131
2633 eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	132 code, err := models.IsrsFromString(string(*dr.RisidxCode))
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	133 if err != nil {
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	134 feedback.Warn("invalid ISRS code %v", err)
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	135 ignored++
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	136 continue
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	137 }
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	138
2633 eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	139 if dr.Lat == nil \|\| dr.Lon == nil {
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	140 feedback.Warn("missing lat/lon: %s", code)
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	141 ignored++
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	142 continue
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	143 }
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	144
2633 eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	145 if dr.Relenc == nil {
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	146 feedback.Warn("missing relenc: %s", code)
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	147 ignored++
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	148 continue
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	149 }
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	150
5616 a4ecd66b5940 DMV: Don't accept relenc length greater 12. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5560 diff changeset	151 if len(*dr.Relenc) > 12 {
a4ecd66b5940 DMV: Don't accept relenc length greater 12. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5560 diff changeset	152 feedback.Warn("relenc too long: %s", *dr.Relenc)
a4ecd66b5940 DMV: Don't accept relenc length greater 12. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5560 diff changeset	153 ignored++
a4ecd66b5940 DMV: Don't accept relenc length greater 12. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5560 diff changeset	154 continue
a4ecd66b5940 DMV: Don't accept relenc length greater 12. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5560 diff changeset	155 }
a4ecd66b5940 DMV: Don't accept relenc length greater 12. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5560 diff changeset	156
3778 1b11ff97c1d1 Add wwname to import of virtual distance marks. Sascha Wilde <wilde@intevation.de> parents: 3551 diff changeset	157 if dr.Wwname.Loc == nil {
1b11ff97c1d1 Add wwname to import of virtual distance marks. Sascha Wilde <wilde@intevation.de> parents: 3551 diff changeset	158 feedback.Warn("missing wwname: %s", code)
1b11ff97c1d1 Add wwname to import of virtual distance marks. Sascha Wilde <wilde@intevation.de> parents: 3551 diff changeset	159 ignored++
1b11ff97c1d1 Add wwname to import of virtual distance marks. Sascha Wilde <wilde@intevation.de> parents: 3551 diff changeset	160 continue
1b11ff97c1d1 Add wwname to import of virtual distance marks. Sascha Wilde <wilde@intevation.de> parents: 3551 diff changeset	161 }
1b11ff97c1d1 Add wwname to import of virtual distance marks. Sascha Wilde <wilde@intevation.de> parents: 3551 diff changeset	162
2633 eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	163 if _, err := insertStmt.ExecContext(
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	164 ctx,
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	165 code.CountryCode,
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	166 code.LoCode,
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	167 code.FairwaySection,
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	168 code.Orc,
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	169 code.Hectometre,
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	170 float64(dr.Lon), float64(dr.Lat),
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	171 string(*dr.Relenc),
3778 1b11ff97c1d1 Add wwname to import of virtual distance marks. Sascha Wilde <wilde@intevation.de> parents: 3551 diff changeset	172 string(*dr.Wwname.Loc),
2633 eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	173 ); err != nil {
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	174 return nil, err
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	175 }
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	176 features++
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	177 }
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	178 }
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	179 feedback.Info("ignored: %d", ignored)
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	180 feedback.Info("features: %d", features)
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	181
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	182 if features == 0 {
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3778 diff changeset	183 return nil, errors.New("no features found")
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	184 }
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	185
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	186 if err = tx.Commit(); err == nil {
1994 a7c4005b723f Improve some import log messages Tom Gottfried <tom@intevation.de> parents: 1918 diff changeset	187 feedback.Info("Refreshing distance marks (virtual) successfully took %s.",
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	188 time.Since(start))
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	189 } else {
1994 a7c4005b723f Improve some import log messages Tom Gottfried <tom@intevation.de> parents: 1918 diff changeset	190 feedback.Error("Refreshing distance marks (virtual) failed after %s.",
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	191 time.Since(start))
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	192 }
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	193
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	194 return nil, nil
1855 bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	195 }

Mercurial > gemma

annotate pkg/imports/dmv.go @ 5670:b75d0b303328