gemma: pkg/imports/dmv.go annotate

annotate pkg/imports/dmv.go @ 3163:d9903cb34842

Handle failing INSERTs gracefully during gauges import Using the special table EXCLUDED in INSERT statements makes functionally no difference, but makes editing of the statements easier. Since reference water levels are not deleted all at once before (re-)importing anymore, take the chance to report those that were deleted.

author	Tom Gottfried <tom@intevation.de>
date	Mon, 06 May 2019 13:25:49 +0200
parents	eb1d119f253f
children	4acbee65275d

rev	line source
1855 bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	7 // Copyright (C) 2018 by via donau
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	13
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	14 package imports
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16 import (
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17 "context"
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 "database/sql"
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	19 "errors"
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	20 "time"
1855 bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	21
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	22 "gemma.intevation.de/gemma/pkg/models"
1855 bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	23 )
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	24
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	25 type DistanceMarksVirtual struct {
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	26 // URL is the URL of the SOAP service.
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	27 URL string `json:"url"`
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	28 // Username is the username used to authenticate.
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	29 Username string `json:"username"`
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	30 // Passwort is the password to authenticate.
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	31 Password string `json:"password"`
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	32 // Insecure indicates if HTTPS traffic
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	33 // should validate certificates or not.
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	34 Insecure bool `json:"insecure"`
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	35 }
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	36
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	37 const DMVJobKind JobKind = "dmv"
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	38
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	39 type dmvJobCreator struct{}
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	40
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	41 func init() {
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	42 RegisterJobCreator(DMVJobKind, dmvJobCreator{})
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	43 }
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	44
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	45 func (dmvJobCreator) Description() string { return "distance marks virtual" }
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	46
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	47 func (dmvJobCreator) AutoAccept() bool { return true }
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	48
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	49 func (dmvJobCreator) Create() Job { return new(DistanceMarksVirtual) }
1855 bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	50
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	51 func (dmvJobCreator) Depends() []string {
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	52 return []string{
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	53 "distance_marks_virtual",
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	54 }
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	55 }
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	56
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	57 // StageDone does nothing as there is no staging for distance marks virtual.
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	58 func (dmvJobCreator) StageDone(context.Context, *sql.Tx, int64) error { return nil }
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	59
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	60 // CleanUp does nothing as there is nothing to cleanup with distance marks virtual.
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	61 func (*DistanceMarksVirtual) CleanUp() error { return nil }
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	62
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	63 const (
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	64 insertDistanceMarksVirtualSQL = `
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	65 INSERT INTO waterway.distance_marks_virtual (
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	66 location_code,
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	67 geom,
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	68 related_enc
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	69 )
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	70 VALUES (
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	71 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int),
1918 62b5f6e48e82 Distance marks virtual/Waterway gauge imports: Better fix for swapped Lat/Lon fields. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1917 diff changeset	72 ST_SetSRID(ST_MakePoint($6, $7), 4326)::geography,
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	73 $8
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	74 ) ON CONFLICT (location_code) DO UPDATE SET
1918 62b5f6e48e82 Distance marks virtual/Waterway gauge imports: Better fix for swapped Lat/Lon fields. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1917 diff changeset	75 geom = ST_SetSRID(ST_MakePoint($6, $7), 4326)::geography,
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	76 related_enc = $8
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	77 `
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	78 )
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	79
1855 bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	80 func (dmv *DistanceMarksVirtual) Do(
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	81 ctx context.Context,
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	82 importID int64,
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	83 conn *sql.Conn,
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	84 feedback Feedback,
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	85 ) (interface{}, error) {
bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	86
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	87 start := time.Now()
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	88
2633 eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	89 responseData, err := getRisData(
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	90 ctx,
3163 d9903cb34842 Handle failing INSERTs gracefully during gauges import Tom Gottfried <tom@intevation.de> parents: 2633 diff changeset	91 conn,
2633 eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	92 feedback,
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	93 dmv.Username,
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	94 dmv.Password,
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	95 dmv.URL,
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	96 dmv.Insecure,
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	97 "dismar")
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	98 if err != nil {
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	99 return nil, err
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	100 }
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	101
3163 d9903cb34842 Handle failing INSERTs gracefully during gauges import Tom Gottfried <tom@intevation.de> parents: 2633 diff changeset	102 tx, err := conn.BeginTx(ctx, nil)
d9903cb34842 Handle failing INSERTs gracefully during gauges import Tom Gottfried <tom@intevation.de> parents: 2633 diff changeset	103 if err != nil {
d9903cb34842 Handle failing INSERTs gracefully during gauges import Tom Gottfried <tom@intevation.de> parents: 2633 diff changeset	104 return nil, err
d9903cb34842 Handle failing INSERTs gracefully during gauges import Tom Gottfried <tom@intevation.de> parents: 2633 diff changeset	105 }
d9903cb34842 Handle failing INSERTs gracefully during gauges import Tom Gottfried <tom@intevation.de> parents: 2633 diff changeset	106 defer tx.Rollback()
d9903cb34842 Handle failing INSERTs gracefully during gauges import Tom Gottfried <tom@intevation.de> parents: 2633 diff changeset	107
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	108 insertStmt, err := tx.PrepareContext(ctx, insertDistanceMarksVirtualSQL)
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	109 if err != nil {
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	110 return nil, err
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	111 }
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	112 defer insertStmt.Close()
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	113
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	114 var ignored, features int
1855 bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	115
2633 eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	116 for _, data := range responseData {
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	117 for _, dr := range data.RisdataReturn {
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	118 if dr.RisidxCode == nil {
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	119 ignored++
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	120 continue
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	121 }
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	122
2633 eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	123 code, err := models.IsrsFromString(string(*dr.RisidxCode))
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	124 if err != nil {
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	125 feedback.Warn("invalid ISRS code %v", err)
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	126 ignored++
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	127 continue
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	128 }
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	129
2633 eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	130 if dr.Lat == nil \|\| dr.Lon == nil {
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	131 feedback.Warn("missing lat/lon: %s", code)
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	132 ignored++
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	133 continue
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	134 }
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	135
2633 eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	136 if dr.Relenc == nil {
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	137 feedback.Warn("missing relenc: %s", code)
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	138 ignored++
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	139 continue
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	140 }
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	141
2633 eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	142 if _, err := insertStmt.ExecContext(
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	143 ctx,
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	144 code.CountryCode,
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	145 code.LoCode,
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	146 code.FairwaySection,
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	147 code.Orc,
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	148 code.Hectometre,
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	149 float64(dr.Lon), float64(dr.Lat),
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	150 string(*dr.Relenc),
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	151 ); err != nil {
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	152 return nil, err
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	153 }
eb1d119f253f Fetch data from ERDMS for all allowed countries Tom Gottfried <tom@intevation.de> parents: 2380 diff changeset	154 features++
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	155 }
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	156 }
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	157 feedback.Info("ignored: %d", ignored)
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	158 feedback.Info("features: %d", features)
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	159
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	160 if features == 0 {
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	161 return nil, errors.New("No features found")
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	162 }
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	163
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	164 if err = tx.Commit(); err == nil {
1994 a7c4005b723f Improve some import log messages Tom Gottfried <tom@intevation.de> parents: 1918 diff changeset	165 feedback.Info("Refreshing distance marks (virtual) successfully took %s.",
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	166 time.Since(start))
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	167 } else {
1994 a7c4005b723f Improve some import log messages Tom Gottfried <tom@intevation.de> parents: 1918 diff changeset	168 feedback.Error("Refreshing distance marks (virtual) failed after %s.",
1859 9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	169 time.Since(start))
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	170 }
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	171
9780cb4ef6a6 Distance marks virtual import: Implemented. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1855 diff changeset	172 return nil, nil
1855 bbd653a43a6a Distance marks virtual import: Added stub. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	173 }

Mercurial > gemma

annotate pkg/imports/dmv.go @ 3163:d9903cb34842