gemma: pkg/imports/gm.go annotate

annotate pkg/imports/gm.go @ 2234:9b2f8e94671e

Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop.

author	Sascha L. Teichmann <sascha.teichmann@intevation.de>
date	Wed, 13 Feb 2019 12:45:48 +0100
parents	137addc77b1b
children	4ca41516115b

rev	line source
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	3 //
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	6 //
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	7 // Copyright (C) 2018 by via donau
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	10 //
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	11 // Author(s):
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	12 // * Raimund Renkert <raimund.renkert@intevation.de>
1694 4a2fad8f57de Imports: Resolved golint issues unrelated to exported symbols commenting. Sascha L. Teichmann <teichmann@intevation.de> parents: 1688 diff changeset	13
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	14 package imports
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	15
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	16 import (
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	17 "context"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	18 "database/sql"
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	19 "fmt"
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	20 "strings"
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	21 "time"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	22
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	23 "gemma.intevation.de/gemma/pkg/models"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	24 "gemma.intevation.de/gemma/pkg/soap/nts"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	25 )
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	26
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	27 // GaugeMeasurement is an import job to import
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	28 // gauges measurement data from a NtS SOAP service.
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	29 type GaugeMeasurement struct {
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	30 // URL is the URL of the SOAP service.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	31 URL string `json:"url"`
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	32 // Insecure indicates if HTTPS traffic
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	33 // should validate certificates or not.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	34 Insecure bool `json:"insecure"`
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	35 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	36
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	37 // gauge holds information about a gauge and the latest measurement
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	38 type gauge struct {
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	39 location models.Isrs
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	40 latest time.Time
2018 8a62ce2a5c70 Moved gaugemeasurement to local package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1981 diff changeset	41 }
8a62ce2a5c70 Moved gaugemeasurement to local package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1981 diff changeset	42
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	43 // GMJobKind is the import queue type identifier.
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	44 const GMJobKind JobKind = "gm"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	45
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	46 const (
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	47 listGaugesSQL = `
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	48 SELECT
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	49 (location).country_code,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	50 (location).locode,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	51 (location).fairway_section,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	52 (location).orc,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	53 (location).hectometre
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	54 FROM waterway.gauges
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	55 WHERE (location).country_code = users.current_user_country()`
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	56
1978 0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	57 // TODO: Currently this statement updates existing data sets. In case we want
0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	58 // 'historization' we need to develop an other mechanism to keep existing
0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	59 // data.
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	60 insertGMSQL = `
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	61 INSERT INTO waterway.gauge_measurements (
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	62 fk_gauge_id,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	63 measure_date,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	64 sender,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	65 language_code,
1766 e51db2a365a2 Insert country code in gm import. Raimund Renkert <raimund.renkert@intevation.de> parents: 1761 diff changeset	66 country_code,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	67 date_issue,
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	68 reference_code,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	69 water_level,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	70 predicted,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	71 is_waterlevel,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	72 value_min,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	73 value_max,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	74 date_info,
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1776 diff changeset	75 source_organization,
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1776 diff changeset	76 staging_done
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	77 ) VALUES(
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	78 ($1, $2, $3, $4, $5),
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	79 $6,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	80 $7,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	81 $8,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	82 $9,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	83 $10,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	84 $11,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	85 $12,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	86 $13,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	87 $14,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	88 $15,
1766 e51db2a365a2 Insert country code in gm import. Raimund Renkert <raimund.renkert@intevation.de> parents: 1761 diff changeset	89 $16,
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	90 $17,
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1776 diff changeset	91 $18,
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1776 diff changeset	92 $19
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	93 )
1978 0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	94 ON CONFLICT ON CONSTRAINT gauge_measurements_fk_gauge_id_measure_date_staging_done_key
0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	95 DO UPDATE SET
0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	96 country_code = EXCLUDED.country_code,
0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	97 sender = EXCLUDED.sender,
0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	98 language_code = EXCLUDED.language_code,
0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	99 date_issue = EXCLUDED.date_issue,
0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	100 reference_code= EXCLUDED.reference_code,
0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	101 water_level = EXCLUDED.water_level,
0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	102 predicted = EXCLUDED.predicted,
0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	103 is_waterlevel = EXCLUDED.is_waterlevel,
0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	104 value_min = EXCLUDED.value_min,
0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	105 value_max = EXCLUDED.value_max,
0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	106 date_info = EXCLUDED.date_info,
0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	107 source_organization = EXCLUDED.source_organization
0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	108 RETURNING id
0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	109 `
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	110 )
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	111
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	112 type gmJobCreator struct{}
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	113
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	114 func init() {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	115 RegisterJobCreator(GMJobKind, gmJobCreator{})
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	116 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	117
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	118 func (gmJobCreator) Description() string { return "gauge measurements" }
1646 a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1641 diff changeset	119
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	120 func (gmJobCreator) Create() Job { return new(GaugeMeasurement) }
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	121
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	122 func (gmJobCreator) Depends() []string {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	123 return []string{
1664 819f67c31dfb Imports: Stripped schema prefixes from table dependencies. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1646 diff changeset	124 "gauges",
819f67c31dfb Imports: Stripped schema prefixes from table dependencies. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1646 diff changeset	125 "gauge_measurements",
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	126 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	127 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	128
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	129 func (gmJobCreator) AutoAccept() bool { return true }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	130
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	131 // StageDone moves the imported gauge measurement out of the staging area.
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	132 // Currently doing nothing.
1676 4407ecaa2192 Imports: Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1664 diff changeset	133 func (gmJobCreator) StageDone(context.Context, *sql.Tx, int64) error {
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	134 return nil
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	135 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	136
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	137 // CleanUp of a gauge measurement import is a NOP.
1676 4407ecaa2192 Imports: Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1664 diff changeset	138 func (*GaugeMeasurement) CleanUp() error { return nil }
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	139
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	140 func loadGauges(ctx context.Context, tx sql.Tx) ([]gauge, error) {
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	141
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	142 rows, err := tx.QueryContext(ctx, listGaugesSQL)
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	143 if err != nil {
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	144 return nil, err
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	145 }
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	146 defer rows.Close()
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	147
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	148 var gauges []*gauge
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	149
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	150 for rows.Next() {
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	151 var g gauge
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	152 if err = rows.Scan(
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	153 &g.location.CountryCode,
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	154 &g.location.LoCode,
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	155 &g.location.FairwaySection,
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	156 &g.location.Orc,
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	157 &g.location.Hectometre,
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	158 ); err != nil {
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	159 return nil, err
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	160 }
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	161 gauges = append(gauges, &g)
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	162 }
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	163
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	164 if err = rows.Err(); err != nil {
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	165 return nil, err
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	166 }
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	167
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	168 return gauges, nil
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	169 }
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	170
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	171 // Do executes the actual bottleneck import.
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	172 func (gm *GaugeMeasurement) Do(
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	173 ctx context.Context,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	174 importID int64,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	175 conn *sql.Conn,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	176 feedback Feedback,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	177 ) (interface{}, error) {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	178
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	179 start := time.Now()
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	180
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	181 tx, err := conn.BeginTx(ctx, nil)
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	182 if err != nil {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	183 return nil, err
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	184 }
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	185 defer tx.Rollback()
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	186
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	187 // Get available gauges from database for use as filter in SOAP request
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	188 gauges, err := loadGauges(ctx, tx)
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	189 if err != nil {
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	190 return nil, err
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	191 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	192
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	193 // TODO get date_issue for selected gauges
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	194 gids, err := gm.doForGM(ctx, gauges, tx, feedback)
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	195 if err != nil {
1761 0145537d5474 Gauge measument import: Die with an error if something went wrong. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1754 diff changeset	196 feedback.Error("Error processing %d gauges: %v", len(gauges), err)
0145537d5474 Gauge measument import: Die with an error if something went wrong. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1754 diff changeset	197 return nil, err
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	198 }
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	199
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	200 if len(gids) == 0 {
1638 2a35bbbb4d93 No new measurement are not an error. Fixed typos. Raimund Renkert <raimund.renkert@intevation.de> parents: 1637 diff changeset	201 feedback.Info("No new gauge measurements found")
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	202 return nil, UnchangedError("No new gauge measurements found")
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	203 }
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	204
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	205 if err = tx.Commit(); err != nil {
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	206 feedback.Info(
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	207 "Importing gauge measurements failed after %s", time.Since(start))
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	208 return nil, err
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	209 }
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	210 feedback.Info(
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	211 "Importing gauge measurements successfully took %s", time.Since(start))
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	212
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	213 // TODO: needs to be filled more useful.
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	214 summary := struct {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	215 GaugeMeasuremets []string `json:"gaugeMeasurements"`
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	216 }{
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	217 GaugeMeasuremets: gids,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	218 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	219 return &summary, err
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	220 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	221
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	222 // rescale returns a scaling function to bring the unit all to cm.
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	223 func rescale(unit string) (func(float32) float32, error) {
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	224
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	225 var scale float32
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	226
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	227 switch strings.ToLower(unit) {
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	228 case "mm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	229 scale = 0.1
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	230 case "cm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	231 scale = 1.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	232 case "dm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	233 scale = 10.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	234 case "m":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	235 scale = 100.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	236 case "hm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	237 scale = 10000.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	238 case "km":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	239 scale = 100000.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	240 default:
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	241 return nil, fmt.Errorf("unknown unit '%s'", unit)
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	242 }
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	243
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	244 fn := func(x float32) float32 { return scale * x }
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	245 return fn, nil
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	246 }
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	247
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	248 func (gm *GaugeMeasurement) doForGM(
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	249 ctx context.Context,
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	250 gauges []*gauge,
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	251 tx *sql.Tx,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	252 feedback Feedback,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	253 ) ([]string, error) {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	254
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	255 client := nts.NewINtSMessageService(gm.URL, gm.Insecure, nil)
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	256
1981 e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	257 mt := nts.Message_type_typeWRM
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	258
2234 9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	259 insertStmt, err := tx.PrepareContext(ctx, insertGMSQL)
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	260 if err != nil {
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	261 return nil, err
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	262 }
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	263 defer insertStmt.Close()
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	264
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	265 var gids []string
1981 e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	266 for _, g := range gauges {
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	267
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	268 var idPair []*nts.Id_pair
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	269 isrs := g.location.String()
1981 e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	270 isrsID := nts.Isrs_code_type(isrs)
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	271 idPair = append(idPair, &nts.Id_pair{
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	272 Id: &isrsID,
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	273 })
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	274
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	275 req := &nts.Get_messages_query{
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	276 Message_type: &mt,
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	277 Ids: idPair,
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	278 }
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	279
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	280 resp, err := client.Get_messages(req)
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	281 if err != nil {
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	282 feedback.Error("%v", err)
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	283 return nil, err
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	284 }
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	285
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	286 if resp.Result_message == nil {
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	287 for _, e := range resp.Result_error {
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	288 if e != nil {
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	289 feedback.Error("No gauge measurements found for %s", g.location.String())
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	290 } else {
1981 e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	291 feedback.Error("unknown")
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	292 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	293 }
1981 e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	294 continue
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	295 }
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	296 result := resp.Result_message
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	297
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	298 for _, msg := range result {
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	299 var gid int64
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	300 feedback.Info("Found measurements for %s", g.location.String())
1981 e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	301 for _, wrm := range msg.Wrm {
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	302 currIsrs, err := models.IsrsFromString(string(*wrm.Geo_object.Id))
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	303 if err != nil {
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	304 feedback.Warn("Invalid ISRS code %v", err)
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	305 continue
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	306 }
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	307 var referenceCode string
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	308 if wrm.Reference_code == nil {
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	309 feedback.Info("'Reference_code' not specified. Assuming 'ZPG'")
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	310 referenceCode = "ZPG"
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	311 } else {
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	312 referenceCode = string(*wrm.Reference_code)
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	313 }
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	314 for _, measure := range wrm.Measure {
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	315 var unit string
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	316 if measure.Unit == nil {
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	317 feedback.Info("'Unit' not specified. Assuming 'cm'")
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	318 unit = "cm"
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	319 } else {
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	320 unit = string(*measure.Unit)
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	321 }
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	322 convert, err := rescale(unit)
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	323 if err != nil {
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	324 return nil, err
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	325 }
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	326 isWaterlevel := *measure.Measure_code == nts.Measure_code_enumWAL
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	327 err = insertStmt.QueryRowContext(
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	328 ctx,
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	329 currIsrs.CountryCode,
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	330 currIsrs.LoCode,
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	331 currIsrs.FairwaySection,
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	332 currIsrs.Orc,
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	333 currIsrs.Hectometre,
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	334 measure.Measuredate,
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	335 msg.Identification.From,
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	336 msg.Identification.Language_code,
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	337 msg.Identification.Country_code,
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	338 msg.Identification.Date_issue,
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	339 referenceCode,
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	340 convert(measure.Value),
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	341 measure.Predicted,
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	342 isWaterlevel,
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	343 convert(measure.Value_min),
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	344 convert(measure.Value_max),
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	345 msg.Identification.Date_issue,
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	346 msg.Identification.Originator,
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	347 true, // staging_done
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	348 ).Scan(&gid)
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	349 if err != nil {
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	350 return nil, err
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	351 }
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	352 }
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	353 feedback.Info("Inserted %d measurements for %s",
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	354 len(wrm.Measure), currIsrs)
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	355 gids = append(gids, currIsrs.String())
e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	356 }
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	357 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	358 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	359 return gids, nil
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	360 }

Mercurial > gemma

annotate pkg/imports/gm.go @ 2234:9b2f8e94671e