gemma: pkg/imports/gm.go annotate

annotate pkg/imports/gm.go @ 4064:3eaa3ee4630f

Reverted changes to AGM diff client code.

author	Sascha L. Teichmann <sascha.teichmann@intevation.de>
date	Thu, 25 Jul 2019 11:48:57 +0200
parents	fe3dd65c0891
children	4332b9e26e2f 5867dcf8e93c

rev	line source
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	3 //
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	6 //
3176 1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import Tom Gottfried <tom@intevation.de> parents: 2849 diff changeset	7 // Copyright (C) 2018, 2019 by via donau
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	10 //
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	11 // Author(s):
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	12 // * Raimund Renkert <raimund.renkert@intevation.de>
2245 7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2244 diff changeset	13 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
3176 1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import Tom Gottfried <tom@intevation.de> parents: 2849 diff changeset	14 // * Tom Gottfried <tom.gottfried@intevation.de>
1694 4a2fad8f57de Imports: Resolved golint issues unrelated to exported symbols commenting. Sascha L. Teichmann <teichmann@intevation.de> parents: 1688 diff changeset	15
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	16 package imports
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	17
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	18 import (
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	19 "context"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	20 "database/sql"
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	21 "fmt"
3582 317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	22 "log"
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	23 "sort"
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	24 "strings"
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	25 "time"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	26
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	27 "gemma.intevation.de/gemma/pkg/models"
4062 6c760abcff0e Move handling of PostgreSQL errors to own package Tom Gottfried <tom@intevation.de> parents: 4058 diff changeset	28 "gemma.intevation.de/gemma/pkg/pgxutils"
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	29 "gemma.intevation.de/gemma/pkg/soap/nts"
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	30 "github.com/jackc/pgx/pgtype"
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	31 )
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	32
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	33 // GaugeMeasurement is an import job to import
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	34 // gauges measurement data from a NtS SOAP service.
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	35 type GaugeMeasurement struct {
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	36 // URL is the URL of the SOAP service.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	37 URL string `json:"url"`
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	38 // Insecure indicates if HTTPS traffic
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	39 // should validate certificates or not.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	40 Insecure bool `json:"insecure"`
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	41 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	42
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	43 // GMJobKind is the import queue type identifier.
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	44 const GMJobKind JobKind = "gm"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	45
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	46 const (
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	47 listGaugesSQL = `
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	48 SELECT
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	49 (location).country_code,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	50 (location).locode,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	51 (location).fairway_section,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	52 (location).orc,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	53 (location).hectometre
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	54 FROM waterway.gauges
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	55 WHERE (location).country_code = users.current_user_country()
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	56 OR pg_has_role('sys_admin', 'MEMBER')
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	57 `
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	58
2849 b8972e4671fa Preserve old predictions on raw gauge measurement import. Sascha Wilde <wilde@intevation.de> parents: 2245 diff changeset	59 // Note: we do not expect corrections of data through this service. So
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	60 // any constraint conflicts are triggered by redundant data which
2849 b8972e4671fa Preserve old predictions on raw gauge measurement import. Sascha Wilde <wilde@intevation.de> parents: 2245 diff changeset	61 // can be dropped.
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	62 insertGMSQL = `
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	63 INSERT INTO waterway.gauge_measurements (
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	64 location,
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	65 validity,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	66 measure_date,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	67 sender,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	68 language_code,
1766 e51db2a365a2 Insert country code in gm import. Raimund Renkert <raimund.renkert@intevation.de> parents: 1761 diff changeset	69 country_code,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	70 date_issue,
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	71 reference_code,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	72 water_level,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	73 date_info,
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1776 diff changeset	74 source_organization,
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1776 diff changeset	75 staging_done
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	76 ) VALUES (
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	77 ($1, $2, $3, $4, $5),
3533 8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	78 COALESCE(
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	79 (SELECT validity FROM waterway.gauges
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	80 WHERE location
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	81 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	82 AND validity @> CAST($6 AS timestamp with time zone)),
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	83 tstzrange(NULL, NULL)),
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	84 $6,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	85 $7,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	86 $8,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	87 $9,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	88 $10,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	89 $11,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	90 $12,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	91 $13,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	92 $14,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	93 true
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	94 )
2849 b8972e4671fa Preserve old predictions on raw gauge measurement import. Sascha Wilde <wilde@intevation.de> parents: 2245 diff changeset	95 ON CONFLICT DO NOTHING
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	96 RETURNING 1
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	97 `
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	98
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	99 insertGPSQL = `
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	100 INSERT INTO waterway.gauge_predictions (
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	101 location,
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	102 validity,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	103 measure_date,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	104 sender,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	105 language_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	106 country_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	107 date_issue,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	108 reference_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	109 water_level,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	110 conf_interval,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	111 date_info,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	112 source_organization
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	113 ) VALUES(
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	114 ($1, $2, $3, $4, $5),
3533 8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	115 COALESCE(
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	116 (SELECT validity FROM waterway.gauges
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	117 WHERE location
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	118 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	119 AND validity @> CAST($6 AS timestamp with time zone)),
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	120 tstzrange(NULL, NULL)),
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	121 $6,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	122 $7,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	123 $8,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	124 $9,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	125 $10,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	126 $11,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	127 $12,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	128 $13,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	129 $14,
3278 831193935739 Avoid storing of NtS messages other than water level Tom Gottfried <tom@intevation.de> parents: 3277 diff changeset	130 $15
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	131 )
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	132 ON CONFLICT DO NOTHING
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	133 RETURNING 1
1978 0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	134 `
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	135 )
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	136
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	137 type gmJobCreator struct{}
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	138
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	139 func init() {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	140 RegisterJobCreator(GMJobKind, gmJobCreator{})
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	141 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	142
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	143 func (gmJobCreator) Description() string { return "gauge measurements" }
1646 a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1641 diff changeset	144
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	145 func (gmJobCreator) Create() Job { return new(GaugeMeasurement) }
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	146
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3176 diff changeset	147 func (gmJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3176 diff changeset	148 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3176 diff changeset	149 {"gauge_measurements"},
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3176 diff changeset	150 {"gauges"},
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	151 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	152 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	153
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	154 func (gmJobCreator) AutoAccept() bool { return true }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	155
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	156 // StageDone moves the imported gauge measurement out of the staging area.
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	157 // Currently doing nothing.
1676 4407ecaa2192 Imports: Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1664 diff changeset	158 func (gmJobCreator) StageDone(context.Context, *sql.Tx, int64) error {
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	159 return nil
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	160 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	161
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	162 // CleanUp of a gauge measurement import is a NOP.
1676 4407ecaa2192 Imports: Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1664 diff changeset	163 func (*GaugeMeasurement) CleanUp() error { return nil }
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	164
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	165 // Do executes the actual bottleneck import.
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	166 func (gm *GaugeMeasurement) Do(
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	167 ctx context.Context,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	168 importID int64,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	169 conn *sql.Conn,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	170 feedback Feedback,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	171 ) (interface{}, error) {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	172
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	173 fetch := func() ([]*nts.RIS_Message_Type, error) {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	174 client := nts.NewINtSMessageService(gm.URL, gm.Insecure, nil)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	175
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	176 mt := nts.Message_type_typeWRM
3709 d4a8d084eab4 import: Use dates_issue parameter in gm imports. Raimund Renkert <raimund.renkert@intevation.de> parents: 3583 diff changeset	177 var dis []*nts.Date_pair
d4a8d084eab4 import: Use dates_issue parameter in gm imports. Raimund Renkert <raimund.renkert@intevation.de> parents: 3583 diff changeset	178 dis = append(dis, &nts.Date_pair{
d4a8d084eab4 import: Use dates_issue parameter in gm imports. Raimund Renkert <raimund.renkert@intevation.de> parents: 3583 diff changeset	179 Date_start: nts.Date{time.Now().Add(time.Duration(-24) * time.Hour)},
d4a8d084eab4 import: Use dates_issue parameter in gm imports. Raimund Renkert <raimund.renkert@intevation.de> parents: 3583 diff changeset	180 Date_end: nts.Date{time.Now()},
d4a8d084eab4 import: Use dates_issue parameter in gm imports. Raimund Renkert <raimund.renkert@intevation.de> parents: 3583 diff changeset	181 })
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	182
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	183 req := &nts.Get_messages_query{
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	184 Message_type: &mt,
3709 d4a8d084eab4 import: Use dates_issue parameter in gm imports. Raimund Renkert <raimund.renkert@intevation.de> parents: 3583 diff changeset	185 Dates_issue: dis,
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	186 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	187
3582 317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	188 const maxTries = 3
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	189
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	190 tries := 0
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	191
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	192 again:
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	193 resp, err := client.Get_messages(req)
3582 317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	194
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	195 if err != nil {
3582 317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	196 if t, ok := err.(interface{ Timeout() bool }); ok && t.Timeout() && tries < maxTries {
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	197 log.Println("warn: NtS SOAP request timed out. Trying again.")
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	198 tries++
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	199 goto again
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	200 }
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	201 return nil, fmt.Errorf(
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	202 "Error requesting NtS service: %v", err)
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	203 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	204
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	205 result := resp.Result_message
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	206 if result == nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	207 for _, e := range resp.Result_error {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	208 if e != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	209 feedback.Error("Error code: %s", *e)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	210 } else {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	211 feedback.Error("Unknown error")
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	212 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	213 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	214 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	215 return result, nil
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	216 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	217
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	218 return storeGaugeMeasurements(
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	219 ctx,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	220 importID,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	221 fetch,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	222 conn,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	223 feedback,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	224 )
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	225 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	226
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	227 func loadGauges(ctx context.Context, conn *sql.Conn) ([]string, error) {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	228
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	229 rows, err := conn.QueryContext(ctx, listGaugesSQL)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	230 if err != nil {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	231 return nil, err
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	232 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	233 defer rows.Close()
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	234
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	235 var gauges []string
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	236
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	237 for rows.Next() {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	238 var g models.Isrs
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	239 if err = rows.Scan(
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	240 &g.CountryCode,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	241 &g.LoCode,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	242 &g.FairwaySection,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	243 &g.Orc,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	244 &g.Hectometre,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	245 ); err != nil {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	246 return nil, err
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	247 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	248 gauges = append(gauges, g.String())
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	249 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	250
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	251 if err = rows.Err(); err != nil {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	252 return nil, err
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	253 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	254
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	255 if len(gauges) == 0 {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	256 return nil, UnchangedError(
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	257 "No gauges for which measurements can be imported in database")
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	258 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	259
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	260 sort.Strings(gauges)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	261
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	262 return gauges, nil
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	263 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	264
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	265 func storeGaugeMeasurements(
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	266 ctx context.Context,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	267 importID int64,
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	268 fetch func() ([]*nts.RIS_Message_Type, error),
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	269 conn *sql.Conn,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	270 feedback Feedback,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	271 ) (interface{}, error) {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	272
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	273 start := time.Now()
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	274
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	275 // Get gauges from database, for which user is allowed to import data
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	276 gauges, err := loadGauges(ctx, conn)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	277 if err != nil {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	278 return nil, err
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	279 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	280
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	281 gids, err := doForGM(ctx, gauges, fetch, conn, feedback)
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	282 if err != nil {
1761 0145537d5474 Gauge measument import: Die with an error if something went wrong. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1754 diff changeset	283 return nil, err
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	284 }
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	285
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	286 if len(gids) == 0 {
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	287 return nil, UnchangedError("No new gauge measurements found")
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	288 }
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	289
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	290 feedback.Info(
3176 1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import Tom Gottfried <tom@intevation.de> parents: 2849 diff changeset	291 "Importing gauge measurements took %s", time.Since(start))
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	292
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	293 // TODO: needs to be filled more useful.
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	294 summary := struct {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	295 GaugeMeasuremets []string `json:"gaugeMeasurements"`
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	296 }{
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	297 GaugeMeasuremets: gids,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	298 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	299 return &summary, err
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	300 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	301
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	302 // rescale returns a scaling function to bring the unit all to cm.
3264 9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	303 func rescale(unit string) (func(*float32), error) {
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	304
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	305 var scale float32
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	306
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	307 switch strings.ToLower(unit) {
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	308 case "mm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	309 scale = 0.1
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	310 case "cm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	311 scale = 1.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	312 case "dm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	313 scale = 10.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	314 case "m":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	315 scale = 100.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	316 case "hm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	317 scale = 10000.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	318 case "km":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	319 scale = 100000.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	320 default:
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	321 return nil, fmt.Errorf("unknown unit '%s'", unit)
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	322 }
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	323
3264 9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	324 fn := func(x *float32) {
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	325 if x != nil {
3266 3dee5cf16a58 Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3264 diff changeset	326 x = scale
3264 9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	327 }
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	328 }
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	329 return fn, nil
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	330 }
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	331
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	332 func doForGM(
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	333 ctx context.Context,
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	334 gauges []string,
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	335 fetch func() ([]*nts.RIS_Message_Type, error),
3176 1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import Tom Gottfried <tom@intevation.de> parents: 2849 diff changeset	336 conn *sql.Conn,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	337 feedback Feedback,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	338 ) ([]string, error) {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	339
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	340 insertGPStmt, err := conn.PrepareContext(ctx, insertGPSQL)
2234 9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	341 if err != nil {
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	342 return nil, err
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	343 }
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	344 defer insertGPStmt.Close()
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	345
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	346 insertGMStmt, err := conn.PrepareContext(ctx, insertGMSQL)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	347 if err != nil {
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	348 return nil, err
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	349 }
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	350 defer insertGMStmt.Close()
2234 9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	351
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	352 // lookup to see if data can be imported for gauge
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	353 isKnown := func(s string) bool {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	354 idx := sort.SearchStrings(gauges, s)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	355 return idx < len(gauges) && gauges[idx] == s
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	356 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	357
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	358 result, err := fetch()
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	359 if err != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	360 return nil, err
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	361 }
1981 e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	362
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	363 var gids []string
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	364 for _, msg := range result {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	365 for _, wrm := range msg.Wrm {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	366 curr := string(*wrm.Geo_object.Id)
3718 7431c6680b90 import: Trim whitespaces in ISRS code and improved log messsage. Raimund Renkert <raimund.renkert@intevation.de> parents: 3709 diff changeset	367 curr = strings.TrimSpace(curr)
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	368 currIsrs, err := models.IsrsFromString(curr)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	369 if err != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	370 feedback.Warn("Invalid ISRS code %v", err)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	371 continue
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	372 }
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	373 feedback.Info("Found measurements/predictions for %s", curr)
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	374 if !isKnown(curr) {
3718 7431c6680b90 import: Trim whitespaces in ISRS code and improved log messsage. Raimund Renkert <raimund.renkert@intevation.de> parents: 3709 diff changeset	375 feedback.Warn("Cannot find gauge %q for import", curr)
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	376 continue
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	377 }
1981 e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	378
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	379 var referenceCode string
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	380 if wrm.Reference_code == nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	381 feedback.Info("'Reference_code' not specified. Assuming 'ZPG'")
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	382 referenceCode = "ZPG"
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	383 } else {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	384 referenceCode = string(*wrm.Reference_code)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	385 }
2849 b8972e4671fa Preserve old predictions on raw gauge measurement import. Sascha Wilde <wilde@intevation.de> parents: 2245 diff changeset	386
3847 d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	387 badValue := 0
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	388 newM, newP := 0, 0
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	389 for _, measure := range wrm.Measure {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	390 var unit string
3723 9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	391 if *measure.Measure_code != nts.Measure_code_enumWAL {
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	392 feedback.Warn("Ignored message with measure_code %s",
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	393 *measure.Measure_code)
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	394 continue
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	395 }
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	396 if measure.Unit == nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	397 feedback.Info("'Unit' not specified. Assuming 'cm'")
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	398 unit = "cm"
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	399 } else {
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	400 unit = string(*measure.Unit)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	401 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	402 convert, err := rescale(unit)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	403 if err != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	404 return nil, err
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	405 }
3264 9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	406 convert(measure.Value)
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	407 convert(measure.Value_min)
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	408 convert(measure.Value_max)
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	409
3847 d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	410 // -99999 is used by some gauges to signal an error
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	411 if *measure.Value == -99999 {
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	412 badValue++
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	413 continue
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	414 }
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	415
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	416 var dummy int
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	417 if measure.Predicted {
3723 9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	418 confInterval := pgtype.Numrange{
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	419 Lower: pgtype.Numeric{Status: pgtype.Null},
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	420 Upper: pgtype.Numeric{Status: pgtype.Null},
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	421 LowerType: pgtype.Inclusive,
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	422 UpperType: pgtype.Inclusive,
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	423 Status: pgtype.Null,
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	424 }
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	425 if measure.Value_min != nil && measure.Value_max != nil {
3723 9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	426 valueMin := pgtype.Numeric{Status: pgtype.Null}
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	427 valueMax := pgtype.Numeric{Status: pgtype.Null}
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	428 valueMin.Set(measure.Value_min)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	429 valueMax.Set(measure.Value_max)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	430 confInterval = pgtype.Numrange{
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	431 Lower: valueMin,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	432 Upper: valueMax,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	433 LowerType: pgtype.Inclusive,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	434 UpperType: pgtype.Inclusive,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	435 Status: pgtype.Present,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	436 }
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	437 }
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	438 err = insertGPStmt.QueryRowContext(
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	439 ctx,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	440 currIsrs.CountryCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	441 currIsrs.LoCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	442 currIsrs.FairwaySection,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	443 currIsrs.Orc,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	444 currIsrs.Hectometre,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	445 measure.Measuredate.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	446 msg.Identification.From,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	447 msg.Identification.Language_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	448 msg.Identification.Country_code,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	449 msg.Identification.Date_issue.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	450 referenceCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	451 measure.Value,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	452 &confInterval,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	453 msg.Identification.Date_issue.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	454 msg.Identification.Originator,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	455 ).Scan(&dummy)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	456 switch {
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	457 case err == sql.ErrNoRows:
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	458 // thats expected, nothing to do
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	459 case err != nil:
4063 fe3dd65c0891 Rename HandlePGError to HandleError. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4062 diff changeset	460 feedback.Warn(pgxutils.HandleError(err).Error())
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	461 default:
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	462 newP++
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	463 }
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	464 } else {
3284 7ae1892773a2 Avoid trying to insert explicit missing values received from NtS Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	465 if measure.Value == nil {
7ae1892773a2 Avoid trying to insert explicit missing values received from NtS Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	466 feedback.Info("Missing value at %s. Ignored",
7ae1892773a2 Avoid trying to insert explicit missing values received from NtS Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	467 measure.Measuredate.Format(time.RFC3339))
7ae1892773a2 Avoid trying to insert explicit missing values received from NtS Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	468 continue
7ae1892773a2 Avoid trying to insert explicit missing values received from NtS Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	469 }
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	470 err = insertGMStmt.QueryRowContext(
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	471 ctx,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	472 currIsrs.CountryCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	473 currIsrs.LoCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	474 currIsrs.FairwaySection,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	475 currIsrs.Orc,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	476 currIsrs.Hectometre,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	477 measure.Measuredate.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	478 msg.Identification.From,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	479 msg.Identification.Language_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	480 msg.Identification.Country_code,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	481 msg.Identification.Date_issue.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	482 referenceCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	483 measure.Value,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	484 msg.Identification.Date_issue.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	485 msg.Identification.Originator,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	486 ).Scan(&dummy)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	487 switch {
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	488 case err == sql.ErrNoRows:
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	489 // thats expected, nothing to do
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	490 case err != nil:
4063 fe3dd65c0891 Rename HandlePGError to HandleError. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4062 diff changeset	491 feedback.Warn(pgxutils.HandleError(err).Error())
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	492 default:
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	493 newM++
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	494 }
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	495 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	496 }
3847 d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	497 if badValue > 0 {
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	498 feedback.Warn("Ignored %d measurements with value -99999",
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	499 badValue)
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	500 }
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	501 feedback.Info("Inserted %d measurements for %s",
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	502 newM, curr)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	503 feedback.Info("Inserted %d predictions for %s",
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	504 newP, curr)
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	505 gids = append(gids, curr)
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	506 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	507 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	508 return gids, nil
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	509 }

Mercurial > gemma

annotate pkg/imports/gm.go @ 4064:3eaa3ee4630f