gemma: pkg/imports/gm.go annotate

annotate pkg/imports/gm.go @ 3705:7006b92c0334

Handle updates (vs. historized and new versions) separately. We need this distinction as updated data currently can not be reviewed. More precisely: it can not be declined after review, as the old data is updated in place. The current exclusion from the review is a workaround and not meant to be the final solution. Note that there are additional minor problems, like the fact that the updated data is not counted as changed data for the import.

author	Sascha Wilde <wilde@intevation.de>
date	Wed, 19 Jun 2019 17:00:08 +0200
parents	391afcc961a2
children	d4a8d084eab4

rev	line source
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	3 //
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	6 //
3176 1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import Tom Gottfried <tom@intevation.de> parents: 2849 diff changeset	7 // Copyright (C) 2018, 2019 by via donau
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	10 //
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	11 // Author(s):
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	12 // * Raimund Renkert <raimund.renkert@intevation.de>
2245 7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2244 diff changeset	13 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
3176 1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import Tom Gottfried <tom@intevation.de> parents: 2849 diff changeset	14 // * Tom Gottfried <tom.gottfried@intevation.de>
1694 4a2fad8f57de Imports: Resolved golint issues unrelated to exported symbols commenting. Sascha L. Teichmann <teichmann@intevation.de> parents: 1688 diff changeset	15
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	16 package imports
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	17
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	18 import (
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	19 "context"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	20 "database/sql"
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	21 "fmt"
3582 317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	22 "log"
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	23 "sort"
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	24 "strings"
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	25 "time"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	26
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	27 "gemma.intevation.de/gemma/pkg/models"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	28 "gemma.intevation.de/gemma/pkg/soap/nts"
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	29 "github.com/jackc/pgx/pgtype"
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	30 )
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	31
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	32 // GaugeMeasurement is an import job to import
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	33 // gauges measurement data from a NtS SOAP service.
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	34 type GaugeMeasurement struct {
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	35 // URL is the URL of the SOAP service.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	36 URL string `json:"url"`
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	37 // Insecure indicates if HTTPS traffic
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	38 // should validate certificates or not.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	39 Insecure bool `json:"insecure"`
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	40 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	41
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	42 // GMJobKind is the import queue type identifier.
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	43 const GMJobKind JobKind = "gm"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	44
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	45 const (
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	46 listGaugesSQL = `
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	47 SELECT
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	48 (location).country_code,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	49 (location).locode,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	50 (location).fairway_section,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	51 (location).orc,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	52 (location).hectometre
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	53 FROM waterway.gauges
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	54 WHERE (location).country_code = users.current_user_country()
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	55 OR pg_has_role('sys_admin', 'MEMBER')
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	56 `
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	57
2849 b8972e4671fa Preserve old predictions on raw gauge measurement import. Sascha Wilde <wilde@intevation.de> parents: 2245 diff changeset	58 // Note: we do not expect corrections of data through this service. So
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	59 // any constraint conflicts are triggered by redundant data which
2849 b8972e4671fa Preserve old predictions on raw gauge measurement import. Sascha Wilde <wilde@intevation.de> parents: 2245 diff changeset	60 // can be dropped.
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	61 insertGMSQL = `
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	62 INSERT INTO waterway.gauge_measurements (
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	63 location,
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	64 validity,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	65 measure_date,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	66 sender,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	67 language_code,
1766 e51db2a365a2 Insert country code in gm import. Raimund Renkert <raimund.renkert@intevation.de> parents: 1761 diff changeset	68 country_code,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	69 date_issue,
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	70 reference_code,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	71 water_level,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	72 date_info,
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1776 diff changeset	73 source_organization,
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1776 diff changeset	74 staging_done
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	75 ) VALUES (
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	76 ($1, $2, $3, $4, $5),
3533 8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	77 COALESCE(
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	78 (SELECT validity FROM waterway.gauges
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	79 WHERE location
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	80 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	81 AND validity @> CAST($6 AS timestamp with time zone)),
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	82 tstzrange(NULL, NULL)),
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	83 $6,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	84 $7,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	85 $8,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	86 $9,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	87 $10,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	88 $11,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	89 $12,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	90 $13,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	91 $14,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	92 true
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	93 )
2849 b8972e4671fa Preserve old predictions on raw gauge measurement import. Sascha Wilde <wilde@intevation.de> parents: 2245 diff changeset	94 ON CONFLICT DO NOTHING
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	95 RETURNING 1
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	96 `
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	97
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	98 insertGPSQL = `
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	99 INSERT INTO waterway.gauge_predictions (
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	100 location,
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	101 validity,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	102 measure_date,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	103 sender,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	104 language_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	105 country_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	106 date_issue,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	107 reference_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	108 water_level,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	109 conf_interval,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	110 date_info,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	111 source_organization
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	112 ) VALUES(
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	113 ($1, $2, $3, $4, $5),
3533 8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	114 COALESCE(
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	115 (SELECT validity FROM waterway.gauges
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	116 WHERE location
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	117 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	118 AND validity @> CAST($6 AS timestamp with time zone)),
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	119 tstzrange(NULL, NULL)),
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	120 $6,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	121 $7,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	122 $8,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	123 $9,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	124 $10,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	125 $11,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	126 $12,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	127 $13,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	128 $14,
3278 831193935739 Avoid storing of NtS messages other than water level Tom Gottfried <tom@intevation.de> parents: 3277 diff changeset	129 $15
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	130 )
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	131 ON CONFLICT DO NOTHING
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	132 RETURNING 1
1978 0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	133 `
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	134 )
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	135
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	136 type gmJobCreator struct{}
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	137
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	138 func init() {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	139 RegisterJobCreator(GMJobKind, gmJobCreator{})
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	140 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	141
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	142 func (gmJobCreator) Description() string { return "gauge measurements" }
1646 a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1641 diff changeset	143
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	144 func (gmJobCreator) Create() Job { return new(GaugeMeasurement) }
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	145
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3176 diff changeset	146 func (gmJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3176 diff changeset	147 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3176 diff changeset	148 {"gauge_measurements"},
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3176 diff changeset	149 {"gauges"},
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	150 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	151 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	152
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	153 func (gmJobCreator) AutoAccept() bool { return true }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	154
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	155 // StageDone moves the imported gauge measurement out of the staging area.
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	156 // Currently doing nothing.
1676 4407ecaa2192 Imports: Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1664 diff changeset	157 func (gmJobCreator) StageDone(context.Context, *sql.Tx, int64) error {
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	158 return nil
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	159 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	160
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	161 // CleanUp of a gauge measurement import is a NOP.
1676 4407ecaa2192 Imports: Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1664 diff changeset	162 func (*GaugeMeasurement) CleanUp() error { return nil }
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	163
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	164 // Do executes the actual bottleneck import.
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	165 func (gm *GaugeMeasurement) Do(
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	166 ctx context.Context,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	167 importID int64,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	168 conn *sql.Conn,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	169 feedback Feedback,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	170 ) (interface{}, error) {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	171
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	172 fetch := func() ([]*nts.RIS_Message_Type, error) {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	173 client := nts.NewINtSMessageService(gm.URL, gm.Insecure, nil)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	174
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	175 mt := nts.Message_type_typeWRM
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	176
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	177 req := &nts.Get_messages_query{
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	178 Message_type: &mt,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	179 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	180
3582 317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	181 const maxTries = 3
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	182
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	183 tries := 0
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	184
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	185 again:
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	186 resp, err := client.Get_messages(req)
3582 317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	187
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	188 if err != nil {
3582 317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	189 if t, ok := err.(interface{ Timeout() bool }); ok && t.Timeout() && tries < maxTries {
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	190 log.Println("warn: NtS SOAP request timed out. Trying again.")
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	191 tries++
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	192 goto again
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	193 }
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	194 return nil, fmt.Errorf(
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	195 "Error requesting NtS service: %v", err)
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	196 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	197
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	198 result := resp.Result_message
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	199 if result == nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	200 for _, e := range resp.Result_error {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	201 if e != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	202 feedback.Error("Error code: %s", *e)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	203 } else {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	204 feedback.Error("Unknown error")
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	205 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	206 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	207 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	208 return result, nil
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	209 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	210
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	211 return storeGaugeMeasurements(
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	212 ctx,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	213 importID,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	214 fetch,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	215 conn,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	216 feedback,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	217 )
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	218 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	219
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	220 func loadGauges(ctx context.Context, conn *sql.Conn) ([]string, error) {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	221
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	222 rows, err := conn.QueryContext(ctx, listGaugesSQL)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	223 if err != nil {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	224 return nil, err
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	225 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	226 defer rows.Close()
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	227
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	228 var gauges []string
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	229
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	230 for rows.Next() {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	231 var g models.Isrs
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	232 if err = rows.Scan(
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	233 &g.CountryCode,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	234 &g.LoCode,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	235 &g.FairwaySection,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	236 &g.Orc,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	237 &g.Hectometre,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	238 ); err != nil {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	239 return nil, err
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	240 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	241 gauges = append(gauges, g.String())
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	242 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	243
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	244 if err = rows.Err(); err != nil {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	245 return nil, err
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	246 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	247
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	248 if len(gauges) == 0 {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	249 return nil, UnchangedError(
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	250 "No gauges for which measurements can be imported in database")
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	251 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	252
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	253 sort.Strings(gauges)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	254
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	255 return gauges, nil
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	256 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	257
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	258 func storeGaugeMeasurements(
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	259 ctx context.Context,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	260 importID int64,
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	261 fetch func() ([]*nts.RIS_Message_Type, error),
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	262 conn *sql.Conn,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	263 feedback Feedback,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	264 ) (interface{}, error) {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	265
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	266 start := time.Now()
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	267
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	268 // Get gauges from database, for which user is allowed to import data
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	269 gauges, err := loadGauges(ctx, conn)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	270 if err != nil {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	271 return nil, err
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	272 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	273
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	274 gids, err := doForGM(ctx, gauges, fetch, conn, feedback)
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	275 if err != nil {
1761 0145537d5474 Gauge measument import: Die with an error if something went wrong. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1754 diff changeset	276 return nil, err
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	277 }
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	278
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	279 if len(gids) == 0 {
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	280 return nil, UnchangedError("No new gauge measurements found")
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	281 }
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	282
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	283 feedback.Info(
3176 1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import Tom Gottfried <tom@intevation.de> parents: 2849 diff changeset	284 "Importing gauge measurements took %s", time.Since(start))
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	285
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	286 // TODO: needs to be filled more useful.
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	287 summary := struct {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	288 GaugeMeasuremets []string `json:"gaugeMeasurements"`
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	289 }{
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	290 GaugeMeasuremets: gids,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	291 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	292 return &summary, err
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	293 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	294
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	295 // rescale returns a scaling function to bring the unit all to cm.
3264 9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	296 func rescale(unit string) (func(*float32), error) {
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	297
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	298 var scale float32
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	299
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	300 switch strings.ToLower(unit) {
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	301 case "mm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	302 scale = 0.1
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	303 case "cm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	304 scale = 1.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	305 case "dm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	306 scale = 10.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	307 case "m":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	308 scale = 100.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	309 case "hm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	310 scale = 10000.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	311 case "km":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	312 scale = 100000.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	313 default:
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	314 return nil, fmt.Errorf("unknown unit '%s'", unit)
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	315 }
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	316
3264 9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	317 fn := func(x *float32) {
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	318 if x != nil {
3266 3dee5cf16a58 Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3264 diff changeset	319 x = scale
3264 9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	320 }
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	321 }
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	322 return fn, nil
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	323 }
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	324
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	325 func doForGM(
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	326 ctx context.Context,
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	327 gauges []string,
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	328 fetch func() ([]*nts.RIS_Message_Type, error),
3176 1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import Tom Gottfried <tom@intevation.de> parents: 2849 diff changeset	329 conn *sql.Conn,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	330 feedback Feedback,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	331 ) ([]string, error) {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	332
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	333 insertGPStmt, err := conn.PrepareContext(ctx, insertGPSQL)
2234 9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	334 if err != nil {
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	335 return nil, err
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	336 }
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	337 defer insertGPStmt.Close()
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	338
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	339 insertGMStmt, err := conn.PrepareContext(ctx, insertGMSQL)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	340 if err != nil {
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	341 return nil, err
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	342 }
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	343 defer insertGMStmt.Close()
2234 9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	344
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	345 // lookup to see if data can be imported for gauge
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	346 isKnown := func(s string) bool {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	347 idx := sort.SearchStrings(gauges, s)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	348 return idx < len(gauges) && gauges[idx] == s
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	349 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	350
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	351 result, err := fetch()
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	352 if err != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	353 return nil, err
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	354 }
1981 e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	355
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	356 var gids []string
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	357 for _, msg := range result {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	358 for _, wrm := range msg.Wrm {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	359 curr := string(*wrm.Geo_object.Id)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	360 currIsrs, err := models.IsrsFromString(curr)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	361 if err != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	362 feedback.Warn("Invalid ISRS code %v", err)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	363 continue
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	364 }
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	365 feedback.Info("Found measurements/predictions for %s", curr)
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	366 if !isKnown(curr) {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	367 feedback.Warn("Cannot import data for %s", curr)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	368 continue
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	369 }
1981 e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	370
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	371 var referenceCode string
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	372 if wrm.Reference_code == nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	373 feedback.Info("'Reference_code' not specified. Assuming 'ZPG'")
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	374 referenceCode = "ZPG"
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	375 } else {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	376 referenceCode = string(*wrm.Reference_code)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	377 }
2849 b8972e4671fa Preserve old predictions on raw gauge measurement import. Sascha Wilde <wilde@intevation.de> parents: 2245 diff changeset	378
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	379 newM, newP := 0, 0
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	380 for _, measure := range wrm.Measure {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	381 var unit string
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	382 if measure.Unit == nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	383 feedback.Info("'Unit' not specified. Assuming 'cm'")
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	384 unit = "cm"
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	385 } else {
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	386 unit = string(*measure.Unit)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	387 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	388 convert, err := rescale(unit)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	389 if err != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	390 return nil, err
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	391 }
3264 9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	392 convert(measure.Value)
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	393 convert(measure.Value_min)
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	394 convert(measure.Value_max)
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	395
3278 831193935739 Avoid storing of NtS messages other than water level Tom Gottfried <tom@intevation.de> parents: 3277 diff changeset	396 if *measure.Measure_code != nts.Measure_code_enumWAL {
831193935739 Avoid storing of NtS messages other than water level Tom Gottfried <tom@intevation.de> parents: 3277 diff changeset	397 feedback.Warn("Ignored message with measure_code %s",
831193935739 Avoid storing of NtS messages other than water level Tom Gottfried <tom@intevation.de> parents: 3277 diff changeset	398 *measure.Measure_code)
831193935739 Avoid storing of NtS messages other than water level Tom Gottfried <tom@intevation.de> parents: 3277 diff changeset	399 continue
831193935739 Avoid storing of NtS messages other than water level Tom Gottfried <tom@intevation.de> parents: 3277 diff changeset	400 }
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	401
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	402 var dummy int
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	403 if measure.Predicted {
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	404 var confInterval pgtype.Numrange
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	405 if measure.Value_min != nil && measure.Value_max != nil {
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	406 var valueMin, valueMax pgtype.Numeric
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	407 valueMin.Set(measure.Value_min)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	408 valueMax.Set(measure.Value_max)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	409 confInterval = pgtype.Numrange{
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	410 Lower: valueMin,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	411 Upper: valueMax,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	412 LowerType: pgtype.Inclusive,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	413 UpperType: pgtype.Inclusive,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	414 Status: pgtype.Present,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	415 }
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	416 }
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	417 err = insertGPStmt.QueryRowContext(
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	418 ctx,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	419 currIsrs.CountryCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	420 currIsrs.LoCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	421 currIsrs.FairwaySection,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	422 currIsrs.Orc,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	423 currIsrs.Hectometre,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	424 measure.Measuredate.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	425 msg.Identification.From,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	426 msg.Identification.Language_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	427 msg.Identification.Country_code,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	428 msg.Identification.Date_issue.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	429 referenceCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	430 measure.Value,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	431 &confInterval,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	432 msg.Identification.Date_issue.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	433 msg.Identification.Originator,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	434 ).Scan(&dummy)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	435 switch {
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	436 case err == sql.ErrNoRows:
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	437 // thats expected, nothing to do
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	438 case err != nil:
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	439 feedback.Warn(handleError(err).Error())
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	440 default:
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	441 newP++
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	442 }
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	443 } else {
3284 7ae1892773a2 Avoid trying to insert explicit missing values received from NtS Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	444 if measure.Value == nil {
7ae1892773a2 Avoid trying to insert explicit missing values received from NtS Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	445 feedback.Info("Missing value at %s. Ignored",
7ae1892773a2 Avoid trying to insert explicit missing values received from NtS Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	446 measure.Measuredate.Format(time.RFC3339))
7ae1892773a2 Avoid trying to insert explicit missing values received from NtS Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	447 continue
7ae1892773a2 Avoid trying to insert explicit missing values received from NtS Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	448 }
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	449 err = insertGMStmt.QueryRowContext(
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	450 ctx,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	451 currIsrs.CountryCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	452 currIsrs.LoCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	453 currIsrs.FairwaySection,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	454 currIsrs.Orc,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	455 currIsrs.Hectometre,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	456 measure.Measuredate.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	457 msg.Identification.From,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	458 msg.Identification.Language_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	459 msg.Identification.Country_code,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	460 msg.Identification.Date_issue.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	461 referenceCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	462 measure.Value,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	463 msg.Identification.Date_issue.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	464 msg.Identification.Originator,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	465 ).Scan(&dummy)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	466 switch {
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	467 case err == sql.ErrNoRows:
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	468 // thats expected, nothing to do
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	469 case err != nil:
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	470 feedback.Warn(handleError(err).Error())
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	471 default:
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	472 newM++
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	473 }
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	474 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	475 }
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	476 feedback.Info("Inserted %d measurements for %s",
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	477 newM, curr)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	478 feedback.Info("Inserted %d predictions for %s",
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	479 newP, curr)
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	480 gids = append(gids, curr)
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	481 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	482 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	483 return gids, nil
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	484 }

Mercurial > gemma

annotate pkg/imports/gm.go @ 3705:7006b92c0334