gemma: pkg/imports/gm.go annotate

annotate pkg/imports/gm.go @ 5541:29804c8e817d aggregate-gm-import-logging

WIP: Handle rest correctly.

author	Sascha L. Teichmann <sascha.teichmann@intevation.de>
date	Tue, 26 Oct 2021 02:02:23 +0200
parents	2e5e51288a6c
children	a4fa1bf9b208

rev	line source
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	3 //
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	6 //
3176 1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import Tom Gottfried <tom@intevation.de> parents: 2849 diff changeset	7 // Copyright (C) 2018, 2019 by via donau
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	10 //
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	11 // Author(s):
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	12 // * Raimund Renkert <raimund.renkert@intevation.de>
2245 7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2244 diff changeset	13 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
3176 1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import Tom Gottfried <tom@intevation.de> parents: 2849 diff changeset	14 // * Tom Gottfried <tom.gottfried@intevation.de>
1694 4a2fad8f57de Imports: Resolved golint issues unrelated to exported symbols commenting. Sascha L. Teichmann <teichmann@intevation.de> parents: 1688 diff changeset	15
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	16 package imports
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	17
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	18 import (
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	19 "context"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	20 "database/sql"
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	21 "fmt"
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	22 "sort"
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	23 "strings"
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	24 "time"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	25
5490 5f47eeea988d Use own logging package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5035 diff changeset	26 "github.com/jackc/pgx/pgtype"
5f47eeea988d Use own logging package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5035 diff changeset	27
5f47eeea988d Use own logging package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5035 diff changeset	28 "gemma.intevation.de/gemma/pkg/log"
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	29 "gemma.intevation.de/gemma/pkg/models"
4062 6c760abcff0e Move handling of PostgreSQL errors to own package Tom Gottfried <tom@intevation.de> parents: 4058 diff changeset	30 "gemma.intevation.de/gemma/pkg/pgxutils"
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	31 "gemma.intevation.de/gemma/pkg/soap/nts"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	32 )
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	33
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	34 // GaugeMeasurement is an import job to import
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	35 // gauges measurement data from a NtS SOAP service.
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	36 type GaugeMeasurement struct {
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	37 // URL is the URL of the SOAP service.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	38 URL string `json:"url"`
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	39 // Insecure indicates if HTTPS traffic
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	40 // should validate certificates or not.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	41 Insecure bool `json:"insecure"`
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	42 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	43
4852 046a07a33b19 Fixed the golint issues of the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4848 diff changeset	44 // Description gives a short info about relevant facts of this import.
4798 ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4505 diff changeset	45 func (gm *GaugeMeasurement) Description() (string, error) {
4799 f32d086b5dbf Removed the mechanical touch of the last commit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4798 diff changeset	46 return gm.URL, nil
4798 ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4505 diff changeset	47 }
ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4505 diff changeset	48
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	49 // GMJobKind is the import queue type identifier.
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	50 const GMJobKind JobKind = "gm"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	51
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	52 const (
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	53 listGaugesSQL = `
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	54 SELECT
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	55 (location).country_code,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	56 (location).locode,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	57 (location).fairway_section,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	58 (location).orc,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	59 (location).hectometre
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	60 FROM waterway.gauges
4158 5466562cca60 Remove utility function with possibly bad performance impact Tom Gottfried <tom@intevation.de> parents: 4147 diff changeset	61 WHERE (location).country_code = (
5466562cca60 Remove utility function with possibly bad performance impact Tom Gottfried <tom@intevation.de> parents: 4147 diff changeset	62 SELECT country FROM users.list_users WHERE username = current_user)
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	63 OR pg_has_role('sys_admin', 'MEMBER')
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	64 `
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	65
2849 b8972e4671fa Preserve old predictions on raw gauge measurement import. Sascha Wilde <wilde@intevation.de> parents: 2245 diff changeset	66 // Note: we do not expect corrections of data through this service. So
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	67 // any constraint conflicts are triggered by redundant data which
2849 b8972e4671fa Preserve old predictions on raw gauge measurement import. Sascha Wilde <wilde@intevation.de> parents: 2245 diff changeset	68 // can be dropped.
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	69 insertGMSQL = `
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	70 INSERT INTO waterway.gauge_measurements (
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	71 location,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	72 measure_date,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	73 sender,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	74 language_code,
1766 e51db2a365a2 Insert country code in gm import. Raimund Renkert <raimund.renkert@intevation.de> parents: 1761 diff changeset	75 country_code,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	76 date_issue,
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	77 reference_code,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	78 water_level,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	79 date_info,
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1776 diff changeset	80 source_organization,
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1776 diff changeset	81 staging_done
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	82 ) VALUES (
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	83 ($1, $2, $3, $4, $5),
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	84 $6,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	85 $7,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	86 $8,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	87 $9,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	88 $10,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	89 $11,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	90 $12,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	91 $13,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	92 $14,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	93 true
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	94 )
2849 b8972e4671fa Preserve old predictions on raw gauge measurement import. Sascha Wilde <wilde@intevation.de> parents: 2245 diff changeset	95 ON CONFLICT DO NOTHING
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	96 RETURNING 1
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	97 `
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	98
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	99 insertGPSQL = `
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	100 INSERT INTO waterway.gauge_predictions (
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	101 location,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	102 measure_date,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	103 sender,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	104 language_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	105 country_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	106 date_issue,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	107 reference_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	108 water_level,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	109 conf_interval,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	110 date_info,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	111 source_organization
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	112 ) VALUES(
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	113 ($1, $2, $3, $4, $5),
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	114 $6,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	115 $7,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	116 $8,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	117 $9,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	118 $10,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	119 $11,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	120 $12,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	121 $13,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	122 $14,
3278 831193935739 Avoid storing of NtS messages other than water level Tom Gottfried <tom@intevation.de> parents: 3277 diff changeset	123 $15
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	124 )
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	125 ON CONFLICT DO NOTHING
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	126 RETURNING 1
1978 0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	127 `
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	128 )
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	129
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	130 type gmJobCreator struct{}
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	131
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	132 func init() {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	133 RegisterJobCreator(GMJobKind, gmJobCreator{})
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	134 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	135
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	136 func (gmJobCreator) Description() string { return "gauge measurements" }
1646 a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1641 diff changeset	137
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	138 func (gmJobCreator) Create() Job { return new(GaugeMeasurement) }
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	139
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3176 diff changeset	140 func (gmJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3176 diff changeset	141 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3176 diff changeset	142 {"gauge_measurements"},
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3176 diff changeset	143 {"gauges"},
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	144 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	145 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	146
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	147 func (gmJobCreator) AutoAccept() bool { return true }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	148
5035 56c589f7435d Enhance comments on implementations of StageDone() Tom Gottfried <tom@intevation.de> parents: 5034 diff changeset	149 // StageDone is a NOP for gauge measurements imports.
5034 59a99655f34d Added feedback support for StageDone. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4852 diff changeset	150 func (gmJobCreator) StageDone(context.Context, *sql.Tx, int64, Feedback) error {
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	151 return nil
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	152 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	153
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	154 // CleanUp of a gauge measurement import is a NOP.
1676 4407ecaa2192 Imports: Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1664 diff changeset	155 func (*GaugeMeasurement) CleanUp() error { return nil }
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	156
4845 8c46b845b406 Fix comment Tom Gottfried <tom@intevation.de> parents: 4813 diff changeset	157 // Do executes the actual import.
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	158 func (gm *GaugeMeasurement) Do(
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	159 ctx context.Context,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	160 importID int64,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	161 conn *sql.Conn,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	162 feedback Feedback,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	163 ) (interface{}, error) {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	164
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	165 fetch := func() ([]*nts.RIS_Message_Type, error) {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	166 client := nts.NewINtSMessageService(gm.URL, gm.Insecure, nil)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	167
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	168 mt := nts.Message_type_typeWRM
4504 b7e3d42fd9e7 When fetching gauge measurements look 1day into past and future. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4197 diff changeset	169 now := time.Now()
4505 30c499f996d7 Simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4504 diff changeset	170 dis := []*nts.Date_pair{{
4504 b7e3d42fd9e7 When fetching gauge measurements look 1day into past and future. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4197 diff changeset	171 Date_start: nts.Date{Time: now.AddDate(0, 0, -1)},
b7e3d42fd9e7 When fetching gauge measurements look 1day into past and future. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4197 diff changeset	172 Date_end: nts.Date{Time: now.AddDate(0, 0, +1)},
4505 30c499f996d7 Simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4504 diff changeset	173 }}
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	174
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	175 req := &nts.Get_messages_query{
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	176 Message_type: &mt,
3709 d4a8d084eab4 import: Use dates_issue parameter in gm imports. Raimund Renkert <raimund.renkert@intevation.de> parents: 3583 diff changeset	177 Dates_issue: dis,
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	178 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	179
3582 317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	180 const maxTries = 3
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	181
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	182 tries := 0
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	183
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	184 again:
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	185 resp, err := client.Get_messages(req)
3582 317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	186
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	187 if err != nil {
3582 317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	188 if t, ok := err.(interface{ Timeout() bool }); ok && t.Timeout() && tries < maxTries {
5490 5f47eeea988d Use own logging package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5035 diff changeset	189 log.Warnln("NtS SOAP request timed out. Trying again.")
3582 317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	190 tries++
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	191 goto again
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	192 }
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	193 return nil, fmt.Errorf(
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	194 "Error requesting NtS service: %v", err)
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	195 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	196
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	197 result := resp.Result_message
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	198 if result == nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	199 for _, e := range resp.Result_error {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	200 if e != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	201 feedback.Error("Error code: %s", *e)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	202 } else {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	203 feedback.Error("Unknown error")
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	204 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	205 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	206 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	207 return result, nil
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	208 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	209
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	210 return storeGaugeMeasurements(
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	211 ctx,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	212 importID,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	213 fetch,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	214 conn,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	215 feedback,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	216 )
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	217 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	218
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	219 func loadGauges(ctx context.Context, conn *sql.Conn) ([]string, error) {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	220
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	221 rows, err := conn.QueryContext(ctx, listGaugesSQL)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	222 if err != nil {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	223 return nil, err
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	224 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	225 defer rows.Close()
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	226
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	227 var gauges []string
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	228
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	229 for rows.Next() {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	230 var g models.Isrs
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	231 if err = rows.Scan(
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	232 &g.CountryCode,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	233 &g.LoCode,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	234 &g.FairwaySection,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	235 &g.Orc,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	236 &g.Hectometre,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	237 ); err != nil {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	238 return nil, err
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	239 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	240 gauges = append(gauges, g.String())
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	241 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	242
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	243 if err = rows.Err(); err != nil {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	244 return nil, err
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	245 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	246
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	247 if len(gauges) == 0 {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	248 return nil, UnchangedError(
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	249 "No gauges for which measurements can be imported in database")
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	250 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	251
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	252 sort.Strings(gauges)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	253
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	254 return gauges, nil
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	255 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	256
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	257 func storeGaugeMeasurements(
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	258 ctx context.Context,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	259 importID int64,
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	260 fetch func() ([]*nts.RIS_Message_Type, error),
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	261 conn *sql.Conn,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	262 feedback Feedback,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	263 ) (interface{}, error) {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	264
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	265 start := time.Now()
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	266
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	267 // Get gauges from database, for which user is allowed to import data
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	268 gauges, err := loadGauges(ctx, conn)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	269 if err != nil {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	270 return nil, err
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	271 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	272
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	273 gids, err := doForGM(ctx, gauges, fetch, conn, feedback)
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	274 if err != nil {
1761 0145537d5474 Gauge measument import: Die with an error if something went wrong. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1754 diff changeset	275 return nil, err
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	276 }
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	277
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	278 if len(gids) == 0 {
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	279 return nil, UnchangedError("No new gauge measurements found")
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	280 }
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	281
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	282 feedback.Info(
3176 1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import Tom Gottfried <tom@intevation.de> parents: 2849 diff changeset	283 "Importing gauge measurements took %s", time.Since(start))
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	284
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	285 // TODO: needs to be filled more useful.
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	286 summary := struct {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	287 GaugeMeasuremets []string `json:"gaugeMeasurements"`
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	288 }{
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	289 GaugeMeasuremets: gids,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	290 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	291 return &summary, err
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	292 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	293
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	294 // rescale returns a scaling function to bring the unit all to cm.
3264 9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	295 func rescale(unit string) (func(*float32), error) {
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	296
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	297 var scale float32
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	298
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	299 switch strings.ToLower(unit) {
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	300 case "mm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	301 scale = 0.1
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	302 case "cm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	303 scale = 1.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	304 case "dm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	305 scale = 10.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	306 case "m":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	307 scale = 100.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	308 case "hm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	309 scale = 10000.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	310 case "km":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	311 scale = 100000.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	312 default:
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	313 return nil, fmt.Errorf("unknown unit '%s'", unit)
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	314 }
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	315
3264 9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	316 fn := func(x *float32) {
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	317 if x != nil {
3266 3dee5cf16a58 Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3264 diff changeset	318 x = scale
3264 9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	319 }
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	320 }
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	321 return fn, nil
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	322 }
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	323
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	324 type gmLog struct {
5529 f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	325 gid string
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	326 unknown bool
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	327 assumedZPG bool
5529 f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	328 ignoredMeasureCodes []string
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	329 rescaleErrors []string
5531 84f0c53e6ffe Completed GM logging aggregation. TODO: cleanup and dedup. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5530 diff changeset	330 missingValues []string
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	331 assumedCM int
5530 3ccbb6162112 WIP: More on GM import aggregation. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5529 diff changeset	332 badValues int
5531 84f0c53e6ffe Completed GM logging aggregation. TODO: cleanup and dedup. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5530 diff changeset	333 measurements int
84f0c53e6ffe Completed GM logging aggregation. TODO: cleanup and dedup. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5530 diff changeset	334 predictions int
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	335 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	336
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	337 type gmLogs []*gmLog
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	338
5529 f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	339 func extend(haystack []string, needle string) []string {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	340 for _, straw := range haystack {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	341 if straw == needle {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	342 return haystack
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	343 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	344 }
5529 f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	345 return append(haystack, needle)
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	346 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	347
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	348 func (gl *gmLog) addRescaleError(err error) {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	349 gl.rescaleErrors = extend(gl.rescaleErrors, err.Error())
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	350 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	351
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	352 func (gl *gmLog) ignoreMeasureCode(mc nts.Measure_code_enum) {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	353 gl.ignoredMeasureCodes = extend(gl.ignoredMeasureCodes, string(mc))
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	354 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	355
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	356 func (gls gmLogs) find(gid string) *gmLog {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	357 for _, gl := range gls {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	358 if gl.gid == gid {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	359 return gl
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	360 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	361 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	362 return nil
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	363 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	364
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	365 func (gls gmLogs) logging(feedback Feedback) {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	366
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	367 gls.logBool(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	368 (*gmLog).getUnkown,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	369 "Cannot find following gauges: ",
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	370 feedback.Warn)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	371
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	372 gls.logBool(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	373 (*gmLog).getAssumedZPG,
5537 2e5e51288a6c Fixed typo in generated logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5533 diff changeset	374 "'Reference_code' not specified. Assuming 'ZPG': ",
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	375 feedback.Warn)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	376
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	377 gls.logInt(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	378 (*gmLog).getAssumedCM,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	379 "'Unit' not specified. Assuming 'cm': ",
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	380 feedback.Warn)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	381
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	382 gls.logInt(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	383 (*gmLog).getBadValues,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	384 "Ignored measurements with value -99999: ",
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	385 feedback.Warn)
5531 84f0c53e6ffe Completed GM logging aggregation. TODO: cleanup and dedup. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5530 diff changeset	386
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	387 gls.logString(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	388 (*gmLog).getMissingValues,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	389 "Missing mandatory values: ",
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	390 feedback.Warn)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	391
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	392 gls.logString(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	393 (*gmLog).getRescaleErrors,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	394 "Cannot convert units: ",
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	395 feedback.Error)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	396
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	397 gls.logString(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	398 (*gmLog).getRescaleErrors,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	399 "Ignored measure codes: ",
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	400 feedback.Warn)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	401
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	402 gls.logInt(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	403 (*gmLog).getPredictions,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	404 "New predictions: ",
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	405 feedback.Info)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	406
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	407 gls.logInt(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	408 (*gmLog).getMeasurements,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	409 "New measurements: ",
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	410 feedback.Info)
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	411
0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	412 gls.logBool(
0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	413 (*gmLog).nothingChanged,
0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	414 "No changes for: ",
0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	415 feedback.Info)
5531 84f0c53e6ffe Completed GM logging aggregation. TODO: cleanup and dedup. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5530 diff changeset	416 }
84f0c53e6ffe Completed GM logging aggregation. TODO: cleanup and dedup. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5530 diff changeset	417
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	418 func (gl *gmLog) getAssumedZPG() bool { return gl.assumedZPG }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	419 func (gl *gmLog) getUnkown() bool { return gl.unknown }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	420 func (gl *gmLog) getIgnoredMeasureCodes() []string { return gl.ignoredMeasureCodes }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	421 func (gl *gmLog) getRescaleErrors() []string { return gl.rescaleErrors }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	422 func (gl *gmLog) getMissingValues() []string { return gl.missingValues }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	423 func (gl *gmLog) getAssumedCM() int { return gl.assumedCM }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	424 func (gl *gmLog) getBadValues() int { return gl.badValues }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	425 func (gl *gmLog) getPredictions() int { return gl.predictions }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	426 func (gl *gmLog) getMeasurements() int { return gl.measurements }
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	427 func (gl *gmLog) nothingChanged() bool { return gl.measurements == 0 && gl.predictions == 0 }
5531 84f0c53e6ffe Completed GM logging aggregation. TODO: cleanup and dedup. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5530 diff changeset	428
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	429 func (gls gmLogs) logBool(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	430 access func(*gmLog) bool,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	431 header string,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	432 log func(string, ...interface{}),
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	433 ) {
5530 3ccbb6162112 WIP: More on GM import aggregation. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5529 diff changeset	434 var sb strings.Builder
3ccbb6162112 WIP: More on GM import aggregation. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5529 diff changeset	435 for _, gl := range gls {
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	436 if access(gl) {
5530 3ccbb6162112 WIP: More on GM import aggregation. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5529 diff changeset	437 if sb.Len() == 0 {
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	438 sb.WriteString(header)
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	439 } else {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	440 sb.WriteString(", ")
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	441 }
5529 f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	442 sb.WriteString(gl.gid)
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	443 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	444 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	445 if sb.Len() > 0 {
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	446 log(sb.String())
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	447 }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	448 }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	449
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	450 func (gls gmLogs) logInt(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	451 access func(*gmLog) int,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	452 header string,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	453 log func(string, ...interface{}),
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	454 ) {
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	455 var sb strings.Builder
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	456 for _, gl := range gls {
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	457 if v := access(gl); v > 0 {
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	458 if sb.Len() == 0 {
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	459 sb.WriteString(header)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	460 } else {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	461 sb.WriteString(", ")
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	462 }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	463 fmt.Fprintf(&sb, "%s (%d)", gl.gid, v)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	464 }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	465 }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	466 if sb.Len() > 0 {
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	467 log(sb.String())
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	468 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	469 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	470
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	471 func (gls gmLogs) logString(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	472 access func(*gmLog) []string,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	473 header string,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	474 log func(string, ...interface{}),
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	475 ) {
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	476 var sb strings.Builder
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	477 for _, gl := range gls {
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	478 if s := access(gl); len(s) > 0 {
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	479 if sb.Len() == 0 {
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	480 sb.WriteString(header)
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	481 } else {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	482 sb.WriteString(", ")
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	483 }
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	484 fmt.Fprintf(&sb, "%s (", gl.gid)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	485 for i, v := range s {
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	486 if i > 0 {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	487 sb.WriteString("; ")
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	488 }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	489 sb.WriteString(v)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	490 }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	491 sb.WriteByte(')')
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	492 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	493 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	494 if sb.Len() > 0 {
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	495 log(sb.String())
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	496 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	497 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	498
5529 f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	499 // logFinder is a helper to search recently used logs
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	500 // or create a new one if no log for a given gauge
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	501 // existed before.
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	502 func logFinder(logs gmLogs) func(string) gmLog {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	503 var lastLog *gmLog
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	504 return func(gid string) *gmLog {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	505 if lastLog != nil && lastLog.gid == gid {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	506 return lastLog
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	507 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	508 if ll := logs.find(gid); ll != nil {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	509 lastLog = ll
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	510 return ll
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	511 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	512 lastLog = &gmLog{gid: gid}
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	513 logs = append(logs, lastLog)
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	514 return lastLog
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	515 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	516 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	517
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	518 func doForGM(
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	519 ctx context.Context,
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	520 gauges []string,
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	521 fetch func() ([]*nts.RIS_Message_Type, error),
3176 1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import Tom Gottfried <tom@intevation.de> parents: 2849 diff changeset	522 conn *sql.Conn,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	523 feedback Feedback,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	524 ) ([]string, error) {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	525
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	526 insertGPStmt, err := conn.PrepareContext(ctx, insertGPSQL)
2234 9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	527 if err != nil {
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	528 return nil, err
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	529 }
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	530 defer insertGPStmt.Close()
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	531
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	532 insertGMStmt, err := conn.PrepareContext(ctx, insertGMSQL)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	533 if err != nil {
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	534 return nil, err
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	535 }
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	536 defer insertGMStmt.Close()
2234 9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	537
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	538 // lookup to see if data can be imported for gauge
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	539 isKnown := func(s string) bool {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	540 idx := sort.SearchStrings(gauges, s)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	541 return idx < len(gauges) && gauges[idx] == s
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	542 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	543
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	544 result, err := fetch()
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	545 if err != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	546 return nil, err
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	547 }
1981 e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	548
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	549 var gids []string
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	550
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	551 // To prevent spamming the log actual logging
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	552 // is defered to be presented in an aggregated way.
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	553 var logs gmLogs
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	554 defer func() { logs.logging(feedback) }()
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	555
5529 f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	556 findLog := logFinder(&logs)
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	557
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	558 for _, msg := range result {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	559 for _, wrm := range msg.Wrm {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	560 curr := string(*wrm.Geo_object.Id)
3718 7431c6680b90 import: Trim whitespaces in ISRS code and improved log messsage. Raimund Renkert <raimund.renkert@intevation.de> parents: 3709 diff changeset	561 curr = strings.TrimSpace(curr)
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	562 currIsrs, err := models.IsrsFromString(curr)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	563 if err != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	564 feedback.Warn("Invalid ISRS code %v", err)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	565 continue
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	566 }
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	567 logger := findLog(curr)
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	568 gids = append(gids, curr)
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	569
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	570 if !isKnown(curr) {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	571 logger.unknown = true
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	572 continue
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	573 }
1981 e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	574
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	575 var referenceCode string
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	576 if wrm.Reference_code == nil {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	577 logger.assumedZPG = true
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	578 referenceCode = "ZPG"
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	579 } else {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	580 referenceCode = string(*wrm.Reference_code)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	581 }
2849 b8972e4671fa Preserve old predictions on raw gauge measurement import. Sascha Wilde <wilde@intevation.de> parents: 2245 diff changeset	582
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	583 for _, measure := range wrm.Measure {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	584 var unit string
3723 9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	585 if *measure.Measure_code != nts.Measure_code_enumWAL {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	586 logger.ignoreMeasureCode(*measure.Measure_code)
3723 9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	587 continue
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	588 }
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	589 if measure.Unit == nil {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	590 logger.assumedCM++
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	591 unit = "cm"
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	592 } else {
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	593 unit = string(*measure.Unit)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	594 }
4813 0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	595
0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	596 if measure.Value == nil {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	597 logger.missingValues = append(
0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	598 logger.missingValues,
5531 84f0c53e6ffe Completed GM logging aggregation. TODO: cleanup and dedup. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5530 diff changeset	599 measure.Measuredate.Time.Format(time.RFC3339))
4813 0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	600 continue
0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	601 }
0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	602
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	603 convert, err := rescale(unit)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	604 if err != nil {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	605 logger.addRescaleError(err)
4848 8584197232ec Handle unknown units gracefully Tom Gottfried <tom@intevation.de> parents: 4845 diff changeset	606 continue
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	607 }
3264 9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	608 convert(measure.Value)
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	609 convert(measure.Value_min)
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	610 convert(measure.Value_max)
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	611
3847 d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	612 // -99999 is used by some gauges to signal an error
4813 0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	613 if *measure.Value == -99999 {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	614 logger.badValues++
3847 d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	615 continue
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	616 }
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	617
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	618 var dummy int
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	619 if measure.Predicted {
3723 9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	620 confInterval := pgtype.Numrange{
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	621 Lower: pgtype.Numeric{Status: pgtype.Null},
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	622 Upper: pgtype.Numeric{Status: pgtype.Null},
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	623 LowerType: pgtype.Inclusive,
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	624 UpperType: pgtype.Inclusive,
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	625 Status: pgtype.Null,
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	626 }
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	627 if measure.Value_min != nil && measure.Value_max != nil {
3723 9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	628 valueMin := pgtype.Numeric{Status: pgtype.Null}
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	629 valueMax := pgtype.Numeric{Status: pgtype.Null}
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	630 valueMin.Set(measure.Value_min)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	631 valueMax.Set(measure.Value_max)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	632 confInterval = pgtype.Numrange{
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	633 Lower: valueMin,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	634 Upper: valueMax,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	635 LowerType: pgtype.Inclusive,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	636 UpperType: pgtype.Inclusive,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	637 Status: pgtype.Present,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	638 }
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	639 }
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	640 err = insertGPStmt.QueryRowContext(
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	641 ctx,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	642 currIsrs.CountryCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	643 currIsrs.LoCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	644 currIsrs.FairwaySection,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	645 currIsrs.Orc,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	646 currIsrs.Hectometre,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	647 measure.Measuredate.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	648 msg.Identification.From,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	649 msg.Identification.Language_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	650 msg.Identification.Country_code,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	651 msg.Identification.Date_issue.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	652 referenceCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	653 measure.Value,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	654 &confInterval,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	655 msg.Identification.Date_issue.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	656 msg.Identification.Originator,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	657 ).Scan(&dummy)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	658 switch {
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	659 case err == sql.ErrNoRows:
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	660 // thats expected, nothing to do
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	661 case err != nil:
4197 5d7ce7f926eb Log fatal errors as errors, not warnings, in imports Tom Gottfried <tom@intevation.de> parents: 4158 diff changeset	662 feedback.Error(pgxutils.ReadableError{Err: err}.Error())
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	663 default:
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	664 logger.predictions++
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	665 }
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	666 } else {
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	667 err = insertGMStmt.QueryRowContext(
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	668 ctx,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	669 currIsrs.CountryCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	670 currIsrs.LoCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	671 currIsrs.FairwaySection,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	672 currIsrs.Orc,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	673 currIsrs.Hectometre,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	674 measure.Measuredate.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	675 msg.Identification.From,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	676 msg.Identification.Language_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	677 msg.Identification.Country_code,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	678 msg.Identification.Date_issue.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	679 referenceCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	680 measure.Value,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	681 msg.Identification.Date_issue.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	682 msg.Identification.Originator,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	683 ).Scan(&dummy)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	684 switch {
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	685 case err == sql.ErrNoRows:
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	686 // thats expected, nothing to do
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	687 case err != nil:
4197 5d7ce7f926eb Log fatal errors as errors, not warnings, in imports Tom Gottfried <tom@intevation.de> parents: 4158 diff changeset	688 feedback.Error(pgxutils.ReadableError{Err: err}.Error())
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	689 default:
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	690 logger.measurements++
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	691 }
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	692 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	693 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	694 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	695 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	696 return gids, nil
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	697 }

Mercurial > gemma

annotate pkg/imports/gm.go @ 5541:29804c8e817d aggregate-gm-import-logging