gemma: pkg/imports/gm.go annotate

annotate pkg/imports/gm.go @ 5598:7724bb582e80

Fixed logging in GM import.

author	Sascha L. Teichmann <sascha.teichmann@intevation.de>
date	Fri, 05 Aug 2022 13:06:43 +0200
parents	f2204f91d286
children	2dd155cc95ec

rev	line source
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	3 //
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	6 //
3176 1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import Tom Gottfried <tom@intevation.de> parents: 2849 diff changeset	7 // Copyright (C) 2018, 2019 by via donau
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	10 //
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	11 // Author(s):
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	12 // * Raimund Renkert <raimund.renkert@intevation.de>
2245 7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2244 diff changeset	13 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
3176 1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import Tom Gottfried <tom@intevation.de> parents: 2849 diff changeset	14 // * Tom Gottfried <tom.gottfried@intevation.de>
1694 4a2fad8f57de Imports: Resolved golint issues unrelated to exported symbols commenting. Sascha L. Teichmann <teichmann@intevation.de> parents: 1688 diff changeset	15
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	16 package imports
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	17
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	18 import (
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	19 "context"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	20 "database/sql"
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	21 "fmt"
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	22 "sort"
5545 a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	23 "strconv"
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	24 "strings"
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	25 "time"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	26
5490 5f47eeea988d Use own logging package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5035 diff changeset	27 "github.com/jackc/pgx/pgtype"
5f47eeea988d Use own logging package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5035 diff changeset	28
5f47eeea988d Use own logging package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5035 diff changeset	29 "gemma.intevation.de/gemma/pkg/log"
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	30 "gemma.intevation.de/gemma/pkg/models"
4062 6c760abcff0e Move handling of PostgreSQL errors to own package Tom Gottfried <tom@intevation.de> parents: 4058 diff changeset	31 "gemma.intevation.de/gemma/pkg/pgxutils"
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	32 "gemma.intevation.de/gemma/pkg/soap/nts"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	33 )
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	34
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	35 // GaugeMeasurement is an import job to import
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	36 // gauges measurement data from a NtS SOAP service.
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	37 type GaugeMeasurement struct {
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	38 // URL is the URL of the SOAP service.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	39 URL string `json:"url"`
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	40 // Insecure indicates if HTTPS traffic
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	41 // should validate certificates or not.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	42 Insecure bool `json:"insecure"`
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	43 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	44
4852 046a07a33b19 Fixed the golint issues of the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4848 diff changeset	45 // Description gives a short info about relevant facts of this import.
5560 f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5551 diff changeset	46 func (gm *GaugeMeasurement) Description([]string) (string, error) {
4799 f32d086b5dbf Removed the mechanical touch of the last commit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4798 diff changeset	47 return gm.URL, nil
4798 ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4505 diff changeset	48 }
ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4505 diff changeset	49
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	50 // GMJobKind is the import queue type identifier.
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	51 const GMJobKind JobKind = "gm"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	52
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	53 const (
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	54 listGaugesSQL = `
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	55 SELECT
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	56 (location).country_code,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	57 (location).locode,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	58 (location).fairway_section,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	59 (location).orc,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	60 (location).hectometre
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	61 FROM waterway.gauges
4158 5466562cca60 Remove utility function with possibly bad performance impact Tom Gottfried <tom@intevation.de> parents: 4147 diff changeset	62 WHERE (location).country_code = (
5466562cca60 Remove utility function with possibly bad performance impact Tom Gottfried <tom@intevation.de> parents: 4147 diff changeset	63 SELECT country FROM users.list_users WHERE username = current_user)
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	64 OR pg_has_role('sys_admin', 'MEMBER')
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	65 `
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	66
2849 b8972e4671fa Preserve old predictions on raw gauge measurement import. Sascha Wilde <wilde@intevation.de> parents: 2245 diff changeset	67 // Note: we do not expect corrections of data through this service. So
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	68 // any constraint conflicts are triggered by redundant data which
2849 b8972e4671fa Preserve old predictions on raw gauge measurement import. Sascha Wilde <wilde@intevation.de> parents: 2245 diff changeset	69 // can be dropped.
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	70 insertGMSQL = `
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	71 INSERT INTO waterway.gauge_measurements (
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	72 location,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	73 measure_date,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	74 sender,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	75 language_code,
1766 e51db2a365a2 Insert country code in gm import. Raimund Renkert <raimund.renkert@intevation.de> parents: 1761 diff changeset	76 country_code,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	77 date_issue,
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	78 reference_code,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	79 water_level,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	80 date_info,
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1776 diff changeset	81 source_organization,
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1776 diff changeset	82 staging_done
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	83 ) VALUES (
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	84 ($1, $2, $3, $4, $5),
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	85 $6,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	86 $7,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	87 $8,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	88 $9,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	89 $10,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	90 $11,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	91 $12,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	92 $13,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	93 $14,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	94 true
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	95 )
2849 b8972e4671fa Preserve old predictions on raw gauge measurement import. Sascha Wilde <wilde@intevation.de> parents: 2245 diff changeset	96 ON CONFLICT DO NOTHING
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	97 RETURNING 1
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	98 `
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	99
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	100 insertGPSQL = `
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	101 INSERT INTO waterway.gauge_predictions (
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	102 location,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	103 measure_date,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	104 sender,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	105 language_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	106 country_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	107 date_issue,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	108 reference_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	109 water_level,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	110 conf_interval,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	111 date_info,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	112 source_organization
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	113 ) VALUES(
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	114 ($1, $2, $3, $4, $5),
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	115 $6,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	116 $7,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	117 $8,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	118 $9,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	119 $10,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	120 $11,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	121 $12,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	122 $13,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	123 $14,
3278 831193935739 Avoid storing of NtS messages other than water level Tom Gottfried <tom@intevation.de> parents: 3277 diff changeset	124 $15
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	125 )
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	126 ON CONFLICT DO NOTHING
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	127 RETURNING 1
1978 0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	128 `
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	129 )
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	130
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	131 type gmJobCreator struct{}
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	132
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	133 func init() {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	134 RegisterJobCreator(GMJobKind, gmJobCreator{})
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	135 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	136
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	137 func (gmJobCreator) Description() string { return "gauge measurements" }
1646 a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1641 diff changeset	138
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	139 func (gmJobCreator) Create() Job { return new(GaugeMeasurement) }
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	140
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3176 diff changeset	141 func (gmJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3176 diff changeset	142 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3176 diff changeset	143 {"gauge_measurements"},
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3176 diff changeset	144 {"gauges"},
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	145 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	146 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	147
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	148 func (gmJobCreator) AutoAccept() bool { return true }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	149
5035 56c589f7435d Enhance comments on implementations of StageDone() Tom Gottfried <tom@intevation.de> parents: 5034 diff changeset	150 // StageDone is a NOP for gauge measurements imports.
5034 59a99655f34d Added feedback support for StageDone. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4852 diff changeset	151 func (gmJobCreator) StageDone(context.Context, *sql.Tx, int64, Feedback) error {
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	152 return nil
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	153 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	154
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	155 // CleanUp of a gauge measurement import is a NOP.
1676 4407ecaa2192 Imports: Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1664 diff changeset	156 func (*GaugeMeasurement) CleanUp() error { return nil }
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	157
4845 8c46b845b406 Fix comment Tom Gottfried <tom@intevation.de> parents: 4813 diff changeset	158 // Do executes the actual import.
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	159 func (gm *GaugeMeasurement) Do(
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	160 ctx context.Context,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	161 importID int64,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	162 conn *sql.Conn,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	163 feedback Feedback,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	164 ) (interface{}, error) {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	165
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	166 fetch := func() ([]*nts.RIS_Message_Type, error) {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	167 client := nts.NewINtSMessageService(gm.URL, gm.Insecure, nil)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	168
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	169 mt := nts.Message_type_typeWRM
4504 b7e3d42fd9e7 When fetching gauge measurements look 1day into past and future. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4197 diff changeset	170 now := time.Now()
4505 30c499f996d7 Simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4504 diff changeset	171 dis := []*nts.Date_pair{{
4504 b7e3d42fd9e7 When fetching gauge measurements look 1day into past and future. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4197 diff changeset	172 Date_start: nts.Date{Time: now.AddDate(0, 0, -1)},
b7e3d42fd9e7 When fetching gauge measurements look 1day into past and future. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4197 diff changeset	173 Date_end: nts.Date{Time: now.AddDate(0, 0, +1)},
4505 30c499f996d7 Simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4504 diff changeset	174 }}
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	175
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	176 req := &nts.Get_messages_query{
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	177 Message_type: &mt,
3709 d4a8d084eab4 import: Use dates_issue parameter in gm imports. Raimund Renkert <raimund.renkert@intevation.de> parents: 3583 diff changeset	178 Dates_issue: dis,
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	179 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	180
3582 317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	181 const maxTries = 3
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	182
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	183 tries := 0
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	184
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	185 again:
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	186 resp, err := client.Get_messages(req)
3582 317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	187
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	188 if err != nil {
3582 317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	189 if t, ok := err.(interface{ Timeout() bool }); ok && t.Timeout() && tries < maxTries {
5490 5f47eeea988d Use own logging package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5035 diff changeset	190 log.Warnln("NtS SOAP request timed out. Trying again.")
3582 317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	191 tries++
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	192 goto again
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	193 }
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	194 return nil, fmt.Errorf(
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	195 "Error requesting NtS service: %v", err)
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	196 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	197
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	198 result := resp.Result_message
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	199 if result == nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	200 for _, e := range resp.Result_error {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	201 if e != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	202 feedback.Error("Error code: %s", *e)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	203 } else {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	204 feedback.Error("Unknown error")
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	205 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	206 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	207 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	208 return result, nil
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	209 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	210
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	211 return storeGaugeMeasurements(
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	212 ctx,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	213 importID,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	214 fetch,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	215 conn,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	216 feedback,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	217 )
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	218 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	219
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	220 func loadGauges(ctx context.Context, conn *sql.Conn) ([]string, error) {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	221
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	222 rows, err := conn.QueryContext(ctx, listGaugesSQL)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	223 if err != nil {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	224 return nil, err
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	225 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	226 defer rows.Close()
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	227
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	228 var gauges []string
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	229
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	230 for rows.Next() {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	231 var g models.Isrs
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	232 if err = rows.Scan(
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	233 &g.CountryCode,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	234 &g.LoCode,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	235 &g.FairwaySection,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	236 &g.Orc,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	237 &g.Hectometre,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	238 ); err != nil {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	239 return nil, err
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	240 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	241 gauges = append(gauges, g.String())
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	242 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	243
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	244 if err = rows.Err(); err != nil {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	245 return nil, err
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	246 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	247
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	248 if len(gauges) == 0 {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	249 return nil, UnchangedError(
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	250 "No gauges for which measurements can be imported in database")
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	251 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	252
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	253 sort.Strings(gauges)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	254
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	255 return gauges, nil
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	256 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	257
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	258 func storeGaugeMeasurements(
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	259 ctx context.Context,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	260 importID int64,
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	261 fetch func() ([]*nts.RIS_Message_Type, error),
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	262 conn *sql.Conn,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	263 feedback Feedback,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	264 ) (interface{}, error) {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	265
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	266 start := time.Now()
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	267
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	268 // Get gauges from database, for which user is allowed to import data
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	269 gauges, err := loadGauges(ctx, conn)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	270 if err != nil {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	271 return nil, err
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	272 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	273
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	274 gids, err := doForGM(ctx, gauges, fetch, conn, feedback)
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	275 if err != nil {
1761 0145537d5474 Gauge measument import: Die with an error if something went wrong. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1754 diff changeset	276 return nil, err
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	277 }
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	278
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	279 if len(gids) == 0 {
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	280 return nil, UnchangedError("No new gauge measurements found")
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	281 }
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	282
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	283 feedback.Info(
3176 1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import Tom Gottfried <tom@intevation.de> parents: 2849 diff changeset	284 "Importing gauge measurements took %s", time.Since(start))
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	285
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	286 // TODO: needs to be filled more useful.
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	287 summary := struct {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	288 GaugeMeasuremets []string `json:"gaugeMeasurements"`
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	289 }{
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	290 GaugeMeasuremets: gids,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	291 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	292 return &summary, err
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	293 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	294
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	295 // rescale returns a scaling function to bring the unit all to cm.
3264 9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	296 func rescale(unit string) (func(*float32), error) {
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	297
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	298 var scale float32
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	299
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	300 switch strings.ToLower(unit) {
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	301 case "mm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	302 scale = 0.1
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	303 case "cm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	304 scale = 1.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	305 case "dm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	306 scale = 10.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	307 case "m":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	308 scale = 100.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	309 case "hm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	310 scale = 10000.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	311 case "km":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	312 scale = 100000.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	313 default:
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	314 return nil, fmt.Errorf("unknown unit '%s'", unit)
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	315 }
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	316
3264 9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	317 fn := func(x *float32) {
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	318 if x != nil {
3266 3dee5cf16a58 Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3264 diff changeset	319 x = scale
3264 9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	320 }
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	321 }
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	322 return fn, nil
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	323 }
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	324
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	325 type gmLog struct {
5529 f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	326 gid string
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	327 unknown bool
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	328 assumedZPG bool
5529 f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	329 ignoredMeasureCodes []string
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	330 rescaleErrors []string
5531 84f0c53e6ffe Completed GM logging aggregation. TODO: cleanup and dedup. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5530 diff changeset	331 missingValues []string
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	332 assumedCM int
5530 3ccbb6162112 WIP: More on GM import aggregation. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5529 diff changeset	333 badValues int
5531 84f0c53e6ffe Completed GM logging aggregation. TODO: cleanup and dedup. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5530 diff changeset	334 measurements int
84f0c53e6ffe Completed GM logging aggregation. TODO: cleanup and dedup. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5530 diff changeset	335 predictions int
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	336 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	337
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	338 type gmLogs []*gmLog
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	339
5529 f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	340 func extend(haystack []string, needle string) []string {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	341 for _, straw := range haystack {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	342 if straw == needle {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	343 return haystack
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	344 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	345 }
5529 f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	346 return append(haystack, needle)
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	347 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	348
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	349 func (gl *gmLog) addRescaleError(err error) {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	350 gl.rescaleErrors = extend(gl.rescaleErrors, err.Error())
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	351 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	352
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	353 func (gl *gmLog) ignoreMeasureCode(mc nts.Measure_code_enum) {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	354 gl.ignoredMeasureCodes = extend(gl.ignoredMeasureCodes, string(mc))
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	355 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	356
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	357 func (gls gmLogs) find(gid string) *gmLog {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	358 for _, gl := range gls {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	359 if gl.gid == gid {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	360 return gl
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	361 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	362 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	363 return nil
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	364 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	365
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	366 func (gls gmLogs) logging(feedback Feedback) {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	367
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	368 gls.logBool(
5551 e9ef27c75e5c Fixed minor typo. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5547 diff changeset	369 (*gmLog).getUnknown,
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	370 "Cannot find following gauges: ",
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	371 feedback.Warn)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	372
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	373 gls.logBool(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	374 (*gmLog).getAssumedZPG,
5537 2e5e51288a6c Fixed typo in generated logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5533 diff changeset	375 "'Reference_code' not specified. Assuming 'ZPG': ",
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	376 feedback.Warn)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	377
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	378 gls.logInt(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	379 (*gmLog).getAssumedCM,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	380 "'Unit' not specified. Assuming 'cm': ",
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	381 feedback.Warn)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	382
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	383 gls.logInt(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	384 (*gmLog).getBadValues,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	385 "Ignored measurements with value -99999: ",
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	386 feedback.Warn)
5531 84f0c53e6ffe Completed GM logging aggregation. TODO: cleanup and dedup. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5530 diff changeset	387
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	388 gls.logString(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	389 (*gmLog).getMissingValues,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	390 "Missing mandatory values: ",
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	391 feedback.Warn)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	392
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	393 gls.logString(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	394 (*gmLog).getRescaleErrors,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	395 "Cannot convert units: ",
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	396 feedback.Error)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	397
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	398 gls.logString(
5598 7724bb582e80 Fixed logging in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5560 diff changeset	399 (*gmLog).getIgnoredMeasureCodes,
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	400 "Ignored measure codes: ",
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	401 feedback.Warn)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	402
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	403 gls.logInt(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	404 (*gmLog).getPredictions,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	405 "New predictions: ",
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	406 feedback.Info)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	407
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	408 gls.logInt(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	409 (*gmLog).getMeasurements,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	410 "New measurements: ",
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	411 feedback.Info)
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	412
0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	413 gls.logBool(
0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	414 (*gmLog).nothingChanged,
0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	415 "No changes for: ",
0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	416 feedback.Info)
5531 84f0c53e6ffe Completed GM logging aggregation. TODO: cleanup and dedup. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5530 diff changeset	417 }
84f0c53e6ffe Completed GM logging aggregation. TODO: cleanup and dedup. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5530 diff changeset	418
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	419 func (gl *gmLog) getAssumedZPG() bool { return gl.assumedZPG }
5551 e9ef27c75e5c Fixed minor typo. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5547 diff changeset	420 func (gl *gmLog) getUnknown() bool { return gl.unknown }
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	421 func (gl *gmLog) getIgnoredMeasureCodes() []string { return gl.ignoredMeasureCodes }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	422 func (gl *gmLog) getRescaleErrors() []string { return gl.rescaleErrors }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	423 func (gl *gmLog) getMissingValues() []string { return gl.missingValues }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	424 func (gl *gmLog) getAssumedCM() int { return gl.assumedCM }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	425 func (gl *gmLog) getBadValues() int { return gl.badValues }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	426 func (gl *gmLog) getPredictions() int { return gl.predictions }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	427 func (gl *gmLog) getMeasurements() int { return gl.measurements }
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	428 func (gl *gmLog) nothingChanged() bool { return gl.measurements == 0 && gl.predictions == 0 }
5531 84f0c53e6ffe Completed GM logging aggregation. TODO: cleanup and dedup. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5530 diff changeset	429
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	430 func (gls gmLogs) logBool(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	431 access func(*gmLog) bool,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	432 header string,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	433 log func(string, ...interface{}),
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	434 ) {
5530 3ccbb6162112 WIP: More on GM import aggregation. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5529 diff changeset	435 var sb strings.Builder
3ccbb6162112 WIP: More on GM import aggregation. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5529 diff changeset	436 for _, gl := range gls {
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	437 if access(gl) {
5530 3ccbb6162112 WIP: More on GM import aggregation. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5529 diff changeset	438 if sb.Len() == 0 {
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	439 sb.WriteString(header)
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	440 } else {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	441 sb.WriteString(", ")
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	442 }
5529 f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	443 sb.WriteString(gl.gid)
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	444 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	445 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	446 if sb.Len() > 0 {
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	447 log(sb.String())
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	448 }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	449 }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	450
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	451 func (gls gmLogs) logInt(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	452 access func(*gmLog) int,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	453 header string,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	454 log func(string, ...interface{}),
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	455 ) {
5545 a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	456 gs := make(gmLogs, 0, len(gls))
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	457 for _, g := range gls {
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	458 if access(g) > 0 {
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	459 gs = append(gs, g)
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	460 }
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	461 }
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	462
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	463 if len(gs) == 0 {
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	464 return
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	465 }
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	466
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	467 sort.SliceStable(gs, func(i, j int) bool {
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	468 return access(gs[i]) < access(gs[j])
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	469 })
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	470
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	471 var sb strings.Builder
5545 a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	472 var last int
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	473
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	474 for _, g := range gs {
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	475 if c := access(g); c != last {
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	476 if sb.Len() == 0 {
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	477 sb.WriteString(header)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	478 } else {
5545 a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	479 sb.WriteString("); ")
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	480 }
5545 a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	481 sb.WriteString(strconv.Itoa(c))
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	482 sb.WriteString(" (")
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	483 last = c
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	484 } else {
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	485 sb.WriteString(", ")
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	486 }
5545 a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	487 sb.WriteString(g.gid)
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	488 }
5545 a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	489
5547 f8c370af9e6f Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5545 diff changeset	490 sb.WriteByte(')')
5545 a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	491 log(sb.String())
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	492 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	493
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	494 func (gls gmLogs) logString(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	495 access func(*gmLog) []string,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	496 header string,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	497 log func(string, ...interface{}),
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	498 ) {
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	499 var sb strings.Builder
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	500 for _, gl := range gls {
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	501 if s := access(gl); len(s) > 0 {
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	502 if sb.Len() == 0 {
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	503 sb.WriteString(header)
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	504 } else {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	505 sb.WriteString(", ")
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	506 }
5547 f8c370af9e6f Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5545 diff changeset	507 sb.WriteString(gl.gid)
f8c370af9e6f Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5545 diff changeset	508 sb.WriteString(" (")
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	509 for i, v := range s {
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	510 if i > 0 {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	511 sb.WriteString("; ")
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	512 }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	513 sb.WriteString(v)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	514 }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	515 sb.WriteByte(')')
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	516 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	517 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	518 if sb.Len() > 0 {
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	519 log(sb.String())
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	520 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	521 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	522
5529 f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	523 // logFinder is a helper to search recently used logs
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	524 // or create a new one if no log for a given gauge
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	525 // existed before.
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	526 func logFinder(logs gmLogs) func(string) gmLog {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	527 var lastLog *gmLog
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	528 return func(gid string) *gmLog {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	529 if lastLog != nil && lastLog.gid == gid {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	530 return lastLog
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	531 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	532 if ll := logs.find(gid); ll != nil {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	533 lastLog = ll
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	534 return ll
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	535 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	536 lastLog = &gmLog{gid: gid}
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	537 logs = append(logs, lastLog)
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	538 return lastLog
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	539 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	540 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	541
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	542 func doForGM(
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	543 ctx context.Context,
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	544 gauges []string,
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	545 fetch func() ([]*nts.RIS_Message_Type, error),
3176 1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import Tom Gottfried <tom@intevation.de> parents: 2849 diff changeset	546 conn *sql.Conn,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	547 feedback Feedback,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	548 ) ([]string, error) {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	549
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	550 insertGPStmt, err := conn.PrepareContext(ctx, insertGPSQL)
2234 9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	551 if err != nil {
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	552 return nil, err
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	553 }
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	554 defer insertGPStmt.Close()
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	555
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	556 insertGMStmt, err := conn.PrepareContext(ctx, insertGMSQL)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	557 if err != nil {
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	558 return nil, err
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	559 }
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	560 defer insertGMStmt.Close()
2234 9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	561
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	562 // lookup to see if data can be imported for gauge
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	563 isKnown := func(s string) bool {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	564 idx := sort.SearchStrings(gauges, s)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	565 return idx < len(gauges) && gauges[idx] == s
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	566 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	567
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	568 result, err := fetch()
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	569 if err != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	570 return nil, err
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	571 }
1981 e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	572
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	573 var gids []string
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	574
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	575 // To prevent spamming the log actual logging
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	576 // is defered to be presented in an aggregated way.
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	577 var logs gmLogs
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	578 defer func() { logs.logging(feedback) }()
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	579
5529 f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	580 findLog := logFinder(&logs)
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	581
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	582 for _, msg := range result {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	583 for _, wrm := range msg.Wrm {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	584 curr := string(*wrm.Geo_object.Id)
3718 7431c6680b90 import: Trim whitespaces in ISRS code and improved log messsage. Raimund Renkert <raimund.renkert@intevation.de> parents: 3709 diff changeset	585 curr = strings.TrimSpace(curr)
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	586 currIsrs, err := models.IsrsFromString(curr)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	587 if err != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	588 feedback.Warn("Invalid ISRS code %v", err)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	589 continue
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	590 }
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	591 logger := findLog(curr)
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	592 gids = append(gids, curr)
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	593
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	594 if !isKnown(curr) {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	595 logger.unknown = true
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	596 continue
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	597 }
1981 e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	598
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	599 var referenceCode string
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	600 if wrm.Reference_code == nil {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	601 logger.assumedZPG = true
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	602 referenceCode = "ZPG"
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	603 } else {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	604 referenceCode = string(*wrm.Reference_code)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	605 }
2849 b8972e4671fa Preserve old predictions on raw gauge measurement import. Sascha Wilde <wilde@intevation.de> parents: 2245 diff changeset	606
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	607 for _, measure := range wrm.Measure {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	608 var unit string
3723 9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	609 if *measure.Measure_code != nts.Measure_code_enumWAL {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	610 logger.ignoreMeasureCode(*measure.Measure_code)
3723 9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	611 continue
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	612 }
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	613 if measure.Unit == nil {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	614 logger.assumedCM++
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	615 unit = "cm"
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	616 } else {
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	617 unit = string(*measure.Unit)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	618 }
4813 0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	619
0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	620 if measure.Value == nil {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	621 logger.missingValues = append(
0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	622 logger.missingValues,
5531 84f0c53e6ffe Completed GM logging aggregation. TODO: cleanup and dedup. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5530 diff changeset	623 measure.Measuredate.Time.Format(time.RFC3339))
4813 0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	624 continue
0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	625 }
0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	626
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	627 convert, err := rescale(unit)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	628 if err != nil {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	629 logger.addRescaleError(err)
4848 8584197232ec Handle unknown units gracefully Tom Gottfried <tom@intevation.de> parents: 4845 diff changeset	630 continue
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	631 }
3264 9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	632 convert(measure.Value)
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	633 convert(measure.Value_min)
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	634 convert(measure.Value_max)
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	635
3847 d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	636 // -99999 is used by some gauges to signal an error
4813 0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	637 if *measure.Value == -99999 {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	638 logger.badValues++
3847 d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	639 continue
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	640 }
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	641
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	642 var dummy int
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	643 if measure.Predicted {
3723 9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	644 confInterval := pgtype.Numrange{
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	645 Lower: pgtype.Numeric{Status: pgtype.Null},
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	646 Upper: pgtype.Numeric{Status: pgtype.Null},
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	647 LowerType: pgtype.Inclusive,
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	648 UpperType: pgtype.Inclusive,
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	649 Status: pgtype.Null,
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	650 }
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	651 if measure.Value_min != nil && measure.Value_max != nil {
3723 9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	652 valueMin := pgtype.Numeric{Status: pgtype.Null}
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	653 valueMax := pgtype.Numeric{Status: pgtype.Null}
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	654 valueMin.Set(measure.Value_min)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	655 valueMax.Set(measure.Value_max)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	656 confInterval = pgtype.Numrange{
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	657 Lower: valueMin,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	658 Upper: valueMax,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	659 LowerType: pgtype.Inclusive,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	660 UpperType: pgtype.Inclusive,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	661 Status: pgtype.Present,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	662 }
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	663 }
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	664 err = insertGPStmt.QueryRowContext(
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	665 ctx,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	666 currIsrs.CountryCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	667 currIsrs.LoCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	668 currIsrs.FairwaySection,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	669 currIsrs.Orc,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	670 currIsrs.Hectometre,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	671 measure.Measuredate.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	672 msg.Identification.From,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	673 msg.Identification.Language_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	674 msg.Identification.Country_code,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	675 msg.Identification.Date_issue.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	676 referenceCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	677 measure.Value,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	678 &confInterval,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	679 msg.Identification.Date_issue.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	680 msg.Identification.Originator,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	681 ).Scan(&dummy)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	682 switch {
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	683 case err == sql.ErrNoRows:
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	684 // thats expected, nothing to do
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	685 case err != nil:
4197 5d7ce7f926eb Log fatal errors as errors, not warnings, in imports Tom Gottfried <tom@intevation.de> parents: 4158 diff changeset	686 feedback.Error(pgxutils.ReadableError{Err: err}.Error())
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	687 default:
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	688 logger.predictions++
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	689 }
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	690 } else {
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	691 err = insertGMStmt.QueryRowContext(
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	692 ctx,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	693 currIsrs.CountryCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	694 currIsrs.LoCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	695 currIsrs.FairwaySection,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	696 currIsrs.Orc,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	697 currIsrs.Hectometre,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	698 measure.Measuredate.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	699 msg.Identification.From,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	700 msg.Identification.Language_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	701 msg.Identification.Country_code,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	702 msg.Identification.Date_issue.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	703 referenceCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	704 measure.Value,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	705 msg.Identification.Date_issue.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	706 msg.Identification.Originator,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	707 ).Scan(&dummy)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	708 switch {
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	709 case err == sql.ErrNoRows:
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	710 // thats expected, nothing to do
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	711 case err != nil:
4197 5d7ce7f926eb Log fatal errors as errors, not warnings, in imports Tom Gottfried <tom@intevation.de> parents: 4158 diff changeset	712 feedback.Error(pgxutils.ReadableError{Err: err}.Error())
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	713 default:
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	714 logger.measurements++
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	715 }
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	716 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	717 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	718 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	719 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	720 return gids, nil
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	721 }

Mercurial > gemma

annotate pkg/imports/gm.go @ 5598:7724bb582e80