gemma: pkg/imports/gm.go annotate

annotate pkg/imports/gm.go @ 5591:0011f50cf216 surveysperbottleneckid

Removed no longer used alternative api for surveys/ endpoint. As bottlenecks in the summary for SR imports are now identified by their id and no longer by the (not guarantied to be unique!) name, there is no longer the need to request survey data by the name+date tuple (which isn't reliable anyway). So the workaround was now reversed.

author	Sascha Wilde <wilde@sha-bang.de>
date	Wed, 06 Apr 2022 13:30:29 +0200
parents	f2204f91d286
children	7724bb582e80

rev	line source
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	3 //
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	6 //
3176 1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import Tom Gottfried <tom@intevation.de> parents: 2849 diff changeset	7 // Copyright (C) 2018, 2019 by via donau
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	10 //
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	11 // Author(s):
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	12 // * Raimund Renkert <raimund.renkert@intevation.de>
2245 7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2244 diff changeset	13 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
3176 1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import Tom Gottfried <tom@intevation.de> parents: 2849 diff changeset	14 // * Tom Gottfried <tom.gottfried@intevation.de>
1694 4a2fad8f57de Imports: Resolved golint issues unrelated to exported symbols commenting. Sascha L. Teichmann <teichmann@intevation.de> parents: 1688 diff changeset	15
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	16 package imports
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	17
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	18 import (
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	19 "context"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	20 "database/sql"
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	21 "fmt"
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	22 "sort"
5545 a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	23 "strconv"
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	24 "strings"
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	25 "time"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	26
5490 5f47eeea988d Use own logging package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5035 diff changeset	27 "github.com/jackc/pgx/pgtype"
5f47eeea988d Use own logging package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5035 diff changeset	28
5f47eeea988d Use own logging package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5035 diff changeset	29 "gemma.intevation.de/gemma/pkg/log"
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	30 "gemma.intevation.de/gemma/pkg/models"
4062 6c760abcff0e Move handling of PostgreSQL errors to own package Tom Gottfried <tom@intevation.de> parents: 4058 diff changeset	31 "gemma.intevation.de/gemma/pkg/pgxutils"
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	32 "gemma.intevation.de/gemma/pkg/soap/nts"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	33 )
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	34
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	35 // GaugeMeasurement is an import job to import
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	36 // gauges measurement data from a NtS SOAP service.
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	37 type GaugeMeasurement struct {
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	38 // URL is the URL of the SOAP service.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	39 URL string `json:"url"`
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	40 // Insecure indicates if HTTPS traffic
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	41 // should validate certificates or not.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	42 Insecure bool `json:"insecure"`
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	43 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	44
4852 046a07a33b19 Fixed the golint issues of the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4848 diff changeset	45 // Description gives a short info about relevant facts of this import.
5560 f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5551 diff changeset	46 func (gm *GaugeMeasurement) Description([]string) (string, error) {
4799 f32d086b5dbf Removed the mechanical touch of the last commit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4798 diff changeset	47 return gm.URL, nil
4798 ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4505 diff changeset	48 }
ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4505 diff changeset	49
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	50 // GMJobKind is the import queue type identifier.
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	51 const GMJobKind JobKind = "gm"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	52
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	53 const (
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	54 listGaugesSQL = `
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	55 SELECT
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	56 (location).country_code,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	57 (location).locode,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	58 (location).fairway_section,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	59 (location).orc,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	60 (location).hectometre
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	61 FROM waterway.gauges
4158 5466562cca60 Remove utility function with possibly bad performance impact Tom Gottfried <tom@intevation.de> parents: 4147 diff changeset	62 WHERE (location).country_code = (
5466562cca60 Remove utility function with possibly bad performance impact Tom Gottfried <tom@intevation.de> parents: 4147 diff changeset	63 SELECT country FROM users.list_users WHERE username = current_user)
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	64 OR pg_has_role('sys_admin', 'MEMBER')
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	65 `
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	66
2849 b8972e4671fa Preserve old predictions on raw gauge measurement import. Sascha Wilde <wilde@intevation.de> parents: 2245 diff changeset	67 // Note: we do not expect corrections of data through this service. So
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	68 // any constraint conflicts are triggered by redundant data which
2849 b8972e4671fa Preserve old predictions on raw gauge measurement import. Sascha Wilde <wilde@intevation.de> parents: 2245 diff changeset	69 // can be dropped.
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	70 insertGMSQL = `
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	71 INSERT INTO waterway.gauge_measurements (
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	72 location,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	73 measure_date,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	74 sender,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	75 language_code,
1766 e51db2a365a2 Insert country code in gm import. Raimund Renkert <raimund.renkert@intevation.de> parents: 1761 diff changeset	76 country_code,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	77 date_issue,
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	78 reference_code,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	79 water_level,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	80 date_info,
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1776 diff changeset	81 source_organization,
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1776 diff changeset	82 staging_done
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	83 ) VALUES (
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	84 ($1, $2, $3, $4, $5),
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	85 $6,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	86 $7,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	87 $8,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	88 $9,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	89 $10,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	90 $11,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	91 $12,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	92 $13,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	93 $14,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	94 true
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	95 )
2849 b8972e4671fa Preserve old predictions on raw gauge measurement import. Sascha Wilde <wilde@intevation.de> parents: 2245 diff changeset	96 ON CONFLICT DO NOTHING
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	97 RETURNING 1
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	98 `
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	99
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	100 insertGPSQL = `
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	101 INSERT INTO waterway.gauge_predictions (
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	102 location,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	103 measure_date,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	104 sender,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	105 language_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	106 country_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	107 date_issue,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	108 reference_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	109 water_level,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	110 conf_interval,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	111 date_info,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	112 source_organization
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	113 ) VALUES(
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	114 ($1, $2, $3, $4, $5),
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	115 $6,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	116 $7,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	117 $8,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	118 $9,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	119 $10,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	120 $11,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	121 $12,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	122 $13,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	123 $14,
3278 831193935739 Avoid storing of NtS messages other than water level Tom Gottfried <tom@intevation.de> parents: 3277 diff changeset	124 $15
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	125 )
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	126 ON CONFLICT DO NOTHING
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	127 RETURNING 1
1978 0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	128 `
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	129 )
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	130
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	131 type gmJobCreator struct{}
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	132
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	133 func init() {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	134 RegisterJobCreator(GMJobKind, gmJobCreator{})
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	135 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	136
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	137 func (gmJobCreator) Description() string { return "gauge measurements" }
1646 a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1641 diff changeset	138
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	139 func (gmJobCreator) Create() Job { return new(GaugeMeasurement) }
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	140
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3176 diff changeset	141 func (gmJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3176 diff changeset	142 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3176 diff changeset	143 {"gauge_measurements"},
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3176 diff changeset	144 {"gauges"},
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	145 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	146 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	147
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	148 func (gmJobCreator) AutoAccept() bool { return true }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	149
5035 56c589f7435d Enhance comments on implementations of StageDone() Tom Gottfried <tom@intevation.de> parents: 5034 diff changeset	150 // StageDone is a NOP for gauge measurements imports.
5034 59a99655f34d Added feedback support for StageDone. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4852 diff changeset	151 func (gmJobCreator) StageDone(context.Context, *sql.Tx, int64, Feedback) error {
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	152 return nil
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	153 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	154
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	155 // CleanUp of a gauge measurement import is a NOP.
1676 4407ecaa2192 Imports: Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1664 diff changeset	156 func (*GaugeMeasurement) CleanUp() error { return nil }
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	157
4845 8c46b845b406 Fix comment Tom Gottfried <tom@intevation.de> parents: 4813 diff changeset	158 // Do executes the actual import.
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	159 func (gm *GaugeMeasurement) Do(
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	160 ctx context.Context,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	161 importID int64,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	162 conn *sql.Conn,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	163 feedback Feedback,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	164 ) (interface{}, error) {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	165
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	166 fetch := func() ([]*nts.RIS_Message_Type, error) {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	167 client := nts.NewINtSMessageService(gm.URL, gm.Insecure, nil)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	168
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	169 mt := nts.Message_type_typeWRM
4504 b7e3d42fd9e7 When fetching gauge measurements look 1day into past and future. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4197 diff changeset	170 now := time.Now()
4505 30c499f996d7 Simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4504 diff changeset	171 dis := []*nts.Date_pair{{
4504 b7e3d42fd9e7 When fetching gauge measurements look 1day into past and future. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4197 diff changeset	172 Date_start: nts.Date{Time: now.AddDate(0, 0, -1)},
b7e3d42fd9e7 When fetching gauge measurements look 1day into past and future. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4197 diff changeset	173 Date_end: nts.Date{Time: now.AddDate(0, 0, +1)},
4505 30c499f996d7 Simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4504 diff changeset	174 }}
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	175
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	176 req := &nts.Get_messages_query{
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	177 Message_type: &mt,
3709 d4a8d084eab4 import: Use dates_issue parameter in gm imports. Raimund Renkert <raimund.renkert@intevation.de> parents: 3583 diff changeset	178 Dates_issue: dis,
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	179 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	180
3582 317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	181 const maxTries = 3
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	182
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	183 tries := 0
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	184
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	185 again:
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	186 resp, err := client.Get_messages(req)
3582 317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	187
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	188 if err != nil {
3582 317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	189 if t, ok := err.(interface{ Timeout() bool }); ok && t.Timeout() && tries < maxTries {
5490 5f47eeea988d Use own logging package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5035 diff changeset	190 log.Warnln("NtS SOAP request timed out. Trying again.")
3582 317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	191 tries++
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	192 goto again
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	193 }
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	194 return nil, fmt.Errorf(
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	195 "Error requesting NtS service: %v", err)
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	196 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	197
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	198 result := resp.Result_message
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	199 if result == nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	200 for _, e := range resp.Result_error {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	201 if e != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	202 feedback.Error("Error code: %s", *e)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	203 } else {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	204 feedback.Error("Unknown error")
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	205 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	206 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	207 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	208 return result, nil
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	209 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	210
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	211 return storeGaugeMeasurements(
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	212 ctx,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	213 importID,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	214 fetch,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	215 conn,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	216 feedback,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	217 )
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	218 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	219
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	220 func loadGauges(ctx context.Context, conn *sql.Conn) ([]string, error) {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	221
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	222 rows, err := conn.QueryContext(ctx, listGaugesSQL)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	223 if err != nil {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	224 return nil, err
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	225 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	226 defer rows.Close()
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	227
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	228 var gauges []string
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	229
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	230 for rows.Next() {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	231 var g models.Isrs
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	232 if err = rows.Scan(
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	233 &g.CountryCode,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	234 &g.LoCode,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	235 &g.FairwaySection,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	236 &g.Orc,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	237 &g.Hectometre,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	238 ); err != nil {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	239 return nil, err
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	240 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	241 gauges = append(gauges, g.String())
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	242 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	243
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	244 if err = rows.Err(); err != nil {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	245 return nil, err
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	246 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	247
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	248 if len(gauges) == 0 {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	249 return nil, UnchangedError(
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	250 "No gauges for which measurements can be imported in database")
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	251 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	252
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	253 sort.Strings(gauges)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	254
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	255 return gauges, nil
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	256 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	257
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	258 func storeGaugeMeasurements(
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	259 ctx context.Context,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	260 importID int64,
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	261 fetch func() ([]*nts.RIS_Message_Type, error),
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	262 conn *sql.Conn,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	263 feedback Feedback,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	264 ) (interface{}, error) {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	265
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	266 start := time.Now()
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	267
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	268 // Get gauges from database, for which user is allowed to import data
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	269 gauges, err := loadGauges(ctx, conn)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	270 if err != nil {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	271 return nil, err
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	272 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	273
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	274 gids, err := doForGM(ctx, gauges, fetch, conn, feedback)
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	275 if err != nil {
1761 0145537d5474 Gauge measument import: Die with an error if something went wrong. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1754 diff changeset	276 return nil, err
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	277 }
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	278
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	279 if len(gids) == 0 {
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	280 return nil, UnchangedError("No new gauge measurements found")
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	281 }
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	282
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	283 feedback.Info(
3176 1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import Tom Gottfried <tom@intevation.de> parents: 2849 diff changeset	284 "Importing gauge measurements took %s", time.Since(start))
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	285
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	286 // TODO: needs to be filled more useful.
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	287 summary := struct {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	288 GaugeMeasuremets []string `json:"gaugeMeasurements"`
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	289 }{
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	290 GaugeMeasuremets: gids,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	291 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	292 return &summary, err
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	293 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	294
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	295 // rescale returns a scaling function to bring the unit all to cm.
3264 9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	296 func rescale(unit string) (func(*float32), error) {
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	297
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	298 var scale float32
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	299
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	300 switch strings.ToLower(unit) {
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	301 case "mm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	302 scale = 0.1
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	303 case "cm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	304 scale = 1.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	305 case "dm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	306 scale = 10.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	307 case "m":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	308 scale = 100.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	309 case "hm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	310 scale = 10000.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	311 case "km":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	312 scale = 100000.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	313 default:
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	314 return nil, fmt.Errorf("unknown unit '%s'", unit)
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	315 }
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	316
3264 9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	317 fn := func(x *float32) {
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	318 if x != nil {
3266 3dee5cf16a58 Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3264 diff changeset	319 x = scale
3264 9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	320 }
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	321 }
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	322 return fn, nil
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	323 }
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	324
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	325 type gmLog struct {
5529 f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	326 gid string
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	327 unknown bool
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	328 assumedZPG bool
5529 f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	329 ignoredMeasureCodes []string
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	330 rescaleErrors []string
5531 84f0c53e6ffe Completed GM logging aggregation. TODO: cleanup and dedup. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5530 diff changeset	331 missingValues []string
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	332 assumedCM int
5530 3ccbb6162112 WIP: More on GM import aggregation. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5529 diff changeset	333 badValues int
5531 84f0c53e6ffe Completed GM logging aggregation. TODO: cleanup and dedup. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5530 diff changeset	334 measurements int
84f0c53e6ffe Completed GM logging aggregation. TODO: cleanup and dedup. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5530 diff changeset	335 predictions int
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	336 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	337
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	338 type gmLogs []*gmLog
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	339
5529 f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	340 func extend(haystack []string, needle string) []string {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	341 for _, straw := range haystack {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	342 if straw == needle {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	343 return haystack
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	344 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	345 }
5529 f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	346 return append(haystack, needle)
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	347 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	348
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	349 func (gl *gmLog) addRescaleError(err error) {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	350 gl.rescaleErrors = extend(gl.rescaleErrors, err.Error())
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	351 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	352
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	353 func (gl *gmLog) ignoreMeasureCode(mc nts.Measure_code_enum) {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	354 gl.ignoredMeasureCodes = extend(gl.ignoredMeasureCodes, string(mc))
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	355 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	356
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	357 func (gls gmLogs) find(gid string) *gmLog {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	358 for _, gl := range gls {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	359 if gl.gid == gid {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	360 return gl
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	361 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	362 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	363 return nil
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	364 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	365
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	366 func (gls gmLogs) logging(feedback Feedback) {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	367
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	368 gls.logBool(
5551 e9ef27c75e5c Fixed minor typo. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5547 diff changeset	369 (*gmLog).getUnknown,
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	370 "Cannot find following gauges: ",
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	371 feedback.Warn)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	372
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	373 gls.logBool(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	374 (*gmLog).getAssumedZPG,
5537 2e5e51288a6c Fixed typo in generated logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5533 diff changeset	375 "'Reference_code' not specified. Assuming 'ZPG': ",
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	376 feedback.Warn)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	377
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	378 gls.logInt(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	379 (*gmLog).getAssumedCM,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	380 "'Unit' not specified. Assuming 'cm': ",
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	381 feedback.Warn)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	382
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	383 gls.logInt(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	384 (*gmLog).getBadValues,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	385 "Ignored measurements with value -99999: ",
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	386 feedback.Warn)
5531 84f0c53e6ffe Completed GM logging aggregation. TODO: cleanup and dedup. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5530 diff changeset	387
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	388 gls.logString(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	389 (*gmLog).getMissingValues,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	390 "Missing mandatory values: ",
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	391 feedback.Warn)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	392
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	393 gls.logString(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	394 (*gmLog).getRescaleErrors,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	395 "Cannot convert units: ",
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	396 feedback.Error)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	397
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	398 gls.logString(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	399 (*gmLog).getRescaleErrors,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	400 "Ignored measure codes: ",
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	401 feedback.Warn)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	402
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	403 gls.logInt(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	404 (*gmLog).getPredictions,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	405 "New predictions: ",
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	406 feedback.Info)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	407
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	408 gls.logInt(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	409 (*gmLog).getMeasurements,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	410 "New measurements: ",
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	411 feedback.Info)
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	412
0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	413 gls.logBool(
0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	414 (*gmLog).nothingChanged,
0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	415 "No changes for: ",
0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	416 feedback.Info)
5531 84f0c53e6ffe Completed GM logging aggregation. TODO: cleanup and dedup. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5530 diff changeset	417 }
84f0c53e6ffe Completed GM logging aggregation. TODO: cleanup and dedup. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5530 diff changeset	418
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	419 func (gl *gmLog) getAssumedZPG() bool { return gl.assumedZPG }
5551 e9ef27c75e5c Fixed minor typo. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5547 diff changeset	420 func (gl *gmLog) getUnknown() bool { return gl.unknown }
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	421 func (gl *gmLog) getIgnoredMeasureCodes() []string { return gl.ignoredMeasureCodes }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	422 func (gl *gmLog) getRescaleErrors() []string { return gl.rescaleErrors }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	423 func (gl *gmLog) getMissingValues() []string { return gl.missingValues }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	424 func (gl *gmLog) getAssumedCM() int { return gl.assumedCM }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	425 func (gl *gmLog) getBadValues() int { return gl.badValues }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	426 func (gl *gmLog) getPredictions() int { return gl.predictions }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	427 func (gl *gmLog) getMeasurements() int { return gl.measurements }
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	428 func (gl *gmLog) nothingChanged() bool { return gl.measurements == 0 && gl.predictions == 0 }
5531 84f0c53e6ffe Completed GM logging aggregation. TODO: cleanup and dedup. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5530 diff changeset	429
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	430 func (gls gmLogs) logBool(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	431 access func(*gmLog) bool,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	432 header string,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	433 log func(string, ...interface{}),
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	434 ) {
5530 3ccbb6162112 WIP: More on GM import aggregation. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5529 diff changeset	435 var sb strings.Builder
3ccbb6162112 WIP: More on GM import aggregation. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5529 diff changeset	436 for _, gl := range gls {
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	437 if access(gl) {
5530 3ccbb6162112 WIP: More on GM import aggregation. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5529 diff changeset	438 if sb.Len() == 0 {
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	439 sb.WriteString(header)
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	440 } else {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	441 sb.WriteString(", ")
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	442 }
5529 f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	443 sb.WriteString(gl.gid)
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	444 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	445 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	446 if sb.Len() > 0 {
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	447 log(sb.String())
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	448 }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	449 }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	450
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	451 func (gls gmLogs) logInt(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	452 access func(*gmLog) int,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	453 header string,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	454 log func(string, ...interface{}),
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	455 ) {
5545 a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	456 gs := make(gmLogs, 0, len(gls))
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	457 for _, g := range gls {
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	458 if access(g) > 0 {
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	459 gs = append(gs, g)
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	460 }
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	461 }
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	462
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	463 if len(gs) == 0 {
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	464 return
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	465 }
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	466
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	467 sort.SliceStable(gs, func(i, j int) bool {
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	468 return access(gs[i]) < access(gs[j])
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	469 })
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	470
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	471 var sb strings.Builder
5545 a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	472 var last int
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	473
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	474 for _, g := range gs {
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	475 if c := access(g); c != last {
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	476 if sb.Len() == 0 {
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	477 sb.WriteString(header)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	478 } else {
5545 a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	479 sb.WriteString("); ")
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	480 }
5545 a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	481 sb.WriteString(strconv.Itoa(c))
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	482 sb.WriteString(" (")
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	483 last = c
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	484 } else {
a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	485 sb.WriteString(", ")
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	486 }
5545 a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	487 sb.WriteString(g.gid)
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	488 }
5545 a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	489
5547 f8c370af9e6f Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5545 diff changeset	490 sb.WriteByte(')')
5545 a4fa1bf9b208 Aggregate to 'bla: 1 (baz); 2 (foo, bar)' instead of 'bla: foo (2), baz (1), bar (2)'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5537 diff changeset	491 log(sb.String())
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	492 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	493
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	494 func (gls gmLogs) logString(
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	495 access func(*gmLog) []string,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	496 header string,
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	497 log func(string, ...interface{}),
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	498 ) {
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	499 var sb strings.Builder
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	500 for _, gl := range gls {
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	501 if s := access(gl); len(s) > 0 {
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	502 if sb.Len() == 0 {
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	503 sb.WriteString(header)
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	504 } else {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	505 sb.WriteString(", ")
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	506 }
5547 f8c370af9e6f Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5545 diff changeset	507 sb.WriteString(gl.gid)
f8c370af9e6f Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5545 diff changeset	508 sb.WriteString(" (")
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	509 for i, v := range s {
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	510 if i > 0 {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	511 sb.WriteString("; ")
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	512 }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	513 sb.WriteString(v)
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	514 }
7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	515 sb.WriteByte(')')
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	516 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	517 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	518 if sb.Len() > 0 {
5532 7e21a9e6353d Cleaned up and simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5531 diff changeset	519 log(sb.String())
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	520 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	521 }
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	522
5529 f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	523 // logFinder is a helper to search recently used logs
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	524 // or create a new one if no log for a given gauge
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	525 // existed before.
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	526 func logFinder(logs gmLogs) func(string) gmLog {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	527 var lastLog *gmLog
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	528 return func(gid string) *gmLog {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	529 if lastLog != nil && lastLog.gid == gid {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	530 return lastLog
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	531 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	532 if ll := logs.find(gid); ll != nil {
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	533 lastLog = ll
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	534 return ll
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	535 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	536 lastLog = &gmLog{gid: gid}
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	537 logs = append(logs, lastLog)
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	538 return lastLog
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	539 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	540 }
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	541
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	542 func doForGM(
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	543 ctx context.Context,
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	544 gauges []string,
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	545 fetch func() ([]*nts.RIS_Message_Type, error),
3176 1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import Tom Gottfried <tom@intevation.de> parents: 2849 diff changeset	546 conn *sql.Conn,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	547 feedback Feedback,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	548 ) ([]string, error) {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	549
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	550 insertGPStmt, err := conn.PrepareContext(ctx, insertGPSQL)
2234 9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	551 if err != nil {
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	552 return nil, err
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	553 }
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	554 defer insertGPStmt.Close()
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	555
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	556 insertGMStmt, err := conn.PrepareContext(ctx, insertGMSQL)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	557 if err != nil {
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	558 return nil, err
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	559 }
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	560 defer insertGMStmt.Close()
2234 9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	561
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	562 // lookup to see if data can be imported for gauge
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	563 isKnown := func(s string) bool {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	564 idx := sort.SearchStrings(gauges, s)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	565 return idx < len(gauges) && gauges[idx] == s
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	566 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	567
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	568 result, err := fetch()
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	569 if err != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	570 return nil, err
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	571 }
1981 e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	572
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	573 var gids []string
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	574
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	575 // To prevent spamming the log actual logging
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	576 // is defered to be presented in an aggregated way.
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	577 var logs gmLogs
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	578 defer func() { logs.logging(feedback) }()
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	579
5529 f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	580 findLog := logFinder(&logs)
f95f268a83bd WIP: Continued work on aggregating GM logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5528 diff changeset	581
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	582 for _, msg := range result {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	583 for _, wrm := range msg.Wrm {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	584 curr := string(*wrm.Geo_object.Id)
3718 7431c6680b90 import: Trim whitespaces in ISRS code and improved log messsage. Raimund Renkert <raimund.renkert@intevation.de> parents: 3709 diff changeset	585 curr = strings.TrimSpace(curr)
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	586 currIsrs, err := models.IsrsFromString(curr)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	587 if err != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	588 feedback.Warn("Invalid ISRS code %v", err)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	589 continue
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	590 }
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	591 logger := findLog(curr)
5528 133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	592 gids = append(gids, curr)
133dc5b3076a WIP: Started to log GM imports in a more agregated way. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	593
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	594 if !isKnown(curr) {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	595 logger.unknown = true
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	596 continue
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	597 }
1981 e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	598
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	599 var referenceCode string
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	600 if wrm.Reference_code == nil {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	601 logger.assumedZPG = true
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	602 referenceCode = "ZPG"
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	603 } else {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	604 referenceCode = string(*wrm.Reference_code)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	605 }
2849 b8972e4671fa Preserve old predictions on raw gauge measurement import. Sascha Wilde <wilde@intevation.de> parents: 2245 diff changeset	606
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	607 for _, measure := range wrm.Measure {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	608 var unit string
3723 9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	609 if *measure.Measure_code != nts.Measure_code_enumWAL {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	610 logger.ignoreMeasureCode(*measure.Measure_code)
3723 9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	611 continue
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	612 }
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	613 if measure.Unit == nil {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	614 logger.assumedCM++
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	615 unit = "cm"
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	616 } else {
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	617 unit = string(*measure.Unit)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	618 }
4813 0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	619
0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	620 if measure.Value == nil {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	621 logger.missingValues = append(
0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	622 logger.missingValues,
5531 84f0c53e6ffe Completed GM logging aggregation. TODO: cleanup and dedup. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5530 diff changeset	623 measure.Measuredate.Time.Format(time.RFC3339))
4813 0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	624 continue
0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	625 }
0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	626
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	627 convert, err := rescale(unit)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	628 if err != nil {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	629 logger.addRescaleError(err)
4848 8584197232ec Handle unknown units gracefully Tom Gottfried <tom@intevation.de> parents: 4845 diff changeset	630 continue
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	631 }
3264 9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	632 convert(measure.Value)
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	633 convert(measure.Value_min)
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	634 convert(measure.Value_max)
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	635
3847 d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	636 // -99999 is used by some gauges to signal an error
4813 0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	637 if *measure.Value == -99999 {
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	638 logger.badValues++
3847 d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	639 continue
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	640 }
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	641
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	642 var dummy int
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	643 if measure.Predicted {
3723 9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	644 confInterval := pgtype.Numrange{
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	645 Lower: pgtype.Numeric{Status: pgtype.Null},
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	646 Upper: pgtype.Numeric{Status: pgtype.Null},
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	647 LowerType: pgtype.Inclusive,
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	648 UpperType: pgtype.Inclusive,
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	649 Status: pgtype.Null,
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	650 }
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	651 if measure.Value_min != nil && measure.Value_max != nil {
3723 9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	652 valueMin := pgtype.Numeric{Status: pgtype.Null}
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	653 valueMax := pgtype.Numeric{Status: pgtype.Null}
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	654 valueMin.Set(measure.Value_min)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	655 valueMax.Set(measure.Value_max)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	656 confInterval = pgtype.Numrange{
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	657 Lower: valueMin,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	658 Upper: valueMax,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	659 LowerType: pgtype.Inclusive,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	660 UpperType: pgtype.Inclusive,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	661 Status: pgtype.Present,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	662 }
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	663 }
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	664 err = insertGPStmt.QueryRowContext(
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	665 ctx,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	666 currIsrs.CountryCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	667 currIsrs.LoCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	668 currIsrs.FairwaySection,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	669 currIsrs.Orc,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	670 currIsrs.Hectometre,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	671 measure.Measuredate.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	672 msg.Identification.From,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	673 msg.Identification.Language_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	674 msg.Identification.Country_code,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	675 msg.Identification.Date_issue.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	676 referenceCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	677 measure.Value,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	678 &confInterval,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	679 msg.Identification.Date_issue.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	680 msg.Identification.Originator,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	681 ).Scan(&dummy)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	682 switch {
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	683 case err == sql.ErrNoRows:
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	684 // thats expected, nothing to do
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	685 case err != nil:
4197 5d7ce7f926eb Log fatal errors as errors, not warnings, in imports Tom Gottfried <tom@intevation.de> parents: 4158 diff changeset	686 feedback.Error(pgxutils.ReadableError{Err: err}.Error())
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	687 default:
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	688 logger.predictions++
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	689 }
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	690 } else {
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	691 err = insertGMStmt.QueryRowContext(
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	692 ctx,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	693 currIsrs.CountryCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	694 currIsrs.LoCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	695 currIsrs.FairwaySection,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	696 currIsrs.Orc,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	697 currIsrs.Hectometre,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	698 measure.Measuredate.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	699 msg.Identification.From,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	700 msg.Identification.Language_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	701 msg.Identification.Country_code,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	702 msg.Identification.Date_issue.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	703 referenceCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	704 measure.Value,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	705 msg.Identification.Date_issue.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	706 msg.Identification.Originator,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	707 ).Scan(&dummy)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	708 switch {
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	709 case err == sql.ErrNoRows:
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	710 // thats expected, nothing to do
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	711 case err != nil:
4197 5d7ce7f926eb Log fatal errors as errors, not warnings, in imports Tom Gottfried <tom@intevation.de> parents: 4158 diff changeset	712 feedback.Error(pgxutils.ReadableError{Err: err}.Error())
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	713 default:
5533 0a2326eda3a5 Improved readability a bit. Fixed a defer bug. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5532 diff changeset	714 logger.measurements++
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	715 }
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	716 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	717 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	718 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	719 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	720 return gids, nil
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	721 }

Mercurial > gemma

annotate pkg/imports/gm.go @ 5591:0011f50cf216 surveysperbottleneckid