gemma: pkg/imports/gm.go annotate

annotate pkg/imports/gm.go @ 5520:05db984d3db1

Improve performance of bottleneck area calculation Avoid buffer calculations by replacing them with simple distance comparisons and calculate the boundary of the result geometry only once per iteration. In some edge cases with very large numbers of iterations, this reduced the runtime of a bottleneck import by a factor of more than twenty.

author	Tom Gottfried <tom@intevation.de>
date	Thu, 21 Oct 2021 19:50:39 +0200
parents	5f47eeea988d
children	133dc5b3076a

rev	line source
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	3 //
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	6 //
3176 1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import Tom Gottfried <tom@intevation.de> parents: 2849 diff changeset	7 // Copyright (C) 2018, 2019 by via donau
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	10 //
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	11 // Author(s):
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	12 // * Raimund Renkert <raimund.renkert@intevation.de>
2245 7d784840a9a7 Uploaded gauge measurements: Initial commit. TODO: Add endpoint. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2244 diff changeset	13 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
3176 1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import Tom Gottfried <tom@intevation.de> parents: 2849 diff changeset	14 // * Tom Gottfried <tom.gottfried@intevation.de>
1694 4a2fad8f57de Imports: Resolved golint issues unrelated to exported symbols commenting. Sascha L. Teichmann <teichmann@intevation.de> parents: 1688 diff changeset	15
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	16 package imports
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	17
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	18 import (
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	19 "context"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	20 "database/sql"
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	21 "fmt"
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	22 "sort"
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	23 "strings"
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	24 "time"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	25
5490 5f47eeea988d Use own logging package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5035 diff changeset	26 "github.com/jackc/pgx/pgtype"
5f47eeea988d Use own logging package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5035 diff changeset	27
5f47eeea988d Use own logging package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5035 diff changeset	28 "gemma.intevation.de/gemma/pkg/log"
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	29 "gemma.intevation.de/gemma/pkg/models"
4062 6c760abcff0e Move handling of PostgreSQL errors to own package Tom Gottfried <tom@intevation.de> parents: 4058 diff changeset	30 "gemma.intevation.de/gemma/pkg/pgxutils"
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	31 "gemma.intevation.de/gemma/pkg/soap/nts"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	32 )
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	33
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	34 // GaugeMeasurement is an import job to import
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	35 // gauges measurement data from a NtS SOAP service.
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	36 type GaugeMeasurement struct {
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	37 // URL is the URL of the SOAP service.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	38 URL string `json:"url"`
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	39 // Insecure indicates if HTTPS traffic
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	40 // should validate certificates or not.
ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	41 Insecure bool `json:"insecure"`
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	42 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	43
4852 046a07a33b19 Fixed the golint issues of the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4848 diff changeset	44 // Description gives a short info about relevant facts of this import.
4798 ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4505 diff changeset	45 func (gm *GaugeMeasurement) Description() (string, error) {
4799 f32d086b5dbf Removed the mechanical touch of the last commit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4798 diff changeset	46 return gm.URL, nil
4798 ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4505 diff changeset	47 }
ca6a5f722471 Added Description method to most imports. Sascha Wilde <wilde@intevation.de> parents: 4505 diff changeset	48
1696 ad5e1cddaa09 Imports: Resolved the remaining golint issues with this package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	49 // GMJobKind is the import queue type identifier.
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	50 const GMJobKind JobKind = "gm"
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	51
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	52 const (
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	53 listGaugesSQL = `
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	54 SELECT
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	55 (location).country_code,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	56 (location).locode,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	57 (location).fairway_section,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	58 (location).orc,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	59 (location).hectometre
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	60 FROM waterway.gauges
4158 5466562cca60 Remove utility function with possibly bad performance impact Tom Gottfried <tom@intevation.de> parents: 4147 diff changeset	61 WHERE (location).country_code = (
5466562cca60 Remove utility function with possibly bad performance impact Tom Gottfried <tom@intevation.de> parents: 4147 diff changeset	62 SELECT country FROM users.list_users WHERE username = current_user)
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	63 OR pg_has_role('sys_admin', 'MEMBER')
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	64 `
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	65
2849 b8972e4671fa Preserve old predictions on raw gauge measurement import. Sascha Wilde <wilde@intevation.de> parents: 2245 diff changeset	66 // Note: we do not expect corrections of data through this service. So
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	67 // any constraint conflicts are triggered by redundant data which
2849 b8972e4671fa Preserve old predictions on raw gauge measurement import. Sascha Wilde <wilde@intevation.de> parents: 2245 diff changeset	68 // can be dropped.
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	69 insertGMSQL = `
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	70 INSERT INTO waterway.gauge_measurements (
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	71 location,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	72 measure_date,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	73 sender,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	74 language_code,
1766 e51db2a365a2 Insert country code in gm import. Raimund Renkert <raimund.renkert@intevation.de> parents: 1761 diff changeset	75 country_code,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	76 date_issue,
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	77 reference_code,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	78 water_level,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	79 date_info,
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1776 diff changeset	80 source_organization,
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1776 diff changeset	81 staging_done
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	82 ) VALUES (
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	83 ($1, $2, $3, $4, $5),
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	84 $6,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	85 $7,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	86 $8,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	87 $9,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	88 $10,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	89 $11,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	90 $12,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	91 $13,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	92 $14,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	93 true
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	94 )
2849 b8972e4671fa Preserve old predictions on raw gauge measurement import. Sascha Wilde <wilde@intevation.de> parents: 2245 diff changeset	95 ON CONFLICT DO NOTHING
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	96 RETURNING 1
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	97 `
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	98
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	99 insertGPSQL = `
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	100 INSERT INTO waterway.gauge_predictions (
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3301 diff changeset	101 location,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	102 measure_date,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	103 sender,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	104 language_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	105 country_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	106 date_issue,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	107 reference_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	108 water_level,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	109 conf_interval,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	110 date_info,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	111 source_organization
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	112 ) VALUES(
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	113 ($1, $2, $3, $4, $5),
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	114 $6,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	115 $7,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	116 $8,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	117 $9,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	118 $10,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	119 $11,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	120 $12,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	121 $13,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	122 $14,
3278 831193935739 Avoid storing of NtS messages other than water level Tom Gottfried <tom@intevation.de> parents: 3277 diff changeset	123 $15
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	124 )
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	125 ON CONFLICT DO NOTHING
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	126 RETURNING 1
1978 0a8fa6893181 Import: Upsert gauge measurement data to avoid unique contraint violation. Raimund Renkert <raimund.renkert@intevation.de> parents: 1954 diff changeset	127 `
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	128 )
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	129
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	130 type gmJobCreator struct{}
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	131
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	132 func init() {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	133 RegisterJobCreator(GMJobKind, gmJobCreator{})
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	134 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	135
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	136 func (gmJobCreator) Description() string { return "gauge measurements" }
1646 a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1641 diff changeset	137
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	138 func (gmJobCreator) Create() Job { return new(GaugeMeasurement) }
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	139
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3176 diff changeset	140 func (gmJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3176 diff changeset	141 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3176 diff changeset	142 {"gauge_measurements"},
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3176 diff changeset	143 {"gauges"},
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	144 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	145 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	146
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	147 func (gmJobCreator) AutoAccept() bool { return true }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1696 diff changeset	148
5035 56c589f7435d Enhance comments on implementations of StageDone() Tom Gottfried <tom@intevation.de> parents: 5034 diff changeset	149 // StageDone is a NOP for gauge measurements imports.
5034 59a99655f34d Added feedback support for StageDone. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4852 diff changeset	150 func (gmJobCreator) StageDone(context.Context, *sql.Tx, int64, Feedback) error {
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	151 return nil
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	152 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	153
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	154 // CleanUp of a gauge measurement import is a NOP.
1676 4407ecaa2192 Imports: Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1664 diff changeset	155 func (*GaugeMeasurement) CleanUp() error { return nil }
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	156
4845 8c46b845b406 Fix comment Tom Gottfried <tom@intevation.de> parents: 4813 diff changeset	157 // Do executes the actual import.
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	158 func (gm *GaugeMeasurement) Do(
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	159 ctx context.Context,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	160 importID int64,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	161 conn *sql.Conn,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	162 feedback Feedback,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	163 ) (interface{}, error) {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	164
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	165 fetch := func() ([]*nts.RIS_Message_Type, error) {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	166 client := nts.NewINtSMessageService(gm.URL, gm.Insecure, nil)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	167
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	168 mt := nts.Message_type_typeWRM
4504 b7e3d42fd9e7 When fetching gauge measurements look 1day into past and future. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4197 diff changeset	169 now := time.Now()
4505 30c499f996d7 Simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4504 diff changeset	170 dis := []*nts.Date_pair{{
4504 b7e3d42fd9e7 When fetching gauge measurements look 1day into past and future. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4197 diff changeset	171 Date_start: nts.Date{Time: now.AddDate(0, 0, -1)},
b7e3d42fd9e7 When fetching gauge measurements look 1day into past and future. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4197 diff changeset	172 Date_end: nts.Date{Time: now.AddDate(0, 0, +1)},
4505 30c499f996d7 Simplified code. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4504 diff changeset	173 }}
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	174
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	175 req := &nts.Get_messages_query{
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	176 Message_type: &mt,
3709 d4a8d084eab4 import: Use dates_issue parameter in gm imports. Raimund Renkert <raimund.renkert@intevation.de> parents: 3583 diff changeset	177 Dates_issue: dis,
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	178 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	179
3582 317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	180 const maxTries = 3
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	181
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	182 tries := 0
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	183
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	184 again:
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	185 resp, err := client.Get_messages(req)
3582 317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	186
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	187 if err != nil {
3582 317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	188 if t, ok := err.(interface{ Timeout() bool }); ok && t.Timeout() && tries < maxTries {
5490 5f47eeea988d Use own logging package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5035 diff changeset	189 log.Warnln("NtS SOAP request timed out. Trying again.")
3582 317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	190 tries++
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	191 goto again
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	192 }
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	193 return nil, fmt.Errorf(
317d176ef38c GM import: Try three times when timeouts happen on SOAP service. Increase SOAP timeout to 3 minutes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3580 diff changeset	194 "Error requesting NtS service: %v", err)
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	195 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	196
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	197 result := resp.Result_message
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	198 if result == nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	199 for _, e := range resp.Result_error {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	200 if e != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	201 feedback.Error("Error code: %s", *e)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	202 } else {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	203 feedback.Error("Unknown error")
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	204 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	205 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	206 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	207 return result, nil
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	208 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	209
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	210 return storeGaugeMeasurements(
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	211 ctx,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	212 importID,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	213 fetch,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	214 conn,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	215 feedback,
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	216 )
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	217 }
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	218
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	219 func loadGauges(ctx context.Context, conn *sql.Conn) ([]string, error) {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	220
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	221 rows, err := conn.QueryContext(ctx, listGaugesSQL)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	222 if err != nil {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	223 return nil, err
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	224 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	225 defer rows.Close()
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	226
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	227 var gauges []string
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	228
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	229 for rows.Next() {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	230 var g models.Isrs
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	231 if err = rows.Scan(
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	232 &g.CountryCode,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	233 &g.LoCode,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	234 &g.FairwaySection,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	235 &g.Orc,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	236 &g.Hectometre,
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	237 ); err != nil {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	238 return nil, err
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	239 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	240 gauges = append(gauges, g.String())
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	241 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	242
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	243 if err = rows.Err(); err != nil {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	244 return nil, err
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	245 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	246
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	247 if len(gauges) == 0 {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	248 return nil, UnchangedError(
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	249 "No gauges for which measurements can be imported in database")
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	250 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	251
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	252 sort.Strings(gauges)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	253
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	254 return gauges, nil
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	255 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	256
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	257 func storeGaugeMeasurements(
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	258 ctx context.Context,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	259 importID int64,
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	260 fetch func() ([]*nts.RIS_Message_Type, error),
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	261 conn *sql.Conn,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	262 feedback Feedback,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	263 ) (interface{}, error) {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	264
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	265 start := time.Now()
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	266
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	267 // Get gauges from database, for which user is allowed to import data
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	268 gauges, err := loadGauges(ctx, conn)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	269 if err != nil {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	270 return nil, err
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	271 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	272
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	273 gids, err := doForGM(ctx, gauges, fetch, conn, feedback)
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	274 if err != nil {
1761 0145537d5474 Gauge measument import: Die with an error if something went wrong. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1754 diff changeset	275 return nil, err
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	276 }
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	277
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	278 if len(gids) == 0 {
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	279 return nil, UnchangedError("No new gauge measurements found")
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	280 }
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	281
137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	282 feedback.Info(
3176 1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import Tom Gottfried <tom@intevation.de> parents: 2849 diff changeset	283 "Importing gauge measurements took %s", time.Since(start))
2233 137addc77b1b Gauge measurement imports: Do all database ops in one transaction. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	284
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	285 // TODO: needs to be filled more useful.
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	286 summary := struct {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	287 GaugeMeasuremets []string `json:"gaugeMeasurements"`
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	288 }{
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	289 GaugeMeasuremets: gids,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	290 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	291 return &summary, err
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	292 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	293
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	294 // rescale returns a scaling function to bring the unit all to cm.
3264 9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	295 func rescale(unit string) (func(*float32), error) {
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	296
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	297 var scale float32
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	298
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	299 switch strings.ToLower(unit) {
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	300 case "mm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	301 scale = 0.1
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	302 case "cm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	303 scale = 1.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	304 case "dm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	305 scale = 10.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	306 case "m":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	307 scale = 100.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	308 case "hm":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	309 scale = 10000.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	310 case "km":
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	311 scale = 100000.0
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	312 default:
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	313 return nil, fmt.Errorf("unknown unit '%s'", unit)
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	314 }
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	315
3264 9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	316 fn := func(x *float32) {
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	317 if x != nil {
3266 3dee5cf16a58 Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3264 diff changeset	318 x = scale
3264 9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	319 }
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	320 }
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	321 return fn, nil
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	322 }
70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	323
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	324 func doForGM(
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	325 ctx context.Context,
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	326 gauges []string,
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	327 fetch func() ([]*nts.RIS_Message_Type, error),
3176 1cb6676d1510 Handle failing INSERTs gracefully during gauge measurements import Tom Gottfried <tom@intevation.de> parents: 2849 diff changeset	328 conn *sql.Conn,
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	329 feedback Feedback,
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	330 ) ([]string, error) {
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	331
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	332 insertGPStmt, err := conn.PrepareContext(ctx, insertGPSQL)
2234 9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	333 if err != nil {
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	334 return nil, err
9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	335 }
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	336 defer insertGPStmt.Close()
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	337
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	338 insertGMStmt, err := conn.PrepareContext(ctx, insertGMSQL)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	339 if err != nil {
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	340 return nil, err
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	341 }
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	342 defer insertGMStmt.Close()
2234 9b2f8e94671e Gauge measurement import: Moved preparation of insert statement out of gauge iteration loop. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2233 diff changeset	343
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	344 // lookup to see if data can be imported for gauge
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	345 isKnown := func(s string) bool {
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	346 idx := sort.SearchStrings(gauges, s)
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	347 return idx < len(gauges) && gauges[idx] == s
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	348 }
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	349
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	350 result, err := fetch()
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	351 if err != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	352 return nil, err
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	353 }
1981 e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	354
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	355 var gids []string
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	356 for _, msg := range result {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	357 for _, wrm := range msg.Wrm {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	358 curr := string(*wrm.Geo_object.Id)
3718 7431c6680b90 import: Trim whitespaces in ISRS code and improved log messsage. Raimund Renkert <raimund.renkert@intevation.de> parents: 3709 diff changeset	359 curr = strings.TrimSpace(curr)
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	360 currIsrs, err := models.IsrsFromString(curr)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	361 if err != nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	362 feedback.Warn("Invalid ISRS code %v", err)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	363 continue
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	364 }
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	365 feedback.Info("Found measurements/predictions for %s", curr)
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	366 if !isKnown(curr) {
3718 7431c6680b90 import: Trim whitespaces in ISRS code and improved log messsage. Raimund Renkert <raimund.renkert@intevation.de> parents: 3709 diff changeset	367 feedback.Warn("Cannot find gauge %q for import", curr)
3301 6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	368 continue
6514b943654e Re-enable checking of gauge availability Tom Gottfried <tom@intevation.de> parents: 3284 diff changeset	369 }
1981 e89368aec538 Import: Make one request per gauge for gauge measurement data. Raimund Renkert <raimund.renkert@intevation.de> parents: 1978 diff changeset	370
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	371 var referenceCode string
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	372 if wrm.Reference_code == nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	373 feedback.Info("'Reference_code' not specified. Assuming 'ZPG'")
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	374 referenceCode = "ZPG"
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	375 } else {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	376 referenceCode = string(*wrm.Reference_code)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	377 }
2849 b8972e4671fa Preserve old predictions on raw gauge measurement import. Sascha Wilde <wilde@intevation.de> parents: 2245 diff changeset	378
3847 d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	379 badValue := 0
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	380 newM, newP := 0, 0
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	381 for _, measure := range wrm.Measure {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	382 var unit string
3723 9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	383 if *measure.Measure_code != nts.Measure_code_enumWAL {
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	384 feedback.Warn("Ignored message with measure_code %s",
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	385 *measure.Measure_code)
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	386 continue
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	387 }
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	388 if measure.Unit == nil {
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	389 feedback.Info("'Unit' not specified. Assuming 'cm'")
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	390 unit = "cm"
1772 70c4dc694d61 Gauge measurement import: Added a rescale function that brings all numerical values into cm. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1766 diff changeset	391 } else {
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	392 unit = string(*measure.Unit)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	393 }
4813 0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	394
0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	395 if measure.Value == nil {
0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	396 feedback.Warn("Missing mandatory value at %s. Ignored (bad service)",
0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	397 measure.Measuredate.Format(time.RFC3339))
0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	398 continue
0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	399 }
0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	400
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	401 convert, err := rescale(unit)
786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	402 if err != nil {
4848 8584197232ec Handle unknown units gracefully Tom Gottfried <tom@intevation.de> parents: 4845 diff changeset	403 feedback.Error(err.Error())
8584197232ec Handle unknown units gracefully Tom Gottfried <tom@intevation.de> parents: 4845 diff changeset	404 continue
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	405 }
3264 9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	406 convert(measure.Value)
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	407 convert(measure.Value_min)
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	408 convert(measure.Value_max)
9ae43313b463 Handle some possibly missing elements in NtS response Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	409
3847 d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	410 // -99999 is used by some gauges to signal an error
4813 0644c2b3af54 Check for mandatory measurement value in GM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4812 diff changeset	411 if *measure.Value == -99999 {
3847 d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	412 badValue++
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	413 continue
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	414 }
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	415
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	416 var dummy int
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	417 if measure.Predicted {
3723 9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	418 confInterval := pgtype.Numrange{
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	419 Lower: pgtype.Numeric{Status: pgtype.Null},
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	420 Upper: pgtype.Numeric{Status: pgtype.Null},
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	421 LowerType: pgtype.Inclusive,
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	422 UpperType: pgtype.Inclusive,
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	423 Status: pgtype.Null,
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	424 }
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	425 if measure.Value_min != nil && measure.Value_max != nil {
3723 9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	426 valueMin := pgtype.Numeric{Status: pgtype.Null}
9759355d7b90 import: Fixed data initialization for database rows. Raimund Renkert <raimund.renkert@intevation.de> parents: 3718 diff changeset	427 valueMax := pgtype.Numeric{Status: pgtype.Null}
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	428 valueMin.Set(measure.Value_min)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	429 valueMax.Set(measure.Value_max)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	430 confInterval = pgtype.Numrange{
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	431 Lower: valueMin,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	432 Upper: valueMax,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	433 LowerType: pgtype.Inclusive,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	434 UpperType: pgtype.Inclusive,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	435 Status: pgtype.Present,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	436 }
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	437 }
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	438 err = insertGPStmt.QueryRowContext(
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	439 ctx,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	440 currIsrs.CountryCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	441 currIsrs.LoCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	442 currIsrs.FairwaySection,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	443 currIsrs.Orc,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	444 currIsrs.Hectometre,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	445 measure.Measuredate.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	446 msg.Identification.From,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	447 msg.Identification.Language_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	448 msg.Identification.Country_code,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	449 msg.Identification.Date_issue.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	450 referenceCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	451 measure.Value,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	452 &confInterval,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	453 msg.Identification.Date_issue.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	454 msg.Identification.Originator,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	455 ).Scan(&dummy)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	456 switch {
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	457 case err == sql.ErrNoRows:
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	458 // thats expected, nothing to do
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	459 case err != nil:
4197 5d7ce7f926eb Log fatal errors as errors, not warnings, in imports Tom Gottfried <tom@intevation.de> parents: 4158 diff changeset	460 feedback.Error(pgxutils.ReadableError{Err: err}.Error())
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	461 default:
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	462 newP++
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	463 }
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	464 } else {
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	465 err = insertGMStmt.QueryRowContext(
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	466 ctx,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	467 currIsrs.CountryCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	468 currIsrs.LoCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	469 currIsrs.FairwaySection,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	470 currIsrs.Orc,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	471 currIsrs.Hectometre,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	472 measure.Measuredate.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	473 msg.Identification.From,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	474 msg.Identification.Language_code,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	475 msg.Identification.Country_code,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	476 msg.Identification.Date_issue.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	477 referenceCode,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	478 measure.Value,
3580 d38b20ccb6f9 GM import: Be more graceful when accepting measure/issue dates. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3533 diff changeset	479 msg.Identification.Date_issue.Time,
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	480 msg.Identification.Originator,
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	481 ).Scan(&dummy)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	482 switch {
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	483 case err == sql.ErrNoRows:
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	484 // thats expected, nothing to do
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	485 case err != nil:
4197 5d7ce7f926eb Log fatal errors as errors, not warnings, in imports Tom Gottfried <tom@intevation.de> parents: 4158 diff changeset	486 feedback.Error(pgxutils.ReadableError{Err: err}.Error())
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	487 default:
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	488 newM++
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	489 }
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	490 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	491 }
3847 d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	492 if badValue > 0 {
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	493 feedback.Warn("Ignored %d measurements with value -99999",
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	494 badValue)
d7b9d5c0ebad gm import: Ignore (and warn for) water levels of -99999. Sascha Wilde <wilde@intevation.de> parents: 3723 diff changeset	495 }
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	496 feedback.Info("Inserted %d measurements for %s",
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	497 newM, curr)
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	498 feedback.Info("Inserted %d predictions for %s",
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3266 diff changeset	499 newP, curr)
2242 786c3fb7efe1 Gauge measurements import: Re-factored to be re-usable for upcoming uploaded gauge measurements. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2238 diff changeset	500 gids = append(gids, curr)
1637 dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	501 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	502 }
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	503 return gids, nil
dd31be75ce6d Implemented gauge measurement import. Raimund Renkert <raimund.renkert@intevation.de> parents: diff changeset	504 }

Mercurial > gemma

annotate pkg/imports/gm.go @ 5520:05db984d3db1