gemma: pkg/imports/agm.go annotate

annotate pkg/imports/agm.go @ 3389:45a629a3a8b8

Fix constraints on relationship between gauges and measurements/predictions measure_date is the time of measurement, not date_issue. There can be only one measurement at a time for a gauge location, independently of gauge versions with differing validity. The 'measure_date' of a prediction can well be beyond the validity of the gauge, since a new gauge version might appear within the time range of the forecast. There can be only one prediction at a time per date of issue, independently of gauge versions with differing validity.

author	Tom Gottfried <tom@intevation.de>
date	Wed, 22 May 2019 16:44:13 +0200
parents	ec6163c6687d
children	45483dd0d801

rev	line source
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	7 // Copyright (C) 2018 by via donau
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	13 // * Sascha Wilde <wilde@intevation.de>
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	14
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15 package imports
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17 import (
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	18 "bufio"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	19 "context"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	20 "database/sql"
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	21 "encoding/csv"
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	22 "encoding/json"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	23 "fmt"
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	24 "io"
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	25 "math"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	26 "os"
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	27 "path/filepath"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	28 "strconv"
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	29 "strings"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	30 "time"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	31
1938 206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	32 "gemma.intevation.de/gemma/pkg/misc"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	33 "gemma.intevation.de/gemma/pkg/models"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	34 )
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	35
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	36 type ApprovedGaugeMeasurements struct {
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	37 Dir string `json:"dir"`
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	38 Originator string `json:"originator"`
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	39 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	40
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	41 // GMAPJobKind is the unique name of an approved gauge measurements import job.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	42 const AGMJobKind JobKind = "agm"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	43
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	44 type agmJobCreator struct{}
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	45
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	46 func init() {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	47 RegisterJobCreator(AGMJobKind, agmJobCreator{})
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	48 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	49
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1743 diff changeset	50 func (agmJobCreator) AutoAccept() bool { return false }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1743 diff changeset	51
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	52 func (agmJobCreator) Description() string {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	53 return "approved gauge measurements"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	54 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	55
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2173 diff changeset	56 func (agmJobCreator) Create() Job { return new(ApprovedGaugeMeasurements) }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	57
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	58 func (agmJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	59 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	60 {"gauge_measurements"},
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	61 {"gauges"},
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	62 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	63 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	64
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	65 const (
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	66 // delete the old and keep the new measures.
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	67 agmStageDoneDeleteSQL = `
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	68 WITH staged AS (
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	69 SELECT key
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1938 diff changeset	70 FROM import.track_imports
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	71 WHERE import_id = $1 AND
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	72 relation = 'waterway.gauge_measurements'::regclass
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	73 ),
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	74 to_delete AS (
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	75 SELECT o.id AS id
1781 0a53c33bc7b2 Fixed broken SQL statement (once again). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1780 diff changeset	76 FROM waterway.gauge_measurements o
0a53c33bc7b2 Fixed broken SQL statement (once again). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1780 diff changeset	77 JOIN waterway.gauge_measurements n
3389 45a629a3a8b8 Fix constraints on relationship between gauges and measurements/predictions Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	78 USING (location, measure_date)
2850 6d4f361c36e8 Do not delete predictions on import of agm. Sascha Wilde <wilde@intevation.de> parents: 2562 diff changeset	79 WHERE n.id IN (SELECT key FROM staged)
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	80 AND o.id NOT IN (SELECT key FROM staged)
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	81 )
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	82 DELETE FROM waterway.gauge_measurements WHERE id IN (SELECT id from to_delete)`
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	83
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	84 agmStageDoneSQL = `
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	85 UPDATE waterway.gauge_measurements SET staging_done = true
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	86 WHERE id IN (
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1938 diff changeset	87 SELECT key FROM import.track_imports
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	88 WHERE import_id = $1 AND
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	89 relation = 'waterway.gauge_measurements'::regclass)`
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	90 )
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	91
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	92 func (agmJobCreator) StageDone(
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	93 ctx context.Context,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	94 tx *sql.Tx,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	95 id int64,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	96 ) error {
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	97 _, err := tx.ExecContext(ctx, agmStageDoneDeleteSQL, id)
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	98 if err == nil {
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	99 _, err = tx.ExecContext(ctx, agmStageDoneSQL, id)
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	100 }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	101 return err
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	102 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	103
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	104 // CleanUp removes the folder containing the CSV file with the
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	105 // the approved gauge measurements.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	106 func (agm *ApprovedGaugeMeasurements) CleanUp() error {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	107 return os.RemoveAll(agm.Dir)
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	108 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	109
1938 206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	110 var guessDate = misc.TimeGuesser([]string{
206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	111 "02.01.2006 15:04",
206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	112 "2006-01-02T15:04:05-07:00",
206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	113 }).Guess
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	114
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	115 type timetz struct{ time.Time }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	116
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	117 func (ttz *timetz) MarshalJSON() ([]byte, error) {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	118 return json.Marshal(ttz.Time.Format("2006-01-02T15:04:05-07:00"))
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	119 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	120
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	121 type agmLine struct {
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	122 CountryCode string `json:"country-code"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	123 Sender string `json:"sender"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	124 LanguageCode string `json:"language-code"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	125 DateIssue timetz `json:"date-issue"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	126 ReferenceCode string `json:"reference-code"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	127 WaterLevel float64 `json:"water-level"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	128 DateInfo timetz `json:"date-info"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	129 SourceOrganization string `json:"source-organization"`
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	130 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	131
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	132 func (a agmLine) hasDiff(b agmLine) bool {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	133 const eps = 0.00001
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	134 return a.CountryCode != b.CountryCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	135 a.Sender != b.Sender \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	136 a.LanguageCode != b.LanguageCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	137 a.ReferenceCode != b.ReferenceCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	138 math.Abs(a.WaterLevel-b.WaterLevel) > eps \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	139 a.SourceOrganization != b.SourceOrganization
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	140 }
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	141
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	142 type agmSummaryEntry struct {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	143 FKGaugeID models.Isrs `json:"fk-gauge-id"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	144 MeasureDate timetz `json:"measure-date"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	145 Versions []*agmLine `json:"versions"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	146 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	147
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	148 const (
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	149 agmSelectSQL = `
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	150 SELECT
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	151 id,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	152 country_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	153 sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	154 language_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	155 date_issue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	156 reference_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	157 water_level,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	158 date_info,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	159 source_organization
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	160 FROM waterway.gauge_measurements
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	161 WHERE
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	162 location
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	163 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	164 AND measure_date = $6
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	165 AND staging_done
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	166 `
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	167
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	168 agmInsertSQL = `
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	169 INSERT INTO waterway.gauge_measurements (
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	170 location,
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	171 validity,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	172 measure_date,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	173 country_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	174 sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	175 language_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	176 date_issue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	177 reference_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	178 water_level,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	179 date_info,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	180 source_organization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	181 staging_done
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	182 ) VALUES (
2120 817cd8b89a86 Approved gauge measuments import: Give type hints when insertinf/querying isrs location codes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2105 diff changeset	183 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int),
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	184 (SELECT validity FROM waterway.gauges
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	185 WHERE location
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	186 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	187 AND validity @> CAST($6 AS timestamp with time zone)),
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	188 $6,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	189 $7,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	190 $8,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	191 $9,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	192 $10,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	193 $11,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	194 $12,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	195 $13,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	196 $14,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	197 false
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	198 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	199 RETURNING id`
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	200
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	201 agmGaugeCheckSQL = `
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	202 SELECT EXISTS(
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	203 SELECT 1 FROM waterway.gauges
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	204 WHERE location = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int))
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	205 `
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	206 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	207
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	208 // Do executes the actual approved gauge measurements import.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	209 func (agm *ApprovedGaugeMeasurements) Do(
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	210 ctx context.Context,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	211 importID int64,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	212 conn *sql.Conn,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	213 feedback Feedback,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	214 ) (interface{}, error) {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	215
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	216 start := time.Now()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	217
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	218 f, err := os.Open(filepath.Join(agm.Dir, "agm.csv"))
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	219 if err != nil {
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	220 return nil, err
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	221 }
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	222 defer f.Close()
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	223
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	224 r := csv.NewReader(bufio.NewReader(f))
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	225 r.Comma = ';'
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	226 r.ReuseRecord = true
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	227
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	228 headers, err := r.Read()
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	229 if err != nil {
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	230 return nil, err
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	231 }
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	232
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	233 var (
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	234 fkGaugeIDIdx = -1
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	235 measureDateIdx = -1
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	236 valueIdx = -1
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	237 )
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	238
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	239 headerFields := []struct {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	240 idx *int
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	241 name string
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	242 }{
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	243 {&fkGaugeIDIdx, "fk_gauge_id"},
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	244 {&measureDateIdx, "measure_date"},
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	245 {&valueIdx, "value"}, // "water_level",
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	246 }
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	247
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	248 nextHeader:
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	249 for i, f := range headers {
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	250 h := strings.Replace(strings.ToLower(
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	251 strings.TrimSpace(f)), " ", "_", -1)
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	252
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	253 for j := range headerFields {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	254 if headerFields[j].name == h {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	255 if *headerFields[j].idx != -1 {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	256 return nil, fmt.Errorf(
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	257 "There is more than one column namend '%s'", h)
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	258 }
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	259 *headerFields[j].idx = i
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	260 continue nextHeader
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	261 }
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	262 }
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	263 }
fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	264
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	265 var missing []string
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	266 for i := range headerFields {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	267 if headerFields[i].name != "unit" && *headerFields[i].idx == -1 {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	268 missing = append(missing, headerFields[i].name)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	269 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	270 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	271 if len(missing) > 0 {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	272 return nil, fmt.Errorf("Missing columns: %s", strings.Join(missing, ", "))
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	273 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	274
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	275 gaugeCheckStmt, err := conn.PrepareContext(ctx, agmGaugeCheckSQL)
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	276 if err != nil {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	277 return nil, err
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	278 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	279 defer gaugeCheckStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	280
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	281 selectStmt, err := conn.PrepareContext(ctx, agmSelectSQL)
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	282 if err != nil {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	283 return nil, err
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	284 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	285 defer selectStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	286
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	287 insertStmt, err := conn.PrepareContext(ctx, agmInsertSQL)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	288 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	289 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	290 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	291 defer insertStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	292
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	293 trackStmt, err := conn.PrepareContext(ctx, trackImportSQL)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	294 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	295 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	296 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	297 defer trackStmt.Close()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	298
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	299 entries := []*agmSummaryEntry{}
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	300
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	301 checkedGauges := map[models.Isrs]bool{}
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	302
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	303 lines:
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	304 for line, ignored := 1, 0; ; line++ {
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	305
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	306 row, err := r.Read()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	307 switch {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	308 case err == io.EOF \|\| len(row) == 0:
3218 c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	309 feedback.Info("Read %d entries in CSV file", line-1)
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	310 if ignored > 0 {
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	311 feedback.Info("%d entries ignored", ignored)
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	312 }
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	313 if ignored == line-1 {
8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	314 return nil, UnchangedError("No entries imported")
8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	315 }
3218 c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	316 feedback.Info("Imported %d entries with changes", len(entries))
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	317 break lines
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	318 case err != nil:
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	319 return nil, fmt.Errorf("CSV parsing failed: %v", err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	320 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	321
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	322 gids := row[fkGaugeIDIdx]
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	323 gid, err := models.IsrsFromString(gids)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	324 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	325 return nil, fmt.Errorf("Invalid ISRS code line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	326 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	327
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	328 if exists, found := checkedGauges[*gid]; found {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	329 if !exists {
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	330 // Just ignore the line since we have already warned
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	331 ignored++
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	332 continue lines
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	333 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	334 } else { // not found in gauge cache
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	335 if err := gaugeCheckStmt.QueryRowContext(
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	336 ctx,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	337 gid.CountryCode,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	338 gid.LoCode,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	339 gid.FairwaySection,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	340 gid.Orc,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	341 gid.Hectometre,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	342 ).Scan(&exists); err != nil {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	343 return nil, err
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	344 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	345 checkedGauges[*gid] = exists
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	346 if !exists {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	347 feedback.Warn("Ignoring data for unknown gauge %s", gid.String())
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	348 ignored++
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	349 continue lines
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	350 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	351 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	352
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	353 md, err := guessDate(row[measureDateIdx])
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	354 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	355 return nil, fmt.Errorf("Invalid 'measure_date' line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	356 }
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	357
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	358 var (
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	359 oldID int64
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	360 oldCountryCode string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	361 oldSender string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	362 oldLanguageCode string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	363 oldDateIssue time.Time
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	364 oldReferenceCode string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	365 oldValue float64
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	366 oldDateInfo time.Time
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	367 oldSourceOrganization string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	368 )
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	369
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	370 err = selectStmt.QueryRowContext(
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	371 ctx,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	372 gid.CountryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	373 gid.LoCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	374 gid.FairwaySection,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	375 gid.Orc,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	376 gid.Hectometre,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	377 md,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	378 ).Scan(
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	379 &oldID,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	380 &oldCountryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	381 &oldSender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	382 &oldLanguageCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	383 &oldDateIssue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	384 &oldReferenceCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	385 &oldValue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	386 &oldDateInfo,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	387 &oldSourceOrganization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	388 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	389
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	390 var newEntry bool
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	391 switch {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	392 case err == sql.ErrNoRows:
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	393 // Complete new one
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	394 newEntry = true
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	395 case err != nil:
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	396 return nil, err
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	397 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	398
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	399 newSender := agm.Originator
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	400 newCountryCode := gid.CountryCode
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	401 newLanguageCode := misc.CCtoLang[gid.CountryCode]
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	402 newDateIssue := time.Now()
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	403 newReferenceCode := "ZPG"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	404
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	405 value, err := strconv.ParseFloat(row[valueIdx], 32)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	406 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	407 return nil, fmt.Errorf("Invalid 'value' line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	408 }
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	409 newValue := value
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	410
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	411 newDateInfo := newDateIssue
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	412
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	413 newSourceOrganization := newSender
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	414
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	415 tx, err := conn.BeginTx(ctx, nil)
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	416 if err != nil {
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	417 return nil, err
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	418 }
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	419 defer tx.Rollback()
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	420
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	421 var newID int64
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	422
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	423 if err := tx.StmtContext(ctx, insertStmt).QueryRowContext(
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	424 ctx,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	425 gid.CountryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	426 gid.LoCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	427 gid.FairwaySection,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	428 gid.Orc,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	429 gid.Hectometre,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	430 md,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	431 newCountryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	432 newSender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	433 newLanguageCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	434 newDateIssue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	435 newReferenceCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	436 newValue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	437 newDateInfo,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	438 newSourceOrganization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	439 ).Scan(&newID); err != nil {
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	440 feedback.Warn(handleError(err).Error())
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	441 if err := tx.Rollback(); err != nil {
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	442 return nil, err
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	443 }
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	444 ignored++
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	445 continue
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	446 }
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	447 if _, err := tx.StmtContext(ctx, trackStmt).ExecContext(
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	448 ctx, importID, "waterway.gauge_measurements", newID,
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	449 ); err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	450 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	451 }
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	452 if err := tx.Commit(); err != nil {
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	453 return nil, fmt.Errorf("Commit failed: %v", err)
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	454 }
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	455
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	456 n := newAGMLine(
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	457 newCountryCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	458 newSender,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	459 newLanguageCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	460 newDateIssue,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	461 newReferenceCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	462 newValue,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	463 newDateInfo,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	464 newSourceOrganization,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	465 )
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	466
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	467 ase := &agmSummaryEntry{
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	468 FKGaugeID: *gid,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	469 MeasureDate: timetz{md},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	470 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	471
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	472 if newEntry {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	473 ase.Versions = []*agmLine{n}
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	474 } else {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	475 o := newAGMLine(
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	476 oldCountryCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	477 oldSender,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	478 oldLanguageCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	479 oldDateIssue,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	480 oldReferenceCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	481 oldValue,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	482 oldDateInfo,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	483 oldSourceOrganization,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	484 )
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	485 // Ignore if there is no diff.
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	486 if !n.hasDiff(o) {
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	487 continue
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	488 }
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	489 ase.Versions = []*agmLine{o, n}
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	490 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	491 entries = append(entries, ase)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	492 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	493
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	494 feedback.Info("Importing approved gauge measurements took %s",
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	495 time.Since(start))
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	496
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	497 return entries, nil
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	498 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	499
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	500 func newAGMLine(
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	501 countryCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	502 sender string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	503 languageCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	504 dateIssue time.Time,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	505 referenceCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	506 waterLevel float64,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	507 dateInfo time.Time,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	508 sourceOrganization string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	509 ) *agmLine {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	510 return &agmLine{
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	511 CountryCode: countryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	512 Sender: sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	513 LanguageCode: languageCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	514 DateIssue: timetz{dateIssue},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	515 ReferenceCode: referenceCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	516 WaterLevel: waterLevel,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	517 DateInfo: timetz{dateInfo},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	518 SourceOrganization: sourceOrganization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	519 }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	520 }

Mercurial > gemma

annotate pkg/imports/agm.go @ 3389:45a629a3a8b8