gemma: pkg/imports/agm.go annotate

annotate pkg/imports/agm.go @ 3302:ec6163c6687d

'Historicise' gauges on import Gauge data sets will be updated or a new version will be inserted depending on temporal validity and a timestamp marking the last update in the RIS-Index of a data set. The trigger on date_info is removed because the value is actually an attribut coming from the RIS-Index. Gauge measurements and predictions are associated to the version with matching temporal validity. Bottlenecks are always associated to the actual version of the gauge, although this might change as soon as bottlenecks are 'historicised', too.

author	Tom Gottfried <tom@intevation.de>
date	Thu, 16 May 2019 18:41:43 +0200
parents	831193935739
children	45a629a3a8b8

rev	line source
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	7 // Copyright (C) 2018 by via donau
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	13 // * Sascha Wilde <wilde@intevation.de>
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	14
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15 package imports
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17 import (
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	18 "bufio"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	19 "context"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	20 "database/sql"
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	21 "encoding/csv"
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	22 "encoding/json"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	23 "fmt"
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	24 "io"
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	25 "math"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	26 "os"
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	27 "path/filepath"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	28 "strconv"
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	29 "strings"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	30 "time"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	31
1938 206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	32 "gemma.intevation.de/gemma/pkg/misc"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	33 "gemma.intevation.de/gemma/pkg/models"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	34 )
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	35
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	36 type ApprovedGaugeMeasurements struct {
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	37 Dir string `json:"dir"`
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	38 Originator string `json:"originator"`
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	39 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	40
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	41 // GMAPJobKind is the unique name of an approved gauge measurements import job.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	42 const AGMJobKind JobKind = "agm"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	43
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	44 type agmJobCreator struct{}
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	45
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	46 func init() {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	47 RegisterJobCreator(AGMJobKind, agmJobCreator{})
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	48 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	49
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1743 diff changeset	50 func (agmJobCreator) AutoAccept() bool { return false }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1743 diff changeset	51
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	52 func (agmJobCreator) Description() string {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	53 return "approved gauge measurements"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	54 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	55
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2173 diff changeset	56 func (agmJobCreator) Create() Job { return new(ApprovedGaugeMeasurements) }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	57
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	58 func (agmJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	59 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	60 {"gauge_measurements"},
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	61 {"gauges"},
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	62 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	63 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	64
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	65 const (
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	66 // delete the old and keep the new measures.
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	67 agmStageDoneDeleteSQL = `
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	68 WITH staged AS (
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	69 SELECT key
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1938 diff changeset	70 FROM import.track_imports
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	71 WHERE import_id = $1 AND
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	72 relation = 'waterway.gauge_measurements'::regclass
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	73 ),
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	74 to_delete AS (
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	75 SELECT o.id AS id
1781 0a53c33bc7b2 Fixed broken SQL statement (once again). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1780 diff changeset	76 FROM waterway.gauge_measurements o
0a53c33bc7b2 Fixed broken SQL statement (once again). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1780 diff changeset	77 JOIN waterway.gauge_measurements n
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	78 USING (location, validity, measure_date)
2850 6d4f361c36e8 Do not delete predictions on import of agm. Sascha Wilde <wilde@intevation.de> parents: 2562 diff changeset	79 WHERE n.id IN (SELECT key FROM staged)
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	80 AND o.id NOT IN (SELECT key FROM staged)
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	81 )
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	82 DELETE FROM waterway.gauge_measurements WHERE id IN (SELECT id from to_delete)`
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	83
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	84 agmStageDoneSQL = `
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	85 UPDATE waterway.gauge_measurements SET staging_done = true
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	86 WHERE id IN (
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1938 diff changeset	87 SELECT key FROM import.track_imports
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	88 WHERE import_id = $1 AND
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	89 relation = 'waterway.gauge_measurements'::regclass)`
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	90 )
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	91
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	92 func (agmJobCreator) StageDone(
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	93 ctx context.Context,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	94 tx *sql.Tx,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	95 id int64,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	96 ) error {
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	97 _, err := tx.ExecContext(ctx, agmStageDoneDeleteSQL, id)
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	98 if err == nil {
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	99 _, err = tx.ExecContext(ctx, agmStageDoneSQL, id)
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	100 }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	101 return err
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	102 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	103
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	104 // CleanUp removes the folder containing the CSV file with the
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	105 // the approved gauge measurements.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	106 func (agm *ApprovedGaugeMeasurements) CleanUp() error {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	107 return os.RemoveAll(agm.Dir)
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	108 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	109
1938 206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	110 var guessDate = misc.TimeGuesser([]string{
206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	111 "02.01.2006 15:04",
206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	112 "2006-01-02T15:04:05-07:00",
206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	113 }).Guess
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	114
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	115 type timetz struct{ time.Time }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	116
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	117 func (ttz *timetz) MarshalJSON() ([]byte, error) {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	118 return json.Marshal(ttz.Time.Format("2006-01-02T15:04:05-07:00"))
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	119 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	120
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	121 type agmLine struct {
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	122 CountryCode string `json:"country-code"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	123 Sender string `json:"sender"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	124 LanguageCode string `json:"language-code"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	125 DateIssue timetz `json:"date-issue"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	126 ReferenceCode string `json:"reference-code"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	127 WaterLevel float64 `json:"water-level"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	128 DateInfo timetz `json:"date-info"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	129 SourceOrganization string `json:"source-organization"`
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	130 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	131
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	132 func (a agmLine) hasDiff(b agmLine) bool {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	133 const eps = 0.00001
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	134 return a.CountryCode != b.CountryCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	135 a.Sender != b.Sender \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	136 a.LanguageCode != b.LanguageCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	137 a.ReferenceCode != b.ReferenceCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	138 math.Abs(a.WaterLevel-b.WaterLevel) > eps \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	139 a.SourceOrganization != b.SourceOrganization
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	140 }
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	141
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	142 type agmSummaryEntry struct {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	143 FKGaugeID models.Isrs `json:"fk-gauge-id"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	144 MeasureDate timetz `json:"measure-date"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	145 Versions []*agmLine `json:"versions"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	146 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	147
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	148 const (
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	149 agmSelectSQL = `
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	150 SELECT
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	151 id,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	152 country_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	153 sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	154 language_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	155 date_issue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	156 reference_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	157 water_level,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	158 date_info,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	159 source_organization
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	160 FROM waterway.gauge_measurements
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	161 WHERE
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	162 location
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	163 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	164 AND measure_date = $6
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	165 AND staging_done
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	166 `
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	167
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	168 agmInsertSQL = `
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	169 INSERT INTO waterway.gauge_measurements (
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	170 location,
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	171 validity,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	172 measure_date,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	173 country_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	174 sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	175 language_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	176 date_issue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	177 reference_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	178 water_level,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	179 date_info,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	180 source_organization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	181 staging_done
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	182 ) VALUES (
2120 817cd8b89a86 Approved gauge measuments import: Give type hints when insertinf/querying isrs location codes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2105 diff changeset	183 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int),
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	184 (SELECT validity FROM waterway.gauges
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	185 WHERE location
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	186 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	187 AND validity @> CAST($6 AS timestamp with time zone)),
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	188 $6,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	189 $7,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	190 $8,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	191 $9,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	192 $10,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	193 $11,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	194 $12,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	195 $13,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	196 $14,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	197 false
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	198 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	199 RETURNING id`
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	200
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	201 agmGaugeCheckSQL = `
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	202 SELECT EXISTS(
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	203 SELECT 1 FROM waterway.gauges
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	204 WHERE location = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int))
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	205 `
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	206 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	207
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	208 // Do executes the actual approved gauge measurements import.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	209 func (agm *ApprovedGaugeMeasurements) Do(
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	210 ctx context.Context,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	211 importID int64,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	212 conn *sql.Conn,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	213 feedback Feedback,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	214 ) (interface{}, error) {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	215
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	216 start := time.Now()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	217
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	218 f, err := os.Open(filepath.Join(agm.Dir, "agm.csv"))
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	219 if err != nil {
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	220 return nil, err
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	221 }
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	222 defer f.Close()
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	223
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	224 r := csv.NewReader(bufio.NewReader(f))
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	225 r.Comma = ';'
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	226 r.ReuseRecord = true
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	227
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	228 headers, err := r.Read()
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	229 if err != nil {
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	230 return nil, err
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	231 }
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	232
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	233 var (
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	234 fkGaugeIDIdx = -1
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	235 measureDateIdx = -1
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	236 valueIdx = -1
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	237 )
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	238
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	239 headerFields := []struct {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	240 idx *int
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	241 name string
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	242 }{
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	243 {&fkGaugeIDIdx, "fk_gauge_id"},
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	244 {&measureDateIdx, "measure_date"},
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	245 {&valueIdx, "value"}, // "water_level",
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	246 }
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	247
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	248 nextHeader:
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	249 for i, f := range headers {
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	250 h := strings.Replace(strings.ToLower(
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	251 strings.TrimSpace(f)), " ", "_", -1)
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	252
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	253 for j := range headerFields {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	254 if headerFields[j].name == h {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	255 if *headerFields[j].idx != -1 {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	256 return nil, fmt.Errorf(
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	257 "There is more than one column namend '%s'", h)
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	258 }
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	259 *headerFields[j].idx = i
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	260 continue nextHeader
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	261 }
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	262 }
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	263 }
fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	264
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	265 var missing []string
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	266 for i := range headerFields {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	267 if headerFields[i].name != "unit" && *headerFields[i].idx == -1 {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	268 missing = append(missing, headerFields[i].name)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	269 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	270 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	271 if len(missing) > 0 {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	272 return nil, fmt.Errorf("Missing columns: %s", strings.Join(missing, ", "))
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	273 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	274
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	275 gaugeCheckStmt, err := conn.PrepareContext(ctx, agmGaugeCheckSQL)
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	276 if err != nil {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	277 return nil, err
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	278 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	279 defer gaugeCheckStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	280
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	281 selectStmt, err := conn.PrepareContext(ctx, agmSelectSQL)
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	282 if err != nil {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	283 return nil, err
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	284 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	285 defer selectStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	286
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	287 insertStmt, err := conn.PrepareContext(ctx, agmInsertSQL)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	288 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	289 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	290 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	291 defer insertStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	292
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	293 trackStmt, err := conn.PrepareContext(ctx, trackImportSQL)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	294 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	295 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	296 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	297 defer trackStmt.Close()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	298
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	299 entries := []*agmSummaryEntry{}
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	300
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	301 checkedGauges := map[models.Isrs]bool{}
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	302
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	303 lines:
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	304 for line, ignored := 1, 0; ; line++ {
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	305
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	306 row, err := r.Read()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	307 switch {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	308 case err == io.EOF \|\| len(row) == 0:
3218 c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	309 feedback.Info("Read %d entries in CSV file", line-1)
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	310 if ignored > 0 {
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	311 feedback.Info("%d entries ignored", ignored)
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	312 }
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	313 if ignored == line-1 {
8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	314 return nil, UnchangedError("No entries imported")
8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	315 }
3218 c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	316 feedback.Info("Imported %d entries with changes", len(entries))
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	317 break lines
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	318 case err != nil:
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	319 return nil, fmt.Errorf("CSV parsing failed: %v", err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	320 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	321
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	322 gids := row[fkGaugeIDIdx]
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	323 gid, err := models.IsrsFromString(gids)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	324 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	325 return nil, fmt.Errorf("Invalid ISRS code line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	326 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	327
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	328 if exists, found := checkedGauges[*gid]; found {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	329 if !exists {
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	330 // Just ignore the line since we have already warned
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	331 ignored++
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	332 continue lines
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	333 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	334 } else { // not found in gauge cache
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	335 if err := gaugeCheckStmt.QueryRowContext(
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	336 ctx,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	337 gid.CountryCode,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	338 gid.LoCode,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	339 gid.FairwaySection,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	340 gid.Orc,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	341 gid.Hectometre,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	342 ).Scan(&exists); err != nil {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	343 return nil, err
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	344 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	345 checkedGauges[*gid] = exists
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	346 if !exists {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	347 feedback.Warn("Ignoring data for unknown gauge %s", gid.String())
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	348 ignored++
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	349 continue lines
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	350 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	351 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	352
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	353 md, err := guessDate(row[measureDateIdx])
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	354 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	355 return nil, fmt.Errorf("Invalid 'measure_date' line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	356 }
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	357
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	358 var (
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	359 oldID int64
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	360 oldCountryCode string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	361 oldSender string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	362 oldLanguageCode string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	363 oldDateIssue time.Time
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	364 oldReferenceCode string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	365 oldValue float64
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	366 oldDateInfo time.Time
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	367 oldSourceOrganization string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	368 )
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	369
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	370 err = selectStmt.QueryRowContext(
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	371 ctx,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	372 gid.CountryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	373 gid.LoCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	374 gid.FairwaySection,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	375 gid.Orc,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	376 gid.Hectometre,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	377 md,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	378 ).Scan(
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	379 &oldID,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	380 &oldCountryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	381 &oldSender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	382 &oldLanguageCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	383 &oldDateIssue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	384 &oldReferenceCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	385 &oldValue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	386 &oldDateInfo,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	387 &oldSourceOrganization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	388 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	389
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	390 var newEntry bool
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	391 switch {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	392 case err == sql.ErrNoRows:
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	393 // Complete new one
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	394 newEntry = true
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	395 case err != nil:
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	396 return nil, err
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	397 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	398
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	399 newSender := agm.Originator
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	400 newCountryCode := gid.CountryCode
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	401 newLanguageCode := misc.CCtoLang[gid.CountryCode]
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	402 newDateIssue := time.Now()
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	403 newReferenceCode := "ZPG"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	404
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	405 value, err := strconv.ParseFloat(row[valueIdx], 32)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	406 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	407 return nil, fmt.Errorf("Invalid 'value' line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	408 }
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	409 newValue := value
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	410
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	411 newDateInfo := newDateIssue
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	412
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	413 newSourceOrganization := newSender
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	414
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	415 tx, err := conn.BeginTx(ctx, nil)
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	416 if err != nil {
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	417 return nil, err
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	418 }
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	419 defer tx.Rollback()
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	420
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	421 var newID int64
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	422
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	423 if err := tx.StmtContext(ctx, insertStmt).QueryRowContext(
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	424 ctx,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	425 gid.CountryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	426 gid.LoCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	427 gid.FairwaySection,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	428 gid.Orc,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	429 gid.Hectometre,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	430 md,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	431 newCountryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	432 newSender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	433 newLanguageCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	434 newDateIssue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	435 newReferenceCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	436 newValue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	437 newDateInfo,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	438 newSourceOrganization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	439 ).Scan(&newID); err != nil {
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	440 feedback.Warn(handleError(err).Error())
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	441 if err := tx.Rollback(); err != nil {
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	442 return nil, err
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	443 }
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	444 ignored++
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	445 continue
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	446 }
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	447 if _, err := tx.StmtContext(ctx, trackStmt).ExecContext(
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	448 ctx, importID, "waterway.gauge_measurements", newID,
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	449 ); err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	450 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	451 }
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	452 if err := tx.Commit(); err != nil {
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	453 return nil, fmt.Errorf("Commit failed: %v", err)
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	454 }
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	455
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	456 n := newAGMLine(
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	457 newCountryCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	458 newSender,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	459 newLanguageCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	460 newDateIssue,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	461 newReferenceCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	462 newValue,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	463 newDateInfo,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	464 newSourceOrganization,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	465 )
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	466
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	467 ase := &agmSummaryEntry{
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	468 FKGaugeID: *gid,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	469 MeasureDate: timetz{md},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	470 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	471
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	472 if newEntry {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	473 ase.Versions = []*agmLine{n}
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	474 } else {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	475 o := newAGMLine(
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	476 oldCountryCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	477 oldSender,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	478 oldLanguageCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	479 oldDateIssue,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	480 oldReferenceCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	481 oldValue,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	482 oldDateInfo,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	483 oldSourceOrganization,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	484 )
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	485 // Ignore if there is no diff.
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	486 if !n.hasDiff(o) {
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	487 continue
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	488 }
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	489 ase.Versions = []*agmLine{o, n}
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	490 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	491 entries = append(entries, ase)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	492 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	493
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	494 feedback.Info("Importing approved gauge measurements took %s",
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	495 time.Since(start))
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	496
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	497 return entries, nil
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	498 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	499
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	500 func newAGMLine(
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	501 countryCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	502 sender string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	503 languageCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	504 dateIssue time.Time,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	505 referenceCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	506 waterLevel float64,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	507 dateInfo time.Time,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	508 sourceOrganization string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	509 ) *agmLine {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	510 return &agmLine{
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	511 CountryCode: countryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	512 Sender: sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	513 LanguageCode: languageCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	514 DateIssue: timetz{dateIssue},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	515 ReferenceCode: referenceCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	516 WaterLevel: waterLevel,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	517 DateInfo: timetz{dateInfo},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	518 SourceOrganization: sourceOrganization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	519 }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	520 }

Mercurial > gemma

annotate pkg/imports/agm.go @ 3302:ec6163c6687d