gemma: pkg/imports/agm.go annotate

annotate pkg/imports/agm.go @ 3277:232fc90e6ee2

Disentangle gauge measurements and predictions Representing both in one table has led to the necessity to make the distinction at many places such as statements, definitions of partial indexes and application code. At least in one place in the AGM import the distinction in application code was too late and measurements matching an approved measurement could have been missed.

author	Tom Gottfried <tom@intevation.de>
date	Wed, 15 May 2019 19:08:49 +0200
parents	56b297592c0a
children	831193935739

rev	line source
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	7 // Copyright (C) 2018 by via donau
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	13 // * Sascha Wilde <wilde@intevation.de>
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	14
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15 package imports
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17 import (
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	18 "bufio"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	19 "context"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	20 "database/sql"
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	21 "encoding/csv"
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	22 "encoding/json"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	23 "fmt"
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	24 "io"
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	25 "math"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	26 "os"
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	27 "path/filepath"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	28 "strconv"
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	29 "strings"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	30 "time"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	31
1938 206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	32 "gemma.intevation.de/gemma/pkg/misc"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	33 "gemma.intevation.de/gemma/pkg/models"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	34 )
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	35
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	36 type ApprovedGaugeMeasurements struct {
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	37 Dir string `json:"dir"`
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	38 Originator string `json:"originator"`
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	39 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	40
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	41 // GMAPJobKind is the unique name of an approved gauge measurements import job.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	42 const AGMJobKind JobKind = "agm"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	43
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	44 type agmJobCreator struct{}
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	45
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	46 func init() {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	47 RegisterJobCreator(AGMJobKind, agmJobCreator{})
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	48 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	49
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1743 diff changeset	50 func (agmJobCreator) AutoAccept() bool { return false }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1743 diff changeset	51
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	52 func (agmJobCreator) Description() string {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	53 return "approved gauge measurements"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	54 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	55
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2173 diff changeset	56 func (agmJobCreator) Create() Job { return new(ApprovedGaugeMeasurements) }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	57
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	58 func (agmJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	59 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	60 {"gauge_measurements"},
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	61 {"gauges"},
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	62 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	63 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	64
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	65 const (
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	66 // delete the old and keep the new measures.
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	67 agmStageDoneDeleteSQL = `
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	68 WITH staged AS (
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	69 SELECT key
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1938 diff changeset	70 FROM import.track_imports
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	71 WHERE import_id = $1 AND
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	72 relation = 'waterway.gauge_measurements'::regclass
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	73 ),
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	74 to_delete AS (
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	75 SELECT o.id AS id
1781 0a53c33bc7b2 Fixed broken SQL statement (once again). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1780 diff changeset	76 FROM waterway.gauge_measurements o
0a53c33bc7b2 Fixed broken SQL statement (once again). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1780 diff changeset	77 JOIN waterway.gauge_measurements n
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	78 ON n.fk_gauge_id = o.fk_gauge_id AND n.measure_date = o.measure_date
2850 6d4f361c36e8 Do not delete predictions on import of agm. Sascha Wilde <wilde@intevation.de> parents: 2562 diff changeset	79 WHERE n.id IN (SELECT key FROM staged)
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	80 AND o.id NOT IN (SELECT key FROM staged)
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	81 )
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	82 DELETE FROM waterway.gauge_measurements WHERE id IN (SELECT id from to_delete)`
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	83
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	84 agmStageDoneSQL = `
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	85 UPDATE waterway.gauge_measurements SET staging_done = true
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	86 WHERE id IN (
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1938 diff changeset	87 SELECT key FROM import.track_imports
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	88 WHERE import_id = $1 AND
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	89 relation = 'waterway.gauge_measurements'::regclass)`
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	90 )
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	91
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	92 func (agmJobCreator) StageDone(
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	93 ctx context.Context,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	94 tx *sql.Tx,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	95 id int64,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	96 ) error {
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	97 _, err := tx.ExecContext(ctx, agmStageDoneDeleteSQL, id)
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	98 if err == nil {
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	99 _, err = tx.ExecContext(ctx, agmStageDoneSQL, id)
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	100 }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	101 return err
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	102 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	103
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	104 // CleanUp removes the folder containing the CSV file with the
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	105 // the approved gauge measurements.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	106 func (agm *ApprovedGaugeMeasurements) CleanUp() error {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	107 return os.RemoveAll(agm.Dir)
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	108 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	109
1938 206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	110 var guessDate = misc.TimeGuesser([]string{
206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	111 "02.01.2006 15:04",
206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	112 "2006-01-02T15:04:05-07:00",
206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	113 }).Guess
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	114
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	115 type timetz struct{ time.Time }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	116
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	117 func (ttz *timetz) MarshalJSON() ([]byte, error) {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	118 return json.Marshal(ttz.Time.Format("2006-01-02T15:04:05-07:00"))
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	119 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	120
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	121 type agmLine struct {
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	122 CountryCode string `json:"country-code"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	123 Sender string `json:"sender"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	124 LanguageCode string `json:"language-code"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	125 DateIssue timetz `json:"date-issue"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	126 ReferenceCode string `json:"reference-code"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	127 WaterLevel float64 `json:"water-level"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	128 DateInfo timetz `json:"date-info"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	129 SourceOrganization string `json:"source-organization"`
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	130 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	131
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	132 func (a agmLine) hasDiff(b agmLine) bool {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	133 const eps = 0.00001
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	134 return a.CountryCode != b.CountryCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	135 a.Sender != b.Sender \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	136 a.LanguageCode != b.LanguageCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	137 a.ReferenceCode != b.ReferenceCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	138 math.Abs(a.WaterLevel-b.WaterLevel) > eps \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	139 a.SourceOrganization != b.SourceOrganization
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	140 }
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	141
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	142 type agmSummaryEntry struct {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	143 FKGaugeID models.Isrs `json:"fk-gauge-id"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	144 MeasureDate timetz `json:"measure-date"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	145 Versions []*agmLine `json:"versions"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	146 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	147
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	148 const (
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	149 agmSelectSQL = `
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	150 SELECT
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	151 id,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	152 country_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	153 sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	154 language_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	155 date_issue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	156 reference_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	157 water_level,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	158 date_info,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	159 source_organization
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	160 FROM waterway.gauge_measurements
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	161 WHERE
2120 817cd8b89a86 Approved gauge measuments import: Give type hints when insertinf/querying isrs location codes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2105 diff changeset	162 fk_gauge_id = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int) AND
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	163 measure_date = $6 AND staging_done`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	164
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	165 agmInsertSQL = `
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	166 INSERT INTO waterway.gauge_measurements (
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	167 fk_gauge_id,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	168 measure_date,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	169 country_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	170 sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	171 language_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	172 date_issue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	173 reference_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	174 water_level,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	175 date_info,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	176 source_organization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	177 is_waterlevel,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	178 staging_done
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	179 ) VALUES(
2120 817cd8b89a86 Approved gauge measuments import: Give type hints when insertinf/querying isrs location codes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2105 diff changeset	180 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int),
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	181 $6,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	182 $7,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	183 $8,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	184 $9,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	185 $10,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	186 $11,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	187 $12,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	188 $13,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	189 $14,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	190 true,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	191 false
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	192 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	193 RETURNING id`
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	194
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	195 agmGaugeCheckSQL = `
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	196 SELECT EXISTS(
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	197 SELECT 1 FROM waterway.gauges
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	198 WHERE location = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int))
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	199 `
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	200 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	201
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	202 // Do executes the actual approved gauge measurements import.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	203 func (agm *ApprovedGaugeMeasurements) Do(
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	204 ctx context.Context,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	205 importID int64,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	206 conn *sql.Conn,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	207 feedback Feedback,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	208 ) (interface{}, error) {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	209
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	210 start := time.Now()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	211
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	212 f, err := os.Open(filepath.Join(agm.Dir, "agm.csv"))
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	213 if err != nil {
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	214 return nil, err
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	215 }
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	216 defer f.Close()
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	217
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	218 r := csv.NewReader(bufio.NewReader(f))
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	219 r.Comma = ';'
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	220 r.ReuseRecord = true
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	221
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	222 headers, err := r.Read()
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	223 if err != nil {
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	224 return nil, err
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	225 }
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	226
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	227 var (
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	228 fkGaugeIDIdx = -1
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	229 measureDateIdx = -1
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	230 valueIdx = -1
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	231 )
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	232
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	233 headerFields := []struct {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	234 idx *int
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	235 name string
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	236 }{
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	237 {&fkGaugeIDIdx, "fk_gauge_id"},
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	238 {&measureDateIdx, "measure_date"},
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	239 {&valueIdx, "value"}, // "water_level",
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	240 }
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	241
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	242 nextHeader:
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	243 for i, f := range headers {
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	244 h := strings.Replace(strings.ToLower(
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	245 strings.TrimSpace(f)), " ", "_", -1)
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	246
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	247 for j := range headerFields {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	248 if headerFields[j].name == h {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	249 if *headerFields[j].idx != -1 {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	250 return nil, fmt.Errorf(
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	251 "There is more than one column namend '%s'", h)
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	252 }
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	253 *headerFields[j].idx = i
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	254 continue nextHeader
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	255 }
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	256 }
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	257 }
fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	258
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	259 var missing []string
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	260 for i := range headerFields {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	261 if headerFields[i].name != "unit" && *headerFields[i].idx == -1 {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	262 missing = append(missing, headerFields[i].name)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	263 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	264 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	265 if len(missing) > 0 {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	266 return nil, fmt.Errorf("Missing columns: %s", strings.Join(missing, ", "))
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	267 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	268
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	269 gaugeCheckStmt, err := conn.PrepareContext(ctx, agmGaugeCheckSQL)
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	270 if err != nil {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	271 return nil, err
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	272 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	273 defer gaugeCheckStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	274
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	275 selectStmt, err := conn.PrepareContext(ctx, agmSelectSQL)
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	276 if err != nil {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	277 return nil, err
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	278 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	279 defer selectStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	280
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	281 insertStmt, err := conn.PrepareContext(ctx, agmInsertSQL)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	282 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	283 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	284 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	285 defer insertStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	286
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	287 trackStmt, err := conn.PrepareContext(ctx, trackImportSQL)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	288 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	289 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	290 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	291 defer trackStmt.Close()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	292
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	293 entries := []*agmSummaryEntry{}
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	294
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	295 checkedGauges := map[models.Isrs]bool{}
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	296
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	297 lines:
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	298 for line, ignored := 1, 0; ; line++ {
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	299
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	300 row, err := r.Read()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	301 switch {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	302 case err == io.EOF \|\| len(row) == 0:
3218 c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	303 feedback.Info("Read %d entries in CSV file", line-1)
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	304 if ignored > 0 {
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	305 feedback.Info("%d entries ignored", ignored)
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	306 }
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	307 if ignored == line-1 {
8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	308 return nil, UnchangedError("No entries imported")
8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	309 }
3218 c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	310 feedback.Info("Imported %d entries with changes", len(entries))
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	311 break lines
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	312 case err != nil:
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	313 return nil, fmt.Errorf("CSV parsing failed: %v", err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	314 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	315
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	316 gids := row[fkGaugeIDIdx]
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	317 gid, err := models.IsrsFromString(gids)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	318 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	319 return nil, fmt.Errorf("Invalid ISRS code line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	320 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	321
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	322 if exists, found := checkedGauges[*gid]; found {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	323 if !exists {
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	324 // Just ignore the line since we have already warned
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	325 ignored++
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	326 continue lines
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	327 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	328 } else { // not found in gauge cache
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	329 if err := gaugeCheckStmt.QueryRowContext(
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	330 ctx,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	331 gid.CountryCode,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	332 gid.LoCode,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	333 gid.FairwaySection,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	334 gid.Orc,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	335 gid.Hectometre,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	336 ).Scan(&exists); err != nil {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	337 return nil, err
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	338 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	339 checkedGauges[*gid] = exists
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	340 if !exists {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	341 feedback.Warn("Ignoring data for unknown gauge %s", gid.String())
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	342 ignored++
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	343 continue lines
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	344 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	345 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	346
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	347 md, err := guessDate(row[measureDateIdx])
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	348 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	349 return nil, fmt.Errorf("Invalid 'measure_date' line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	350 }
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	351
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	352 var (
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	353 oldID int64
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	354 oldCountryCode string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	355 oldSender string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	356 oldLanguageCode string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	357 oldDateIssue time.Time
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	358 oldReferenceCode string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	359 oldValue float64
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	360 oldDateInfo time.Time
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	361 oldSourceOrganization string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	362 )
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	363
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	364 err = selectStmt.QueryRowContext(
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	365 ctx,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	366 gid.CountryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	367 gid.LoCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	368 gid.FairwaySection,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	369 gid.Orc,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	370 gid.Hectometre,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	371 md,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	372 ).Scan(
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	373 &oldID,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	374 &oldCountryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	375 &oldSender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	376 &oldLanguageCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	377 &oldDateIssue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	378 &oldReferenceCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	379 &oldValue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	380 &oldDateInfo,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	381 &oldSourceOrganization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	382 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	383
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	384 var newEntry bool
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	385 switch {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	386 case err == sql.ErrNoRows:
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	387 // Complete new one
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	388 newEntry = true
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	389 case err != nil:
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	390 return nil, err
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	391 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	392
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	393 newSender := agm.Originator
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	394 newCountryCode := gid.CountryCode
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	395 newLanguageCode := misc.CCtoLang[gid.CountryCode]
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	396 newDateIssue := time.Now()
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	397 newReferenceCode := "ZPG"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	398
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	399 value, err := strconv.ParseFloat(row[valueIdx], 32)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	400 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	401 return nil, fmt.Errorf("Invalid 'value' line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	402 }
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	403 newValue := value
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	404
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	405 newDateInfo := newDateIssue
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	406
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	407 newSourceOrganization := newSender
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	408
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	409 tx, err := conn.BeginTx(ctx, nil)
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	410 if err != nil {
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	411 return nil, err
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	412 }
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	413 defer tx.Rollback()
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	414
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	415 var newID int64
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	416
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	417 if err := tx.StmtContext(ctx, insertStmt).QueryRowContext(
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	418 ctx,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	419 gid.CountryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	420 gid.LoCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	421 gid.FairwaySection,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	422 gid.Orc,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	423 gid.Hectometre,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	424 md,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	425 newCountryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	426 newSender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	427 newLanguageCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	428 newDateIssue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	429 newReferenceCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	430 newValue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	431 newDateInfo,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	432 newSourceOrganization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	433 ).Scan(&newID); err != nil {
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	434 feedback.Warn(handleError(err).Error())
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	435 if err := tx.Rollback(); err != nil {
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	436 return nil, err
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	437 }
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	438 ignored++
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	439 continue
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	440 }
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	441 if _, err := tx.StmtContext(ctx, trackStmt).ExecContext(
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	442 ctx, importID, "waterway.gauge_measurements", newID,
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	443 ); err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	444 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	445 }
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	446 if err := tx.Commit(); err != nil {
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	447 return nil, fmt.Errorf("Commit failed: %v", err)
56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	448 }
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	449
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	450 n := newAGMLine(
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	451 newCountryCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	452 newSender,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	453 newLanguageCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	454 newDateIssue,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	455 newReferenceCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	456 newValue,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	457 newDateInfo,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	458 newSourceOrganization,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	459 )
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	460
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	461 ase := &agmSummaryEntry{
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	462 FKGaugeID: *gid,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	463 MeasureDate: timetz{md},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	464 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	465
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	466 if newEntry {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	467 ase.Versions = []*agmLine{n}
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	468 } else {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	469 o := newAGMLine(
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	470 oldCountryCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	471 oldSender,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	472 oldLanguageCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	473 oldDateIssue,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	474 oldReferenceCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	475 oldValue,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	476 oldDateInfo,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	477 oldSourceOrganization,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	478 )
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	479 // Ignore if there is no diff.
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	480 if !n.hasDiff(o) {
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	481 continue
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	482 }
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	483 ase.Versions = []*agmLine{o, n}
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	484 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	485 entries = append(entries, ase)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	486 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	487
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	488 feedback.Info("Importing approved gauge measurements took %s",
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	489 time.Since(start))
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	490
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	491 return entries, nil
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	492 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	493
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	494 func newAGMLine(
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	495 countryCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	496 sender string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	497 languageCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	498 dateIssue time.Time,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	499 referenceCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	500 waterLevel float64,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	501 dateInfo time.Time,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	502 sourceOrganization string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	503 ) *agmLine {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	504 return &agmLine{
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	505 CountryCode: countryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	506 Sender: sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	507 LanguageCode: languageCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	508 DateIssue: timetz{dateIssue},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	509 ReferenceCode: referenceCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	510 WaterLevel: waterLevel,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	511 DateInfo: timetz{dateInfo},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	512 SourceOrganization: sourceOrganization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	513 }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	514 }

Mercurial > gemma

annotate pkg/imports/agm.go @ 3277:232fc90e6ee2