gemma: pkg/imports/agm.go annotate

annotate pkg/imports/agm.go @ 3533:8e083b271fca

Improve error messages if no matching gauge version found Avoid hitting the NOT NULL constraint of the referencing validity column in order to hit the foreign key constraint instead and emit an appropriate error message in all such cases.

author	Tom Gottfried <tom@intevation.de>
date	Wed, 29 May 2019 18:14:20 +0200
parents	6e748f31777a
children	baa51bb82364

rev	line source
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	7 // Copyright (C) 2018 by via donau
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	13 // * Sascha Wilde <wilde@intevation.de>
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	14
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15 package imports
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17 import (
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	18 "bufio"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	19 "context"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	20 "database/sql"
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	21 "encoding/csv"
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	22 "encoding/json"
3502 45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	23 "errors"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	24 "fmt"
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	25 "io"
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	26 "math"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	27 "os"
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	28 "path/filepath"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	29 "strconv"
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	30 "strings"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	31 "time"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	32
1938 206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	33 "gemma.intevation.de/gemma/pkg/misc"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	34 "gemma.intevation.de/gemma/pkg/models"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	35 )
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	36
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	37 type ApprovedGaugeMeasurements struct {
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	38 Dir string `json:"dir"`
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	39 Originator string `json:"originator"`
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	40 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	41
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	42 // GMAPJobKind is the unique name of an approved gauge measurements import job.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	43 const AGMJobKind JobKind = "agm"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	44
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	45 type agmJobCreator struct{}
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	46
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	47 func init() {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	48 RegisterJobCreator(AGMJobKind, agmJobCreator{})
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	49 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	50
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1743 diff changeset	51 func (agmJobCreator) AutoAccept() bool { return false }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1743 diff changeset	52
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	53 func (agmJobCreator) Description() string {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	54 return "approved gauge measurements"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	55 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	56
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2173 diff changeset	57 func (agmJobCreator) Create() Job { return new(ApprovedGaugeMeasurements) }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	58
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	59 func (agmJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	60 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	61 {"gauge_measurements"},
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	62 {"gauges"},
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	63 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	64 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	65
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	66 const (
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	67 // delete the old and keep the new measures.
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	68 agmStageDoneDeleteSQL = `
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	69 WITH staged AS (
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	70 SELECT key
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1938 diff changeset	71 FROM import.track_imports
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	72 WHERE import_id = $1 AND
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	73 relation = 'waterway.gauge_measurements'::regclass
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	74 ),
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	75 to_delete AS (
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	76 SELECT o.id AS id
1781 0a53c33bc7b2 Fixed broken SQL statement (once again). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1780 diff changeset	77 FROM waterway.gauge_measurements o
0a53c33bc7b2 Fixed broken SQL statement (once again). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1780 diff changeset	78 JOIN waterway.gauge_measurements n
3389 45a629a3a8b8 Fix constraints on relationship between gauges and measurements/predictions Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	79 USING (location, measure_date)
2850 6d4f361c36e8 Do not delete predictions on import of agm. Sascha Wilde <wilde@intevation.de> parents: 2562 diff changeset	80 WHERE n.id IN (SELECT key FROM staged)
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	81 AND o.id NOT IN (SELECT key FROM staged)
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	82 )
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	83 DELETE FROM waterway.gauge_measurements WHERE id IN (SELECT id from to_delete)`
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	84
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	85 agmStageDoneSQL = `
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	86 UPDATE waterway.gauge_measurements SET staging_done = true
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	87 WHERE id IN (
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1938 diff changeset	88 SELECT key FROM import.track_imports
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	89 WHERE import_id = $1 AND
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	90 relation = 'waterway.gauge_measurements'::regclass)`
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	91 )
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	92
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	93 func (agmJobCreator) StageDone(
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	94 ctx context.Context,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	95 tx *sql.Tx,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	96 id int64,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	97 ) error {
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	98 _, err := tx.ExecContext(ctx, agmStageDoneDeleteSQL, id)
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	99 if err == nil {
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	100 _, err = tx.ExecContext(ctx, agmStageDoneSQL, id)
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	101 }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	102 return err
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	103 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	104
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	105 // CleanUp removes the folder containing the CSV file with the
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	106 // the approved gauge measurements.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	107 func (agm *ApprovedGaugeMeasurements) CleanUp() error {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	108 return os.RemoveAll(agm.Dir)
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	109 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	110
1938 206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	111 var guessDate = misc.TimeGuesser([]string{
206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	112 "02.01.2006 15:04",
206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	113 "2006-01-02T15:04:05-07:00",
206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	114 }).Guess
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	115
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	116 type timetz struct{ time.Time }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	117
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	118 func (ttz *timetz) MarshalJSON() ([]byte, error) {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	119 return json.Marshal(ttz.Time.Format("2006-01-02T15:04:05-07:00"))
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	120 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	121
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	122 type agmLine struct {
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	123 CountryCode string `json:"country-code"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	124 Sender string `json:"sender"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	125 LanguageCode string `json:"language-code"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	126 DateIssue timetz `json:"date-issue"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	127 ReferenceCode string `json:"reference-code"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	128 WaterLevel float64 `json:"water-level"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	129 DateInfo timetz `json:"date-info"`
232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	130 SourceOrganization string `json:"source-organization"`
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	131 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	132
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	133 func (a agmLine) hasDiff(b agmLine) bool {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	134 const eps = 0.00001
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	135 return a.CountryCode != b.CountryCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	136 a.Sender != b.Sender \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	137 a.LanguageCode != b.LanguageCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	138 a.ReferenceCode != b.ReferenceCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	139 math.Abs(a.WaterLevel-b.WaterLevel) > eps \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	140 a.SourceOrganization != b.SourceOrganization
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	141 }
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	142
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	143 type agmSummaryEntry struct {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	144 FKGaugeID models.Isrs `json:"fk-gauge-id"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	145 MeasureDate timetz `json:"measure-date"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	146 Versions []*agmLine `json:"versions"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	147 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	148
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	149 const (
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	150 agmSelectSQL = `
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	151 SELECT
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	152 id,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	153 country_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	154 sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	155 language_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	156 date_issue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	157 reference_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	158 water_level,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	159 date_info,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	160 source_organization
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	161 FROM waterway.gauge_measurements
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	162 WHERE
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	163 location
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	164 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	165 AND measure_date = $6
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	166 AND staging_done
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	167 `
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	168
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	169 agmInsertSQL = `
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	170 INSERT INTO waterway.gauge_measurements (
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	171 location,
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	172 validity,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	173 measure_date,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	174 country_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	175 sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	176 language_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	177 date_issue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	178 reference_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	179 water_level,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	180 date_info,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	181 source_organization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	182 staging_done
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	183 ) VALUES (
2120 817cd8b89a86 Approved gauge measuments import: Give type hints when insertinf/querying isrs location codes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2105 diff changeset	184 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int),
3533 8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3504 diff changeset	185 COALESCE(
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3504 diff changeset	186 (SELECT validity FROM waterway.gauges
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3504 diff changeset	187 WHERE location
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3504 diff changeset	188 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3504 diff changeset	189 AND validity @> CAST($6 AS timestamp with time zone)),
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3504 diff changeset	190 tstzrange(NULL, NULL)),
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	191 $6,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	192 $7,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	193 $8,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	194 $9,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	195 $10,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	196 $11,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	197 $12,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	198 $13,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	199 $14,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	200 false
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	201 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	202 RETURNING id`
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	203
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	204 agmGaugeCheckSQL = `
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	205 SELECT EXISTS(
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	206 SELECT 1 FROM waterway.gauges
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	207 WHERE location = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int))
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	208 `
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	209 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	210
3502 45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	211 var errContinue = errors.New("continue")
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	212
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	213 // Do executes the actual approved gauge measurements import.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	214 func (agm *ApprovedGaugeMeasurements) Do(
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	215 ctx context.Context,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	216 importID int64,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	217 conn *sql.Conn,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	218 feedback Feedback,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	219 ) (interface{}, error) {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	220
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	221 start := time.Now()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	222
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	223 f, err := os.Open(filepath.Join(agm.Dir, "agm.csv"))
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	224 if err != nil {
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	225 return nil, err
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	226 }
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	227 defer f.Close()
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	228
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	229 r := csv.NewReader(bufio.NewReader(f))
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	230 r.Comma = ';'
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	231 r.ReuseRecord = true
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	232
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	233 headers, err := r.Read()
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	234 if err != nil {
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	235 return nil, err
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	236 }
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	237
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	238 var (
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	239 fkGaugeIDIdx = -1
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	240 measureDateIdx = -1
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	241 valueIdx = -1
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	242 )
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	243
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	244 headerFields := []struct {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	245 idx *int
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	246 name string
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	247 }{
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	248 {&fkGaugeIDIdx, "fk_gauge_id"},
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	249 {&measureDateIdx, "measure_date"},
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	250 {&valueIdx, "value"}, // "water_level",
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	251 }
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	252
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	253 nextHeader:
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	254 for i, f := range headers {
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	255 h := strings.Replace(strings.ToLower(
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	256 strings.TrimSpace(f)), " ", "_", -1)
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	257
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	258 for j := range headerFields {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	259 if headerFields[j].name == h {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	260 if *headerFields[j].idx != -1 {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	261 return nil, fmt.Errorf(
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	262 "There is more than one column namend '%s'", h)
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	263 }
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	264 *headerFields[j].idx = i
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	265 continue nextHeader
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	266 }
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	267 }
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	268 }
fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	269
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	270 var missing []string
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	271 for i := range headerFields {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	272 if headerFields[i].name != "unit" && *headerFields[i].idx == -1 {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	273 missing = append(missing, headerFields[i].name)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	274 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	275 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	276 if len(missing) > 0 {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	277 return nil, fmt.Errorf("Missing columns: %s", strings.Join(missing, ", "))
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	278 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	279
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	280 gaugeCheckStmt, err := conn.PrepareContext(ctx, agmGaugeCheckSQL)
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	281 if err != nil {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	282 return nil, err
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	283 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	284 defer gaugeCheckStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	285
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	286 selectStmt, err := conn.PrepareContext(ctx, agmSelectSQL)
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	287 if err != nil {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	288 return nil, err
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	289 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	290 defer selectStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	291
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	292 insertStmt, err := conn.PrepareContext(ctx, agmInsertSQL)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	293 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	294 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	295 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	296 defer insertStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	297
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	298 trackStmt, err := conn.PrepareContext(ctx, trackImportSQL)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	299 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	300 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	301 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	302 defer trackStmt.Close()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	303
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	304 entries := []*agmSummaryEntry{}
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	305
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	306 checkedGauges := map[models.Isrs]bool{}
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	307
3504 6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	308 warnLimiter := misc.WarningLimiter{Log: feedback.Warn, MaxWarnings: 100}
6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	309 warn := warnLimiter.Warn
6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	310 defer warnLimiter.Close()
6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	311
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	312 lines:
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	313 for line, ignored := 1, 0; ; line++ {
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	314
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	315 row, err := r.Read()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	316 switch {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	317 case err == io.EOF \|\| len(row) == 0:
3218 c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	318 feedback.Info("Read %d entries in CSV file", line-1)
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	319 if ignored > 0 {
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	320 feedback.Info("%d entries ignored", ignored)
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	321 }
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	322 if ignored == line-1 {
8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	323 return nil, UnchangedError("No entries imported")
8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	324 }
3218 c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	325 feedback.Info("Imported %d entries with changes", len(entries))
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	326 break lines
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	327 case err != nil:
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	328 return nil, fmt.Errorf("CSV parsing failed: %v", err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	329 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	330
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	331 gids := row[fkGaugeIDIdx]
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	332 gid, err := models.IsrsFromString(gids)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	333 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	334 return nil, fmt.Errorf("Invalid ISRS code line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	335 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	336
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	337 if exists, found := checkedGauges[*gid]; found {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	338 if !exists {
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	339 // Just ignore the line since we have already warned
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	340 ignored++
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	341 continue lines
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	342 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	343 } else { // not found in gauge cache
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	344 if err := gaugeCheckStmt.QueryRowContext(
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	345 ctx,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	346 gid.CountryCode,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	347 gid.LoCode,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	348 gid.FairwaySection,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	349 gid.Orc,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	350 gid.Hectometre,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	351 ).Scan(&exists); err != nil {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	352 return nil, err
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	353 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	354 checkedGauges[*gid] = exists
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	355 if !exists {
3504 6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	356 warn("Ignoring data for unknown gauge %s", gid.String())
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	357 ignored++
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	358 continue lines
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	359 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	360 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	361
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	362 md, err := guessDate(row[measureDateIdx])
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	363 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	364 return nil, fmt.Errorf("Invalid 'measure_date' line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	365 }
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	366
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	367 var (
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	368 oldID int64
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	369 oldCountryCode string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	370 oldSender string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	371 oldLanguageCode string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	372 oldDateIssue time.Time
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	373 oldReferenceCode string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	374 oldValue float64
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	375 oldDateInfo time.Time
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	376 oldSourceOrganization string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	377 )
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	378
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	379 err = selectStmt.QueryRowContext(
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	380 ctx,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	381 gid.CountryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	382 gid.LoCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	383 gid.FairwaySection,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	384 gid.Orc,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	385 gid.Hectometre,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	386 md,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	387 ).Scan(
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	388 &oldID,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	389 &oldCountryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	390 &oldSender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	391 &oldLanguageCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	392 &oldDateIssue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	393 &oldReferenceCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	394 &oldValue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	395 &oldDateInfo,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	396 &oldSourceOrganization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	397 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	398
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	399 var newEntry bool
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	400 switch {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	401 case err == sql.ErrNoRows:
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	402 // Complete new one
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	403 newEntry = true
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	404 case err != nil:
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	405 return nil, err
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	406 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	407
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	408 newSender := agm.Originator
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	409 newCountryCode := gid.CountryCode
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	410 newLanguageCode := misc.CCtoLang[gid.CountryCode]
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	411 newDateIssue := time.Now()
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	412 newReferenceCode := "ZPG"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	413
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	414 value, err := strconv.ParseFloat(row[valueIdx], 32)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	415 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	416 return nil, fmt.Errorf("Invalid 'value' line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	417 }
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	418 newValue := value
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	419
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	420 newDateInfo := newDateIssue
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	421
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	422 newSourceOrganization := newSender
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	423
3502 45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	424 switch err := func() error {
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	425 tx, err := conn.BeginTx(ctx, nil)
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	426 if err != nil {
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	427 return err
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	428 }
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	429 defer tx.Rollback()
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	430
3502 45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	431 var newID int64
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	432
3502 45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	433 if err := tx.StmtContext(ctx, insertStmt).QueryRowContext(
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	434 ctx,
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	435 gid.CountryCode,
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	436 gid.LoCode,
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	437 gid.FairwaySection,
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	438 gid.Orc,
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	439 gid.Hectometre,
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	440 md,
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	441 newCountryCode,
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	442 newSender,
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	443 newLanguageCode,
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	444 newDateIssue,
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	445 newReferenceCode,
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	446 newValue,
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	447 newDateInfo,
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	448 newSourceOrganization,
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	449 ).Scan(&newID); err != nil {
3504 6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	450 warn(handleError(err).Error())
3502 45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	451 ignored++
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	452 return errContinue
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	453 }
3502 45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	454
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	455 if _, err := tx.StmtContext(ctx, trackStmt).ExecContext(
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	456 ctx, importID, "waterway.gauge_measurements", newID,
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	457 ); err != nil {
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	458 return err
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	459 }
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	460
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	461 if err = tx.Commit(); err != nil {
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	462 err = fmt.Errorf("Commit failed: %v", err)
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	463 }
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	464 return err
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	465 }(); {
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	466 case err == errContinue:
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	467 continue lines
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	468 case err != nil:
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	469 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	470 }
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	471
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	472 n := newAGMLine(
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	473 newCountryCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	474 newSender,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	475 newLanguageCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	476 newDateIssue,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	477 newReferenceCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	478 newValue,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	479 newDateInfo,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	480 newSourceOrganization,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	481 )
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	482
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	483 ase := &agmSummaryEntry{
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	484 FKGaugeID: *gid,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	485 MeasureDate: timetz{md},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	486 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	487
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	488 if newEntry {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	489 ase.Versions = []*agmLine{n}
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	490 } else {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	491 o := newAGMLine(
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	492 oldCountryCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	493 oldSender,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	494 oldLanguageCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	495 oldDateIssue,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	496 oldReferenceCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	497 oldValue,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	498 oldDateInfo,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	499 oldSourceOrganization,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	500 )
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	501 // Ignore if there is no diff.
3277 232fc90e6ee2 Disentangle gauge measurements and predictions Tom Gottfried <tom@intevation.de> parents: 3220 diff changeset	502 if !n.hasDiff(o) {
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	503 continue
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	504 }
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	505 ase.Versions = []*agmLine{o, n}
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	506 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	507 entries = append(entries, ase)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	508 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	509
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	510 feedback.Info("Importing approved gauge measurements took %s",
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	511 time.Since(start))
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	512
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	513 return entries, nil
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	514 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	515
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	516 func newAGMLine(
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	517 countryCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	518 sender string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	519 languageCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	520 dateIssue time.Time,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	521 referenceCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	522 waterLevel float64,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	523 dateInfo time.Time,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	524 sourceOrganization string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	525 ) *agmLine {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	526 return &agmLine{
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	527 CountryCode: countryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	528 Sender: sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	529 LanguageCode: languageCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	530 DateIssue: timetz{dateIssue},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	531 ReferenceCode: referenceCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	532 WaterLevel: waterLevel,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	533 DateInfo: timetz{dateInfo},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	534 SourceOrganization: sourceOrganization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	535 }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	536 }

Mercurial > gemma

annotate pkg/imports/agm.go @ 3533:8e083b271fca