gemma: pkg/imports/agm.go annotate

annotate pkg/imports/agm.go @ 4027:b17453420eff

Avoid doing one SELECT per line in import file When the import is run by a waterway admin (more row level security processing involved than as sys_admin), this approximately halves the overall duration of the import

author	Tom Gottfried <tom@intevation.de>
date	Mon, 22 Jul 2019 19:19:00 +0200
parents	82037bbd2c7c
children	040a5dc95eb9

rev	line source
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	7 // Copyright (C) 2018, 2019 by via donau
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	13 // * Sascha Wilde <wilde@intevation.de>
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	14 // * Tom Gottfried <tom.gottfried@intevation.de>
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16 package imports
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 import (
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	19 "bufio"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	20 "context"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	21 "database/sql"
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	22 "encoding/csv"
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	23 "encoding/json"
3502 45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	24 "errors"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	25 "fmt"
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	26 "io"
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	27 "math"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	28 "os"
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	29 "path/filepath"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	30 "strconv"
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	31 "strings"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	32 "time"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	33
1938 206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	34 "gemma.intevation.de/gemma/pkg/misc"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	35 "gemma.intevation.de/gemma/pkg/models"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	36 )
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	37
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	38 type ApprovedGaugeMeasurements struct {
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	39 Dir string `json:"dir"`
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	40 Originator string `json:"originator"`
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	41 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	42
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	43 // GMAPJobKind is the unique name of an approved gauge measurements import job.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	44 const AGMJobKind JobKind = "agm"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	45
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	46 type agmJobCreator struct{}
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	47
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	48 func init() {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	49 RegisterJobCreator(AGMJobKind, agmJobCreator{})
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	50 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	51
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1743 diff changeset	52 func (agmJobCreator) AutoAccept() bool { return false }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1743 diff changeset	53
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	54 func (agmJobCreator) Description() string {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	55 return "approved gauge measurements"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	56 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	57
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2173 diff changeset	58 func (agmJobCreator) Create() Job { return new(ApprovedGaugeMeasurements) }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	59
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	60 func (agmJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	61 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	62 {"gauge_measurements"},
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	63 {"gauges"},
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	64 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	65 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	66
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	67 const (
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	68 // delete the old and keep the new measures.
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	69 agmStageDoneDeleteSQL = `
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	70 WITH staged AS (
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	71 SELECT key
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1938 diff changeset	72 FROM import.track_imports
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	73 WHERE import_id = $1 AND
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	74 relation = 'waterway.gauge_measurements'::regclass
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	75 ),
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	76 to_delete AS (
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	77 SELECT o.id AS id
1781 0a53c33bc7b2 Fixed broken SQL statement (once again). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1780 diff changeset	78 FROM waterway.gauge_measurements o
0a53c33bc7b2 Fixed broken SQL statement (once again). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1780 diff changeset	79 JOIN waterway.gauge_measurements n
3389 45a629a3a8b8 Fix constraints on relationship between gauges and measurements/predictions Tom Gottfried <tom@intevation.de> parents: 3302 diff changeset	80 USING (location, measure_date)
2850 6d4f361c36e8 Do not delete predictions on import of agm. Sascha Wilde <wilde@intevation.de> parents: 2562 diff changeset	81 WHERE n.id IN (SELECT key FROM staged)
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	82 AND o.id NOT IN (SELECT key FROM staged)
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	83 )
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	84 DELETE FROM waterway.gauge_measurements WHERE id IN (SELECT id from to_delete)`
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	85
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	86 agmStageDoneSQL = `
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	87 UPDATE waterway.gauge_measurements SET staging_done = true
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	88 WHERE id IN (
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1938 diff changeset	89 SELECT key FROM import.track_imports
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	90 WHERE import_id = $1 AND
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	91 relation = 'waterway.gauge_measurements'::regclass)`
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	92 )
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	93
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	94 func (agmJobCreator) StageDone(
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	95 ctx context.Context,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	96 tx *sql.Tx,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	97 id int64,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	98 ) error {
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	99 _, err := tx.ExecContext(ctx, agmStageDoneDeleteSQL, id)
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	100 if err == nil {
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	101 _, err = tx.ExecContext(ctx, agmStageDoneSQL, id)
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	102 }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	103 return err
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	104 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	105
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	106 // CleanUp removes the folder containing the CSV file with the
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	107 // the approved gauge measurements.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	108 func (agm *ApprovedGaugeMeasurements) CleanUp() error {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	109 return os.RemoveAll(agm.Dir)
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	110 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	111
1938 206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	112 var guessDate = misc.TimeGuesser([]string{
206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	113 "02.01.2006 15:04",
206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	114 "2006-01-02T15:04:05-07:00",
206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	115 }).Guess
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	116
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	117 type timetz struct{ time.Time }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	118
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	119 func (ttz *timetz) MarshalJSON() ([]byte, error) {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	120 return json.Marshal(ttz.Time.Format("2006-01-02T15:04:05-07:00"))
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	121 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	122
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	123 type agmLine struct {
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	124 Location models.Isrs `json:"fk-gauge-id"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	125 CountryCode string `json:"country-code"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	126 Sender string `json:"sender"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	127 LanguageCode string `json:"language-code"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	128 DateIssue timetz `json:"date-issue"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	129 ReferenceCode string `json:"reference-code"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	130 MeasureDate timetz `json:"measure-date"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	131 WaterLevel float64 `json:"water-level"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	132 DateInfo timetz `json:"date-info"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	133 SourceOrganization string `json:"source-organization"`
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	134 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	135
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	136 func (a agmLine) hasDiff(b agmLine) bool {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	137 const eps = 0.00001
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	138 return a.CountryCode != b.CountryCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	139 a.Sender != b.Sender \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	140 a.LanguageCode != b.LanguageCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	141 a.ReferenceCode != b.ReferenceCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	142 math.Abs(a.WaterLevel-b.WaterLevel) > eps \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	143 a.SourceOrganization != b.SourceOrganization
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	144 }
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	145
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	146 type agmSummaryEntry struct {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	147 FKGaugeID models.Isrs `json:"fk-gauge-id"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	148 MeasureDate timetz `json:"measure-date"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	149 Versions []*agmLine `json:"versions"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	150 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	151
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	152 const (
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	153 agmSelectSQL = `
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	154 SELECT
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	155 country_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	156 sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	157 language_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	158 date_issue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	159 reference_code,
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	160 measure_date,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	161 water_level,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	162 date_info,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	163 source_organization
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	164 FROM waterway.gauge_measurements
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	165 WHERE
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	166 location
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	167 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	168 AND measure_date BETWEEN $6 AND $7
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	169 AND staging_done
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	170 `
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	171
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	172 agmInsertSQL = `
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	173 INSERT INTO waterway.gauge_measurements (
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	174 location,
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	175 validity,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	176 measure_date,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	177 country_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	178 sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	179 language_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	180 date_issue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	181 reference_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	182 water_level,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	183 date_info,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	184 source_organization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	185 staging_done
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	186 ) VALUES (
2120 817cd8b89a86 Approved gauge measuments import: Give type hints when insertinf/querying isrs location codes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2105 diff changeset	187 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int),
3533 8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3504 diff changeset	188 COALESCE(
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3504 diff changeset	189 (SELECT validity FROM waterway.gauges
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3504 diff changeset	190 WHERE location
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3504 diff changeset	191 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3504 diff changeset	192 AND validity @> CAST($6 AS timestamp with time zone)),
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3504 diff changeset	193 tstzrange(NULL, NULL)),
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	194 $6,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	195 $7,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	196 $8,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	197 $9,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	198 $10,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	199 $11,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	200 $12,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	201 $13,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	202 $14,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	203 false
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	204 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	205 RETURNING id`
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	206
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	207 agmGaugeCheckSQL = `
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	208 SELECT EXISTS(
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	209 SELECT 1 FROM waterway.gauges
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	210 WHERE location = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int))
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	211 `
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	212 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	213
3502 45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	214 var errContinue = errors.New("continue")
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	215
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	216 // Do executes the actual approved gauge measurements import.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	217 func (agm *ApprovedGaugeMeasurements) Do(
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	218 ctx context.Context,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	219 importID int64,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	220 conn *sql.Conn,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	221 feedback Feedback,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	222 ) (interface{}, error) {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	223
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	224 start := time.Now()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	225
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	226 f, err := os.Open(filepath.Join(agm.Dir, "agm.csv"))
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	227 if err != nil {
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	228 return nil, err
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	229 }
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	230 defer f.Close()
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	231
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	232 r := csv.NewReader(bufio.NewReader(f))
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	233 r.Comma = ';'
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	234 r.ReuseRecord = true
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	235
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	236 headers, err := r.Read()
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	237 if err != nil {
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	238 return nil, err
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	239 }
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	240
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	241 var (
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	242 fkGaugeIDIdx = -1
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	243 measureDateIdx = -1
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	244 valueIdx = -1
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	245 )
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	246
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	247 headerFields := []struct {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	248 idx *int
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	249 name string
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	250 }{
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	251 {&fkGaugeIDIdx, "fk_gauge_id"},
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	252 {&measureDateIdx, "measure_date"},
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	253 {&valueIdx, "value"}, // "water_level",
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	254 }
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	255
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	256 nextHeader:
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	257 for i, f := range headers {
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	258 h := strings.Replace(strings.ToLower(
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	259 strings.TrimSpace(f)), " ", "_", -1)
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	260
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	261 for j := range headerFields {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	262 if headerFields[j].name == h {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	263 if *headerFields[j].idx != -1 {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	264 return nil, fmt.Errorf(
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	265 "There is more than one column namend '%s'", h)
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	266 }
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	267 *headerFields[j].idx = i
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	268 continue nextHeader
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	269 }
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	270 }
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	271 }
fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	272
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	273 var missing []string
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	274 for i := range headerFields {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	275 if headerFields[i].name != "unit" && *headerFields[i].idx == -1 {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	276 missing = append(missing, headerFields[i].name)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	277 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	278 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	279 if len(missing) > 0 {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	280 return nil, fmt.Errorf("Missing columns: %s", strings.Join(missing, ", "))
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	281 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	282
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	283 gaugeCheckStmt, err := conn.PrepareContext(ctx, agmGaugeCheckSQL)
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	284 if err != nil {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	285 return nil, err
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	286 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	287 defer gaugeCheckStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	288
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	289 selectStmt, err := conn.PrepareContext(ctx, agmSelectSQL)
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	290 if err != nil {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	291 return nil, err
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	292 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	293 defer selectStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	294
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	295 insertStmt, err := conn.PrepareContext(ctx, agmInsertSQL)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	296 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	297 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	298 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	299 defer insertStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	300
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	301 trackStmt, err := conn.PrepareContext(ctx, trackImportSQL)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	302 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	303 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	304 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	305 defer trackStmt.Close()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	306
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	307 entries := []*agmSummaryEntry{}
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	308
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	309 checkedGauges := map[models.Isrs]bool{}
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	310
3504 6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	311 warnLimiter := misc.WarningLimiter{Log: feedback.Warn, MaxWarnings: 100}
6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	312 warn := warnLimiter.Warn
6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	313 defer warnLimiter.Close()
6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	314
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	315 agmLines := []*agmLine{}
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	316 ignored := 0
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	317 mdMinMax := map[models.Isrs][2]time.Time{}
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	318
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	319 lines:
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	320 for line := 1; ; line++ {
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	321
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	322 row, err := r.Read()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	323 switch {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	324 case err == io.EOF \|\| len(row) == 0:
3218 c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	325 feedback.Info("Read %d entries in CSV file", line-1)
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	326 if ignored > 0 {
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	327 feedback.Info("%d entries ignored", ignored)
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	328 }
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	329 if ignored == line-1 {
8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	330 return nil, UnchangedError("No entries imported")
8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	331 }
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	332 break lines
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	333 case err != nil:
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	334 return nil, fmt.Errorf("CSV parsing failed: %v", err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	335 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	336
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	337 gids := row[fkGaugeIDIdx]
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	338 gid, err := models.IsrsFromString(gids)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	339 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	340 return nil, fmt.Errorf("Invalid ISRS code line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	341 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	342
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	343 if exists, found := checkedGauges[*gid]; found {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	344 if !exists {
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	345 // Just ignore the line since we have already warned
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	346 ignored++
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	347 continue lines
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	348 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	349 } else { // not found in gauge cache
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	350 if err := gaugeCheckStmt.QueryRowContext(
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	351 ctx,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	352 gid.CountryCode,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	353 gid.LoCode,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	354 gid.FairwaySection,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	355 gid.Orc,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	356 gid.Hectometre,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	357 ).Scan(&exists); err != nil {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	358 return nil, err
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	359 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	360 checkedGauges[*gid] = exists
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	361 if !exists {
3504 6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	362 warn("Ignoring data for unknown gauge %s", gid.String())
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	363 ignored++
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	364 continue lines
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	365 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	366 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	367
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	368 md, err := guessDate(row[measureDateIdx])
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	369 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	370 return nil, fmt.Errorf("Invalid 'measure_date' line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	371 }
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	372 if _, hasGid := mdMinMax[*gid]; hasGid {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	373 if md.Before(mdMinMax[*gid][0]) {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	374 mdMinMax[gid] = [2]time.Time{md, mdMinMax[gid][1]}
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	375 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	376 if md.After(mdMinMax[*gid][1]) {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	377 mdMinMax[gid] = [2]time.Time{mdMinMax[gid][0], md}
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	378 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	379 } else {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	380 mdMinMax[*gid] = [2]time.Time{md, md}
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	381 }
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	382
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	383 newSender := agm.Originator
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	384 newCountryCode := gid.CountryCode
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	385 newLanguageCode := misc.CCtoLang[gid.CountryCode]
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	386 newDateIssue := time.Now()
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	387 newReferenceCode := "ZPG"
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	388
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	389 value, err := strconv.ParseFloat(row[valueIdx], 32)
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	390 if err != nil {
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	391 return nil, fmt.Errorf("Invalid 'value' line %d: %v", line, err)
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	392 }
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	393 newValue := value
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	394
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	395 newDateInfo := newDateIssue
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	396
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	397 newSourceOrganization := newSender
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	398
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	399 agmLines = append(agmLines, newAGMLine(
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	400 *gid,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	401 newCountryCode,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	402 newSender,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	403 newLanguageCode,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	404 newDateIssue,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	405 newReferenceCode,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	406 md,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	407 newValue,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	408 newDateInfo,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	409 newSourceOrganization,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	410 ))
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	411 }
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	412
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	413 oldGMLines := map[models.Isrs]map[int64]*agmLine{}
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	414 for gid, minMax := range mdMinMax {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	415 oldGMLines[gid], err = getOldGMLines(
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	416 ctx, selectStmt, gid, minMax[0], minMax[1])
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	417 if err != nil {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	418 return nil, err
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	419 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	420 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	421
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	422 agmLines:
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	423 for _, line := range agmLines {
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	424
3502 45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	425 switch err := func() error {
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	426 tx, err := conn.BeginTx(ctx, nil)
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	427 if err != nil {
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	428 return err
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	429 }
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	430 defer tx.Rollback()
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	431
3502 45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	432 var newID int64
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	433
3502 45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	434 if err := tx.StmtContext(ctx, insertStmt).QueryRowContext(
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	435 ctx,
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	436 line.Location.CountryCode,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	437 line.Location.LoCode,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	438 line.Location.FairwaySection,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	439 line.Location.Orc,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	440 line.Location.Hectometre,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	441 line.MeasureDate.Time,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	442 line.CountryCode,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	443 line.Sender,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	444 line.LanguageCode,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	445 line.DateIssue.Time,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	446 line.ReferenceCode,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	447 line.WaterLevel,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	448 line.DateInfo.Time,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	449 line.SourceOrganization,
3502 45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	450 ).Scan(&newID); err != nil {
3504 6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	451 warn(handleError(err).Error())
3502 45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	452 ignored++
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	453 return errContinue
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	454 }
3502 45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	455
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	456 if _, err := tx.StmtContext(ctx, trackStmt).ExecContext(
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	457 ctx, importID, "waterway.gauge_measurements", newID,
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	458 ); err != nil {
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	459 return err
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	460 }
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	461
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	462 if err = tx.Commit(); err != nil {
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	463 err = fmt.Errorf("Commit failed: %v", err)
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	464 }
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	465 return err
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	466 }(); {
45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	467 case err == errContinue:
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	468 continue agmLines
3502 45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	469 case err != nil:
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	470 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	471 }
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	472
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	473 ase := &agmSummaryEntry{
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	474 FKGaugeID: line.Location,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	475 MeasureDate: line.MeasureDate,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	476 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	477
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	478 if o, hasOld := oldGMLines[line.Location][line.MeasureDate.Time.Unix()]; !hasOld {
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	479 ase.Versions = []*agmLine{line}
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	480 } else {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	481 // Ignore if there is no diff.
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	482 if !line.hasDiff(o) {
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	483 continue
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	484 }
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	485 ase.Versions = []*agmLine{o, line}
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	486 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	487 entries = append(entries, ase)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	488 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	489
4026 82037bbd2c7c Do not require review if nothing changed Tom Gottfried <tom@intevation.de> parents: 4023 diff changeset	490 if len(entries) == 0 {
82037bbd2c7c Do not require review if nothing changed Tom Gottfried <tom@intevation.de> parents: 4023 diff changeset	491 return nil, UnchangedError("No entries with changes")
82037bbd2c7c Do not require review if nothing changed Tom Gottfried <tom@intevation.de> parents: 4023 diff changeset	492 }
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	493 feedback.Info("Imported %d entries with changes", len(entries))
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	494 feedback.Info("Importing approved gauge measurements took %s",
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	495 time.Since(start))
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	496
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	497 return entries, nil
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	498 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	499
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	500 func getOldGMLines(
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	501 ctx context.Context,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	502 stmt *sql.Stmt,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	503 location models.Isrs,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	504 from time.Time,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	505 to time.Time,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	506 ) (map[int64]*agmLine, error) {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	507 var (
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	508 oldCountryCode string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	509 oldSender string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	510 oldLanguageCode string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	511 oldDateIssue time.Time
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	512 oldReferenceCode string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	513 oldMeasureDate time.Time
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	514 oldValue float64
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	515 oldDateInfo time.Time
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	516 oldSourceOrganization string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	517 )
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	518 gmLines := map[int64]*agmLine{}
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	519
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	520 gms, err := stmt.QueryContext(
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	521 ctx,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	522 location.CountryCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	523 location.LoCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	524 location.FairwaySection,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	525 location.Orc,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	526 location.Hectometre,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	527 from,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	528 to,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	529 )
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	530 if err != nil {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	531 return nil, err
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	532 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	533 defer gms.Close()
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	534 for gms.Next() {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	535 if err = gms.Scan(
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	536 &oldCountryCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	537 &oldSender,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	538 &oldLanguageCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	539 &oldDateIssue,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	540 &oldReferenceCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	541 &oldMeasureDate,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	542 &oldValue,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	543 &oldDateInfo,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	544 &oldSourceOrganization,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	545 ); err != nil {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	546 return nil, err
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	547 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	548 gmLines[oldMeasureDate.Unix()] = newAGMLine(
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	549 location,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	550 oldCountryCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	551 oldSender,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	552 oldLanguageCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	553 oldDateIssue,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	554 oldReferenceCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	555 oldMeasureDate,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	556 oldValue,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	557 oldDateInfo,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	558 oldSourceOrganization,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	559 )
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	560 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	561 if err = gms.Err(); err != nil {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	562 return nil, err
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	563 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	564 return gmLines, nil
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	565 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	566
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	567 func newAGMLine(
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	568 location models.Isrs,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	569 countryCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	570 sender string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	571 languageCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	572 dateIssue time.Time,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	573 referenceCode string,
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	574 measureDate time.Time,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	575 waterLevel float64,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	576 dateInfo time.Time,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	577 sourceOrganization string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	578 ) *agmLine {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	579 return &agmLine{
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	580 Location: location,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	581 CountryCode: countryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	582 Sender: sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	583 LanguageCode: languageCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	584 DateIssue: timetz{dateIssue},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	585 ReferenceCode: referenceCode,
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	586 MeasureDate: timetz{measureDate},
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	587 WaterLevel: waterLevel,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	588 DateInfo: timetz{dateInfo},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	589 SourceOrganization: sourceOrganization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	590 }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	591 }

Mercurial > gemma

annotate pkg/imports/agm.go @ 4027:b17453420eff