gemma: pkg/imports/agm.go annotate

annotate pkg/imports/agm.go @ 5591:0011f50cf216 surveysperbottleneckid

Removed no longer used alternative api for surveys/ endpoint. As bottlenecks in the summary for SR imports are now identified by their id and no longer by the (not guarantied to be unique!) name, there is no longer the need to request survey data by the name+date tuple (which isn't reliable anyway). So the workaround was now reversed.

author	Sascha Wilde <wilde@sha-bang.de>
date	Wed, 06 Apr 2022 13:30:29 +0200
parents	56c589f7435d
children	6270951dda28

rev	line source
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	7 // Copyright (C) 2018, 2019 by via donau
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	13 // * Sascha Wilde <wilde@intevation.de>
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	14 // * Tom Gottfried <tom.gottfried@intevation.de>
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16 package imports
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 import (
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	19 "bufio"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	20 "context"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	21 "database/sql"
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	22 "encoding/csv"
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	23 "encoding/json"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	24 "fmt"
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	25 "io"
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	26 "math"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	27 "os"
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	28 "path/filepath"
4049 4a7c2140e44b AGM import: Generate entries for deleted items. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4037 diff changeset	29 "sort"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	30 "strconv"
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	31 "strings"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	32 "time"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	33
4075 cb74aa69954e Moved TimeParser to common package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4074 diff changeset	34 "gemma.intevation.de/gemma/pkg/common"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	35 "gemma.intevation.de/gemma/pkg/models"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	36 )
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	37
4178 51e90370eced A few more doc strings to make 'golint' a little bit more happy with the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4177 diff changeset	38 // ApprovedGaugeMeasurements is a Job to import
51e90370eced A few more doc strings to make 'golint' a little bit more happy with the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4177 diff changeset	39 // approved gauge measurements from a CVS file
51e90370eced A few more doc strings to make 'golint' a little bit more happy with the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4177 diff changeset	40 // into the database.
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	41 type ApprovedGaugeMeasurements struct {
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	42 Dir string `json:"dir"`
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	43 Originator string `json:"originator"`
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	44 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	45
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4080 diff changeset	46 // AGMJobKind is the unique name of an approved gauge measurements import job.
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	47 const AGMJobKind JobKind = "agm"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	48
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	49 type agmJobCreator struct{}
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	50
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	51 func init() {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	52 RegisterJobCreator(AGMJobKind, agmJobCreator{})
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	53 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	54
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1743 diff changeset	55 func (agmJobCreator) AutoAccept() bool { return false }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1743 diff changeset	56
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	57 func (agmJobCreator) Description() string {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	58 return "approved gauge measurements"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	59 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	60
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2173 diff changeset	61 func (agmJobCreator) Create() Job { return new(ApprovedGaugeMeasurements) }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	62
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	63 func (agmJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	64 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	65 {"gauge_measurements"},
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	66 {"gauges"},
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	67 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	68 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	69
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	70 const (
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	71 agmStageDoneDeleteSQL = `
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	72 DELETE FROM waterway.gauge_measurements WHERE id IN (
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	73 SELECT key
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1938 diff changeset	74 FROM import.track_imports
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	75 WHERE import_id = $1 AND
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	76 relation = 'waterway.gauge_measurements'::regclass AND
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	77 deletion
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	78 )`
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	79
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	80 agmStageDoneSQL = `
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	81 UPDATE waterway.gauge_measurements SET staging_done = true
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	82 WHERE id IN (
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	83 SELECT key
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	84 FROM import.track_imports
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	85 WHERE import_id = $1 AND
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	86 relation = 'waterway.gauge_measurements'::regclass AND
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	87 NOT deletion
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	88 )`
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	89 )
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	90
5035 56c589f7435d Enhance comments on implementations of StageDone() Tom Gottfried <tom@intevation.de> parents: 5034 diff changeset	91 // StageDone replaces gauge measurements with those in the staging area
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	92 func (agmJobCreator) StageDone(
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	93 ctx context.Context,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	94 tx *sql.Tx,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	95 id int64,
5034 59a99655f34d Added feedback support for StageDone. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4178 diff changeset	96 _ Feedback,
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	97 ) error {
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	98 _, err := tx.ExecContext(ctx, agmStageDoneDeleteSQL, id)
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	99 if err == nil {
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	100 _, err = tx.ExecContext(ctx, agmStageDoneSQL, id)
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	101 }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	102 return err
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	103 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	104
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	105 // CleanUp removes the folder containing the CSV file with the
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	106 // the approved gauge measurements.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	107 func (agm *ApprovedGaugeMeasurements) CleanUp() error {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	108 return os.RemoveAll(agm.Dir)
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	109 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	110
4075 cb74aa69954e Moved TimeParser to common package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4074 diff changeset	111 var guessDate = common.TimeParser([]string{
1938 206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	112 "02.01.2006 15:04",
206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	113 "2006-01-02T15:04:05-07:00",
4074 eb2f949ddfa2 Rename TimeGuesser to TimeParser. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4060 diff changeset	114 }).Parse
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	115
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	116 type timetz struct{ time.Time }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	117
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	118 func (ttz *timetz) MarshalJSON() ([]byte, error) {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	119 return json.Marshal(ttz.Time.Format("2006-01-02T15:04:05-07:00"))
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	120 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	121
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	122 type agmLine struct {
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	123 id int64
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	124 Location models.Isrs `json:"fk-gauge-id"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	125 CountryCode string `json:"country-code"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	126 Sender string `json:"sender"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	127 LanguageCode string `json:"language-code"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	128 DateIssue timetz `json:"date-issue"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	129 ReferenceCode string `json:"reference-code"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	130 MeasureDate timetz `json:"measure-date"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	131 WaterLevel float64 `json:"water-level"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	132 DateInfo timetz `json:"date-info"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	133 SourceOrganization string `json:"source-organization"`
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	134 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	135
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	136 func (a agmLine) hasDiff(b agmLine) bool {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	137 const eps = 0.00001
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	138 return a.CountryCode != b.CountryCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	139 a.Sender != b.Sender \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	140 a.LanguageCode != b.LanguageCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	141 a.ReferenceCode != b.ReferenceCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	142 math.Abs(a.WaterLevel-b.WaterLevel) > eps \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	143 a.SourceOrganization != b.SourceOrganization
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	144 }
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	145
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	146 type agmSummaryEntry struct {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	147 FKGaugeID models.Isrs `json:"fk-gauge-id"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	148 MeasureDate timetz `json:"measure-date"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	149 Versions []*agmLine `json:"versions"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	150 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	151
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	152 const (
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	153 agmSelectSQL = `
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	154 SELECT
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	155 id,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	156 country_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	157 sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	158 language_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	159 date_issue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	160 reference_code,
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	161 measure_date,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	162 water_level,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	163 date_info,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	164 source_organization
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	165 FROM waterway.gauge_measurements
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	166 WHERE
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	167 location
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	168 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	169 AND measure_date BETWEEN $6 AND $7
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	170 AND staging_done
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	171 `
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	172
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	173 agmInsertSQL = `
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	174 INSERT INTO waterway.gauge_measurements (
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	175 location,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	176 measure_date,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	177 country_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	178 sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	179 language_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	180 date_issue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	181 reference_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	182 water_level,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	183 date_info,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	184 source_organization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	185 staging_done
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	186 ) VALUES (
2120 817cd8b89a86 Approved gauge measuments import: Give type hints when insertinf/querying isrs location codes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2105 diff changeset	187 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int),
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	188 $6,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	189 $7,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	190 $8,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	191 $9,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	192 $10,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	193 $11,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	194 $12,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	195 $13,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	196 $14,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	197 false
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	198 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	199 RETURNING id`
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	200
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	201 agmGaugeCheckSQL = `
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	202 SELECT EXISTS(
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	203 SELECT 1 FROM waterway.gauges
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	204 WHERE location = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int))
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	205 `
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	206 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	207
4030 040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	208 func parseAGMHeaders(headers []string, fkGaugeIDIdx, measureDateIdx, valueIdx *int) error {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	209
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	210 headerFields := []struct {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	211 idx *int
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	212 name string
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	213 }{
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	214 {fkGaugeIDIdx, "fk_gauge_id"},
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	215 {measureDateIdx, "measure_date"},
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	216 {valueIdx, "value"}, // "water_level",
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	217 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	218
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	219 nextHeader:
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	220 for i, f := range headers {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	221 h := strings.Replace(strings.ToLower(
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	222 strings.TrimSpace(f)), " ", "_", -1)
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	223
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	224 for j := range headerFields {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	225 if headerFields[j].name == h {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	226 if *headerFields[j].idx != -1 {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	227 return fmt.Errorf(
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4171 diff changeset	228 "there is more than one column namend '%s'", h)
4030 040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	229 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	230 *headerFields[j].idx = i
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	231 continue nextHeader
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	232 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	233 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	234 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	235
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	236 var missing []string
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	237 for i := range headerFields {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	238 if headerFields[i].name != "unit" && *headerFields[i].idx == -1 {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	239 missing = append(missing, headerFields[i].name)
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	240 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	241 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	242 if len(missing) > 0 {
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4171 diff changeset	243 return fmt.Errorf("missing columns: %s", strings.Join(missing, ", "))
4030 040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	244 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	245
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	246 return nil
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	247 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	248
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	249 // Do executes the actual approved gauge measurements import.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	250 func (agm *ApprovedGaugeMeasurements) Do(
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	251 ctx context.Context,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	252 importID int64,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	253 conn *sql.Conn,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	254 feedback Feedback,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	255 ) (interface{}, error) {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	256
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	257 start := time.Now()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	258
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	259 f, err := os.Open(filepath.Join(agm.Dir, "agm.csv"))
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	260 if err != nil {
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	261 return nil, err
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	262 }
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	263 defer f.Close()
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	264
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	265 r := csv.NewReader(bufio.NewReader(f))
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	266 r.Comma = ';'
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	267 r.ReuseRecord = true
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	268
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	269 headers, err := r.Read()
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	270 if err != nil {
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	271 return nil, err
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	272 }
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	273
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	274 var (
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	275 fkGaugeIDIdx = -1
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	276 measureDateIdx = -1
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	277 valueIdx = -1
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	278 )
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	279
4030 040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	280 if err := parseAGMHeaders(
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	281 headers,
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	282 &fkGaugeIDIdx, &measureDateIdx, &valueIdx,
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	283 ); err != nil {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	284 return nil, err
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	285 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	286
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	287 gaugeCheckStmt, err := conn.PrepareContext(ctx, agmGaugeCheckSQL)
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	288 if err != nil {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	289 return nil, err
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	290 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	291 defer gaugeCheckStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	292
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	293 selectStmt, err := conn.PrepareContext(ctx, agmSelectSQL)
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	294 if err != nil {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	295 return nil, err
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	296 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	297 defer selectStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	298
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	299 insertStmt, err := conn.PrepareContext(ctx, agmInsertSQL)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	300 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	301 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	302 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	303 defer insertStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	304
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	305 trackStmt, err := conn.PrepareContext(ctx, trackImportDeletionSQL)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	306 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	307 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	308 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	309 defer trackStmt.Close()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	310
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	311 entries := []*agmSummaryEntry{}
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	312
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	313 checkedGauges := map[models.Isrs]bool{}
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	314
4171 5a650cde0574 Move warning limiter to common package to flesh up the misc package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4170 diff changeset	315 warnLimiter := common.WarningLimiter{Log: feedback.Warn, MaxWarnings: 100}
3504 6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	316 warn := warnLimiter.Warn
6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	317 defer warnLimiter.Close()
6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	318
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	319 agmLines := []*agmLine{}
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	320 ignored := 0
4035 f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	321 mdMinMax := map[models.Isrs]*[2]time.Time{}
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	322
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	323 lines:
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	324 for line := 1; ; line++ {
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	325
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	326 row, err := r.Read()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	327 switch {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	328 case err == io.EOF \|\| len(row) == 0:
3218 c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	329 feedback.Info("Read %d entries in CSV file", line-1)
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	330 if ignored > 0 {
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	331 feedback.Info("%d entries ignored", ignored)
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	332 }
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	333 if ignored == line-1 {
8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	334 return nil, UnchangedError("No entries imported")
8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	335 }
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	336 break lines
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	337 case err != nil:
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	338 return nil, fmt.Errorf("CSV parsing failed: %v", err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	339 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	340
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	341 gids := row[fkGaugeIDIdx]
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	342 gid, err := models.IsrsFromString(gids)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	343 if err != nil {
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4171 diff changeset	344 return nil, fmt.Errorf("invalid ISRS code line %d: %v", line, err)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	345 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	346
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	347 if exists, found := checkedGauges[*gid]; found {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	348 if !exists {
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	349 // Just ignore the line since we have already warned
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	350 ignored++
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	351 continue lines
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	352 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	353 } else { // not found in gauge cache
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	354 if err := gaugeCheckStmt.QueryRowContext(
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	355 ctx,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	356 gid.CountryCode,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	357 gid.LoCode,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	358 gid.FairwaySection,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	359 gid.Orc,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	360 gid.Hectometre,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	361 ).Scan(&exists); err != nil {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	362 return nil, err
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	363 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	364 checkedGauges[*gid] = exists
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	365 if !exists {
3504 6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	366 warn("Ignoring data for unknown gauge %s", gid.String())
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	367 ignored++
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	368 continue lines
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	369 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	370 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	371
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	372 md, err := guessDate(row[measureDateIdx])
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	373 if err != nil {
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4171 diff changeset	374 return nil, fmt.Errorf("invalid 'measure_date' line %d: %v", line, err)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	375 }
4035 f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	376 if v := mdMinMax[*gid]; v != nil {
f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	377 if md.Before(v[0]) {
f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	378 v[0] = md
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	379 }
4035 f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	380 if md.After(v[1]) {
f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	381 v[1] = md
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	382 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	383 } else {
4035 f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	384 mdMinMax[*gid] = &[2]time.Time{md, md}
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	385 }
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	386
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	387 newSender := agm.Originator
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	388 newCountryCode := gid.CountryCode
4170 afabd52212d2 Moved country -> spoken language lookup table to common package to flesh out the misc package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4146 diff changeset	389 newLanguageCode := common.CCtoLang[gid.CountryCode]
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	390 newDateIssue := time.Now()
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	391 newReferenceCode := "ZPG"
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	392
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	393 value, err := strconv.ParseFloat(row[valueIdx], 32)
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	394 if err != nil {
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4171 diff changeset	395 return nil, fmt.Errorf("invalid 'value' line %d: %v", line, err)
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	396 }
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	397 newValue := value
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	398
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	399 newDateInfo := newDateIssue
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	400
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	401 newSourceOrganization := newSender
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	402
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	403 agmLines = append(agmLines, newAGMLine(
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	404 *gid,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	405 newCountryCode,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	406 newSender,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	407 newLanguageCode,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	408 newDateIssue,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	409 newReferenceCode,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	410 md,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	411 newValue,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	412 newDateInfo,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	413 newSourceOrganization,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	414 ))
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	415 }
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	416
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	417 oldGMLines := map[models.Isrs]map[int64]*agmLine{}
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	418 for gid, minMax := range mdMinMax {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	419 oldGMLines[gid], err = getOldGMLines(
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	420 ctx, selectStmt, gid, minMax[0], minMax[1])
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	421 if err != nil {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	422 return nil, err
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	423 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	424 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	425
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	426 tx, err := conn.BeginTx(ctx, nil)
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	427 if err != nil {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	428 return nil, err
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	429 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	430 defer tx.Rollback()
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	431
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	432 txInsertStmt := tx.StmtContext(ctx, insertStmt)
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	433 txTrackStmt := tx.StmtContext(ctx, trackStmt)
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	434
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	435 agmLines:
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	436 for _, line := range agmLines {
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	437
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	438 var ase *agmSummaryEntry
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	439
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	440 if old := oldGMLines[line.Location]; old != nil {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	441 ut := line.MeasureDate.Unix()
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	442 if o, ok := old[ut]; ok {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	443 if !o.hasDiff(line) { // identical
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	444 // don't delete
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	445 delete(old, ut)
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	446 continue agmLines
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	447 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	448 ase = &agmSummaryEntry{
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	449 FKGaugeID: line.Location,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	450 MeasureDate: line.MeasureDate,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	451 Versions: []*agmLine{o, line},
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	452 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	453 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	454 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	455 if ase == nil {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	456 ase = &agmSummaryEntry{
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	457 FKGaugeID: line.Location,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	458 MeasureDate: line.MeasureDate,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	459 Versions: []*agmLine{line},
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	460 }
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	461 }
3502 45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	462
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	463 var newID int64
3502 45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	464
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	465 if err := txInsertStmt.QueryRowContext(
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	466 ctx,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	467 line.Location.CountryCode,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	468 line.Location.LoCode,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	469 line.Location.FairwaySection,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	470 line.Location.Orc,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	471 line.Location.Hectometre,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	472 line.MeasureDate.Time,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	473 line.CountryCode,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	474 line.Sender,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	475 line.LanguageCode,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	476 line.DateIssue.Time,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	477 line.ReferenceCode,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	478 line.WaterLevel,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	479 line.DateInfo.Time,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	480 line.SourceOrganization,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	481 ).Scan(&newID); err != nil {
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	482 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	483 }
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	484
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	485 if _, err := txTrackStmt.ExecContext(
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	486 ctx, importID, "waterway.gauge_measurements",
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	487 newID,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	488 false,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	489 ); err != nil {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	490 return nil, err
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	491 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	492
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	493 entries = append(entries, ase)
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	494 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	495
4037 a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	496 var removed int
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	497
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	498 // Issue deletes
4060 15c9d4064f0f AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4049 diff changeset	499 for _, old := range oldGMLines {
4037 a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	500 removed += len(old)
4060 15c9d4064f0f AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4049 diff changeset	501 for _, line := range old {
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	502 if _, err := txTrackStmt.ExecContext(
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	503 ctx, importID, "waterway.gauge_measurements",
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	504 line.id,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	505 true,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	506 ); err != nil {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	507 return nil, err
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	508 }
4049 4a7c2140e44b AGM import: Generate entries for deleted items. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4037 diff changeset	509 entries = append(entries, &agmSummaryEntry{
4a7c2140e44b AGM import: Generate entries for deleted items. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4037 diff changeset	510 FKGaugeID: line.Location,
4a7c2140e44b AGM import: Generate entries for deleted items. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4037 diff changeset	511 MeasureDate: line.MeasureDate,
4a7c2140e44b AGM import: Generate entries for deleted items. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4037 diff changeset	512 Versions: []*agmLine{line, nil},
4a7c2140e44b AGM import: Generate entries for deleted items. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4037 diff changeset	513 })
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	514 }
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	515 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	516
4037 a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	517 feedback.Info("Measurements to update/insert: %d", len(entries))
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	518 feedback.Info("Measurements to delete: %d", removed)
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	519
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	520 if len(entries) == 0 && removed == 0 {
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	521 return nil, UnchangedError("No changes from AGM import")
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	522 }
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	523
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	524 if err = tx.Commit(); err != nil {
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4171 diff changeset	525 return nil, fmt.Errorf("commit failed: %v", err)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	526 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	527
4060 15c9d4064f0f AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4049 diff changeset	528 // Sort here to mix the deletes right beside the matching inserts/updates.
15c9d4064f0f AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4049 diff changeset	529 // This also makes the output deterministic.
15c9d4064f0f AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4049 diff changeset	530 sort.Slice(entries, func(i, j int) bool {
15c9d4064f0f AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4049 diff changeset	531 return entries[i].FKGaugeID.Less(&entries[j].FKGaugeID)
15c9d4064f0f AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4049 diff changeset	532 })
15c9d4064f0f AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4049 diff changeset	533
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	534 feedback.Info("Imported %d entries with changes", len(entries))
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	535 feedback.Info("Importing approved gauge measurements took %s",
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	536 time.Since(start))
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	537
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	538 return entries, nil
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	539 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	540
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	541 func getOldGMLines(
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	542 ctx context.Context,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	543 stmt *sql.Stmt,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	544 location models.Isrs,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	545 from time.Time,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	546 to time.Time,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	547 ) (map[int64]*agmLine, error) {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	548 var (
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	549 oldID int64
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	550 oldCountryCode string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	551 oldSender string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	552 oldLanguageCode string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	553 oldDateIssue time.Time
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	554 oldReferenceCode string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	555 oldMeasureDate time.Time
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	556 oldValue float64
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	557 oldDateInfo time.Time
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	558 oldSourceOrganization string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	559 )
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	560 gmLines := map[int64]*agmLine{}
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	561
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	562 gms, err := stmt.QueryContext(
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	563 ctx,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	564 location.CountryCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	565 location.LoCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	566 location.FairwaySection,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	567 location.Orc,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	568 location.Hectometre,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	569 from,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	570 to,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	571 )
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	572 if err != nil {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	573 return nil, err
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	574 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	575 defer gms.Close()
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	576 for gms.Next() {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	577 if err = gms.Scan(
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	578 &oldID,
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	579 &oldCountryCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	580 &oldSender,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	581 &oldLanguageCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	582 &oldDateIssue,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	583 &oldReferenceCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	584 &oldMeasureDate,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	585 &oldValue,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	586 &oldDateInfo,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	587 &oldSourceOrganization,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	588 ); err != nil {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	589 return nil, err
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	590 }
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	591 line := newAGMLine(
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	592 location,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	593 oldCountryCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	594 oldSender,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	595 oldLanguageCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	596 oldDateIssue,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	597 oldReferenceCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	598 oldMeasureDate,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	599 oldValue,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	600 oldDateInfo,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	601 oldSourceOrganization,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	602 )
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	603 line.id = oldID
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	604 gmLines[oldMeasureDate.Unix()] = line
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	605 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	606 if err = gms.Err(); err != nil {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	607 return nil, err
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	608 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	609 return gmLines, nil
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	610 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	611
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	612 func newAGMLine(
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	613 location models.Isrs,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	614 countryCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	615 sender string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	616 languageCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	617 dateIssue time.Time,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	618 referenceCode string,
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	619 measureDate time.Time,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	620 waterLevel float64,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	621 dateInfo time.Time,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	622 sourceOrganization string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	623 ) *agmLine {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	624 return &agmLine{
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	625 Location: location,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	626 CountryCode: countryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	627 Sender: sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	628 LanguageCode: languageCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	629 DateIssue: timetz{dateIssue},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	630 ReferenceCode: referenceCode,
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	631 MeasureDate: timetz{measureDate},
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	632 WaterLevel: waterLevel,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	633 DateInfo: timetz{dateInfo},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	634 SourceOrganization: sourceOrganization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	635 }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	636 }

Mercurial > gemma

annotate pkg/imports/agm.go @ 5591:0011f50cf216 surveysperbottleneckid