gemma: pkg/imports/agm.go annotate

annotate pkg/imports/agm.go @ 5670:b75d0b303328

Various fixes and improvements of gauges import: - Allow update of erased data (and thereby set erased to false) - Fix source_organization to work with ERDMS2 - Give ISRS of new and updated gauges in summary - Fixed reference of null pointers if revlevels are missing - Fixed reference of null pointer on update errors - Added ISRS to reference_code warning

author	Sascha Wilde <wilde@sha-bang.de>
date	Fri, 08 Dec 2023 17:29:56 +0100
parents	56c589f7435d
children	6270951dda28

rev	line source
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	7 // Copyright (C) 2018, 2019 by via donau
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	13 // * Sascha Wilde <wilde@intevation.de>
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	14 // * Tom Gottfried <tom.gottfried@intevation.de>
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16 package imports
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 import (
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	19 "bufio"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	20 "context"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	21 "database/sql"
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	22 "encoding/csv"
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	23 "encoding/json"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	24 "fmt"
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	25 "io"
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	26 "math"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	27 "os"
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	28 "path/filepath"
4049 4a7c2140e44b AGM import: Generate entries for deleted items. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4037 diff changeset	29 "sort"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	30 "strconv"
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	31 "strings"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	32 "time"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	33
4075 cb74aa69954e Moved TimeParser to common package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4074 diff changeset	34 "gemma.intevation.de/gemma/pkg/common"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	35 "gemma.intevation.de/gemma/pkg/models"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	36 )
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	37
4178 51e90370eced A few more doc strings to make 'golint' a little bit more happy with the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4177 diff changeset	38 // ApprovedGaugeMeasurements is a Job to import
51e90370eced A few more doc strings to make 'golint' a little bit more happy with the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4177 diff changeset	39 // approved gauge measurements from a CVS file
51e90370eced A few more doc strings to make 'golint' a little bit more happy with the imports package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4177 diff changeset	40 // into the database.
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	41 type ApprovedGaugeMeasurements struct {
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	42 Dir string `json:"dir"`
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	43 Originator string `json:"originator"`
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	44 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	45
4146 f464cbcdf2f2 BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ... Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4080 diff changeset	46 // AGMJobKind is the unique name of an approved gauge measurements import job.
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	47 const AGMJobKind JobKind = "agm"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	48
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	49 type agmJobCreator struct{}
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	50
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	51 func init() {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	52 RegisterJobCreator(AGMJobKind, agmJobCreator{})
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	53 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	54
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1743 diff changeset	55 func (agmJobCreator) AutoAccept() bool { return false }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1743 diff changeset	56
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	57 func (agmJobCreator) Description() string {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	58 return "approved gauge measurements"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	59 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	60
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2173 diff changeset	61 func (agmJobCreator) Create() Job { return new(ApprovedGaugeMeasurements) }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	62
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	63 func (agmJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	64 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	65 {"gauge_measurements"},
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	66 {"gauges"},
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	67 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	68 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	69
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	70 const (
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	71 agmStageDoneDeleteSQL = `
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	72 DELETE FROM waterway.gauge_measurements WHERE id IN (
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	73 SELECT key
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1938 diff changeset	74 FROM import.track_imports
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	75 WHERE import_id = $1 AND
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	76 relation = 'waterway.gauge_measurements'::regclass AND
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	77 deletion
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	78 )`
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	79
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	80 agmStageDoneSQL = `
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	81 UPDATE waterway.gauge_measurements SET staging_done = true
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	82 WHERE id IN (
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	83 SELECT key
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	84 FROM import.track_imports
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	85 WHERE import_id = $1 AND
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	86 relation = 'waterway.gauge_measurements'::regclass AND
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	87 NOT deletion
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	88 )`
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	89 )
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	90
5035 56c589f7435d Enhance comments on implementations of StageDone() Tom Gottfried <tom@intevation.de> parents: 5034 diff changeset	91 // StageDone replaces gauge measurements with those in the staging area
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	92 func (agmJobCreator) StageDone(
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	93 ctx context.Context,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	94 tx *sql.Tx,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	95 id int64,
5034 59a99655f34d Added feedback support for StageDone. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4178 diff changeset	96 _ Feedback,
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	97 ) error {
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	98 _, err := tx.ExecContext(ctx, agmStageDoneDeleteSQL, id)
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	99 if err == nil {
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	100 _, err = tx.ExecContext(ctx, agmStageDoneSQL, id)
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	101 }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	102 return err
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	103 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	104
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	105 // CleanUp removes the folder containing the CSV file with the
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	106 // the approved gauge measurements.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	107 func (agm *ApprovedGaugeMeasurements) CleanUp() error {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	108 return os.RemoveAll(agm.Dir)
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	109 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	110
4075 cb74aa69954e Moved TimeParser to common package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4074 diff changeset	111 var guessDate = common.TimeParser([]string{
1938 206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	112 "02.01.2006 15:04",
206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	113 "2006-01-02T15:04:05-07:00",
4074 eb2f949ddfa2 Rename TimeGuesser to TimeParser. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4060 diff changeset	114 }).Parse
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	115
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	116 type timetz struct{ time.Time }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	117
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	118 func (ttz *timetz) MarshalJSON() ([]byte, error) {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	119 return json.Marshal(ttz.Time.Format("2006-01-02T15:04:05-07:00"))
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	120 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	121
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	122 type agmLine struct {
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	123 id int64
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	124 Location models.Isrs `json:"fk-gauge-id"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	125 CountryCode string `json:"country-code"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	126 Sender string `json:"sender"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	127 LanguageCode string `json:"language-code"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	128 DateIssue timetz `json:"date-issue"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	129 ReferenceCode string `json:"reference-code"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	130 MeasureDate timetz `json:"measure-date"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	131 WaterLevel float64 `json:"water-level"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	132 DateInfo timetz `json:"date-info"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	133 SourceOrganization string `json:"source-organization"`
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	134 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	135
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	136 func (a agmLine) hasDiff(b agmLine) bool {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	137 const eps = 0.00001
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	138 return a.CountryCode != b.CountryCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	139 a.Sender != b.Sender \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	140 a.LanguageCode != b.LanguageCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	141 a.ReferenceCode != b.ReferenceCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	142 math.Abs(a.WaterLevel-b.WaterLevel) > eps \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	143 a.SourceOrganization != b.SourceOrganization
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	144 }
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	145
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	146 type agmSummaryEntry struct {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	147 FKGaugeID models.Isrs `json:"fk-gauge-id"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	148 MeasureDate timetz `json:"measure-date"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	149 Versions []*agmLine `json:"versions"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	150 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	151
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	152 const (
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	153 agmSelectSQL = `
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	154 SELECT
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	155 id,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	156 country_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	157 sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	158 language_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	159 date_issue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	160 reference_code,
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	161 measure_date,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	162 water_level,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	163 date_info,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	164 source_organization
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	165 FROM waterway.gauge_measurements
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	166 WHERE
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	167 location
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	168 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	169 AND measure_date BETWEEN $6 AND $7
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	170 AND staging_done
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	171 `
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	172
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	173 agmInsertSQL = `
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	174 INSERT INTO waterway.gauge_measurements (
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	175 location,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	176 measure_date,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	177 country_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	178 sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	179 language_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	180 date_issue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	181 reference_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	182 water_level,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	183 date_info,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	184 source_organization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	185 staging_done
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	186 ) VALUES (
2120 817cd8b89a86 Approved gauge measuments import: Give type hints when insertinf/querying isrs location codes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2105 diff changeset	187 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int),
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	188 $6,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	189 $7,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	190 $8,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	191 $9,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	192 $10,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	193 $11,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	194 $12,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	195 $13,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	196 $14,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	197 false
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	198 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	199 RETURNING id`
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	200
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	201 agmGaugeCheckSQL = `
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	202 SELECT EXISTS(
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	203 SELECT 1 FROM waterway.gauges
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	204 WHERE location = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int))
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	205 `
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	206 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	207
4030 040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	208 func parseAGMHeaders(headers []string, fkGaugeIDIdx, measureDateIdx, valueIdx *int) error {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	209
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	210 headerFields := []struct {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	211 idx *int
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	212 name string
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	213 }{
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	214 {fkGaugeIDIdx, "fk_gauge_id"},
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	215 {measureDateIdx, "measure_date"},
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	216 {valueIdx, "value"}, // "water_level",
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	217 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	218
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	219 nextHeader:
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	220 for i, f := range headers {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	221 h := strings.Replace(strings.ToLower(
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	222 strings.TrimSpace(f)), " ", "_", -1)
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	223
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	224 for j := range headerFields {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	225 if headerFields[j].name == h {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	226 if *headerFields[j].idx != -1 {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	227 return fmt.Errorf(
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4171 diff changeset	228 "there is more than one column namend '%s'", h)
4030 040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	229 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	230 *headerFields[j].idx = i
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	231 continue nextHeader
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	232 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	233 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	234 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	235
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	236 var missing []string
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	237 for i := range headerFields {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	238 if headerFields[i].name != "unit" && *headerFields[i].idx == -1 {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	239 missing = append(missing, headerFields[i].name)
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	240 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	241 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	242 if len(missing) > 0 {
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4171 diff changeset	243 return fmt.Errorf("missing columns: %s", strings.Join(missing, ", "))
4030 040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	244 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	245
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	246 return nil
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	247 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	248
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	249 // Do executes the actual approved gauge measurements import.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	250 func (agm *ApprovedGaugeMeasurements) Do(
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	251 ctx context.Context,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	252 importID int64,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	253 conn *sql.Conn,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	254 feedback Feedback,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	255 ) (interface{}, error) {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	256
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	257 start := time.Now()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	258
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	259 f, err := os.Open(filepath.Join(agm.Dir, "agm.csv"))
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	260 if err != nil {
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	261 return nil, err
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	262 }
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	263 defer f.Close()
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	264
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	265 r := csv.NewReader(bufio.NewReader(f))
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	266 r.Comma = ';'
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	267 r.ReuseRecord = true
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	268
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	269 headers, err := r.Read()
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	270 if err != nil {
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	271 return nil, err
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	272 }
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	273
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	274 var (
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	275 fkGaugeIDIdx = -1
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	276 measureDateIdx = -1
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	277 valueIdx = -1
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	278 )
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	279
4030 040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	280 if err := parseAGMHeaders(
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	281 headers,
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	282 &fkGaugeIDIdx, &measureDateIdx, &valueIdx,
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	283 ); err != nil {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	284 return nil, err
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	285 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	286
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	287 gaugeCheckStmt, err := conn.PrepareContext(ctx, agmGaugeCheckSQL)
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	288 if err != nil {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	289 return nil, err
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	290 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	291 defer gaugeCheckStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	292
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	293 selectStmt, err := conn.PrepareContext(ctx, agmSelectSQL)
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	294 if err != nil {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	295 return nil, err
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	296 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	297 defer selectStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	298
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	299 insertStmt, err := conn.PrepareContext(ctx, agmInsertSQL)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	300 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	301 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	302 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	303 defer insertStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	304
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	305 trackStmt, err := conn.PrepareContext(ctx, trackImportDeletionSQL)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	306 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	307 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	308 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	309 defer trackStmt.Close()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	310
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	311 entries := []*agmSummaryEntry{}
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	312
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	313 checkedGauges := map[models.Isrs]bool{}
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	314
4171 5a650cde0574 Move warning limiter to common package to flesh up the misc package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4170 diff changeset	315 warnLimiter := common.WarningLimiter{Log: feedback.Warn, MaxWarnings: 100}
3504 6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	316 warn := warnLimiter.Warn
6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	317 defer warnLimiter.Close()
6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	318
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	319 agmLines := []*agmLine{}
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	320 ignored := 0
4035 f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	321 mdMinMax := map[models.Isrs]*[2]time.Time{}
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	322
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	323 lines:
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	324 for line := 1; ; line++ {
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	325
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	326 row, err := r.Read()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	327 switch {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	328 case err == io.EOF \|\| len(row) == 0:
3218 c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	329 feedback.Info("Read %d entries in CSV file", line-1)
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	330 if ignored > 0 {
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	331 feedback.Info("%d entries ignored", ignored)
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	332 }
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	333 if ignored == line-1 {
8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	334 return nil, UnchangedError("No entries imported")
8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	335 }
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	336 break lines
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	337 case err != nil:
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	338 return nil, fmt.Errorf("CSV parsing failed: %v", err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	339 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	340
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	341 gids := row[fkGaugeIDIdx]
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	342 gid, err := models.IsrsFromString(gids)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	343 if err != nil {
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4171 diff changeset	344 return nil, fmt.Errorf("invalid ISRS code line %d: %v", line, err)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	345 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	346
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	347 if exists, found := checkedGauges[*gid]; found {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	348 if !exists {
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	349 // Just ignore the line since we have already warned
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	350 ignored++
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	351 continue lines
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	352 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	353 } else { // not found in gauge cache
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	354 if err := gaugeCheckStmt.QueryRowContext(
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	355 ctx,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	356 gid.CountryCode,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	357 gid.LoCode,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	358 gid.FairwaySection,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	359 gid.Orc,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	360 gid.Hectometre,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	361 ).Scan(&exists); err != nil {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	362 return nil, err
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	363 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	364 checkedGauges[*gid] = exists
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	365 if !exists {
3504 6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	366 warn("Ignoring data for unknown gauge %s", gid.String())
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	367 ignored++
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	368 continue lines
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	369 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	370 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	371
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	372 md, err := guessDate(row[measureDateIdx])
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	373 if err != nil {
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4171 diff changeset	374 return nil, fmt.Errorf("invalid 'measure_date' line %d: %v", line, err)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	375 }
4035 f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	376 if v := mdMinMax[*gid]; v != nil {
f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	377 if md.Before(v[0]) {
f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	378 v[0] = md
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	379 }
4035 f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	380 if md.After(v[1]) {
f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	381 v[1] = md
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	382 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	383 } else {
4035 f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	384 mdMinMax[*gid] = &[2]time.Time{md, md}
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	385 }
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	386
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	387 newSender := agm.Originator
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	388 newCountryCode := gid.CountryCode
4170 afabd52212d2 Moved country -> spoken language lookup table to common package to flesh out the misc package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4146 diff changeset	389 newLanguageCode := common.CCtoLang[gid.CountryCode]
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	390 newDateIssue := time.Now()
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	391 newReferenceCode := "ZPG"
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	392
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	393 value, err := strconv.ParseFloat(row[valueIdx], 32)
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	394 if err != nil {
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4171 diff changeset	395 return nil, fmt.Errorf("invalid 'value' line %d: %v", line, err)
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	396 }
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	397 newValue := value
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	398
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	399 newDateInfo := newDateIssue
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	400
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	401 newSourceOrganization := newSender
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	402
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	403 agmLines = append(agmLines, newAGMLine(
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	404 *gid,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	405 newCountryCode,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	406 newSender,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	407 newLanguageCode,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	408 newDateIssue,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	409 newReferenceCode,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	410 md,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	411 newValue,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	412 newDateInfo,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	413 newSourceOrganization,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	414 ))
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	415 }
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	416
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	417 oldGMLines := map[models.Isrs]map[int64]*agmLine{}
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	418 for gid, minMax := range mdMinMax {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	419 oldGMLines[gid], err = getOldGMLines(
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	420 ctx, selectStmt, gid, minMax[0], minMax[1])
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	421 if err != nil {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	422 return nil, err
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	423 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	424 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	425
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	426 tx, err := conn.BeginTx(ctx, nil)
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	427 if err != nil {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	428 return nil, err
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	429 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	430 defer tx.Rollback()
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	431
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	432 txInsertStmt := tx.StmtContext(ctx, insertStmt)
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	433 txTrackStmt := tx.StmtContext(ctx, trackStmt)
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	434
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	435 agmLines:
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	436 for _, line := range agmLines {
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	437
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	438 var ase *agmSummaryEntry
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	439
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	440 if old := oldGMLines[line.Location]; old != nil {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	441 ut := line.MeasureDate.Unix()
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	442 if o, ok := old[ut]; ok {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	443 if !o.hasDiff(line) { // identical
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	444 // don't delete
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	445 delete(old, ut)
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	446 continue agmLines
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	447 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	448 ase = &agmSummaryEntry{
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	449 FKGaugeID: line.Location,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	450 MeasureDate: line.MeasureDate,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	451 Versions: []*agmLine{o, line},
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	452 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	453 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	454 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	455 if ase == nil {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	456 ase = &agmSummaryEntry{
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	457 FKGaugeID: line.Location,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	458 MeasureDate: line.MeasureDate,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	459 Versions: []*agmLine{line},
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	460 }
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	461 }
3502 45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	462
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	463 var newID int64
3502 45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	464
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	465 if err := txInsertStmt.QueryRowContext(
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	466 ctx,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	467 line.Location.CountryCode,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	468 line.Location.LoCode,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	469 line.Location.FairwaySection,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	470 line.Location.Orc,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	471 line.Location.Hectometre,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	472 line.MeasureDate.Time,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	473 line.CountryCode,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	474 line.Sender,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	475 line.LanguageCode,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	476 line.DateIssue.Time,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	477 line.ReferenceCode,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	478 line.WaterLevel,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	479 line.DateInfo.Time,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	480 line.SourceOrganization,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	481 ).Scan(&newID); err != nil {
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	482 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	483 }
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	484
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	485 if _, err := txTrackStmt.ExecContext(
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	486 ctx, importID, "waterway.gauge_measurements",
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	487 newID,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	488 false,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	489 ); err != nil {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	490 return nil, err
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	491 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	492
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	493 entries = append(entries, ase)
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	494 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	495
4037 a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	496 var removed int
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	497
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	498 // Issue deletes
4060 15c9d4064f0f AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4049 diff changeset	499 for _, old := range oldGMLines {
4037 a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	500 removed += len(old)
4060 15c9d4064f0f AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4049 diff changeset	501 for _, line := range old {
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	502 if _, err := txTrackStmt.ExecContext(
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	503 ctx, importID, "waterway.gauge_measurements",
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	504 line.id,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	505 true,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	506 ); err != nil {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	507 return nil, err
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	508 }
4049 4a7c2140e44b AGM import: Generate entries for deleted items. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4037 diff changeset	509 entries = append(entries, &agmSummaryEntry{
4a7c2140e44b AGM import: Generate entries for deleted items. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4037 diff changeset	510 FKGaugeID: line.Location,
4a7c2140e44b AGM import: Generate entries for deleted items. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4037 diff changeset	511 MeasureDate: line.MeasureDate,
4a7c2140e44b AGM import: Generate entries for deleted items. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4037 diff changeset	512 Versions: []*agmLine{line, nil},
4a7c2140e44b AGM import: Generate entries for deleted items. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4037 diff changeset	513 })
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	514 }
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	515 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	516
4037 a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	517 feedback.Info("Measurements to update/insert: %d", len(entries))
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	518 feedback.Info("Measurements to delete: %d", removed)
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	519
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	520 if len(entries) == 0 && removed == 0 {
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	521 return nil, UnchangedError("No changes from AGM import")
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	522 }
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	523
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	524 if err = tx.Commit(); err != nil {
4177 8b75ac5e243e Made 'staticcheck' happy with pgxutils package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4171 diff changeset	525 return nil, fmt.Errorf("commit failed: %v", err)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	526 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	527
4060 15c9d4064f0f AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4049 diff changeset	528 // Sort here to mix the deletes right beside the matching inserts/updates.
15c9d4064f0f AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4049 diff changeset	529 // This also makes the output deterministic.
15c9d4064f0f AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4049 diff changeset	530 sort.Slice(entries, func(i, j int) bool {
15c9d4064f0f AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4049 diff changeset	531 return entries[i].FKGaugeID.Less(&entries[j].FKGaugeID)
15c9d4064f0f AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4049 diff changeset	532 })
15c9d4064f0f AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4049 diff changeset	533
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	534 feedback.Info("Imported %d entries with changes", len(entries))
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	535 feedback.Info("Importing approved gauge measurements took %s",
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	536 time.Since(start))
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	537
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	538 return entries, nil
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	539 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	540
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	541 func getOldGMLines(
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	542 ctx context.Context,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	543 stmt *sql.Stmt,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	544 location models.Isrs,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	545 from time.Time,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	546 to time.Time,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	547 ) (map[int64]*agmLine, error) {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	548 var (
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	549 oldID int64
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	550 oldCountryCode string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	551 oldSender string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	552 oldLanguageCode string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	553 oldDateIssue time.Time
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	554 oldReferenceCode string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	555 oldMeasureDate time.Time
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	556 oldValue float64
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	557 oldDateInfo time.Time
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	558 oldSourceOrganization string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	559 )
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	560 gmLines := map[int64]*agmLine{}
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	561
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	562 gms, err := stmt.QueryContext(
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	563 ctx,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	564 location.CountryCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	565 location.LoCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	566 location.FairwaySection,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	567 location.Orc,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	568 location.Hectometre,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	569 from,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	570 to,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	571 )
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	572 if err != nil {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	573 return nil, err
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	574 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	575 defer gms.Close()
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	576 for gms.Next() {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	577 if err = gms.Scan(
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	578 &oldID,
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	579 &oldCountryCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	580 &oldSender,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	581 &oldLanguageCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	582 &oldDateIssue,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	583 &oldReferenceCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	584 &oldMeasureDate,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	585 &oldValue,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	586 &oldDateInfo,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	587 &oldSourceOrganization,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	588 ); err != nil {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	589 return nil, err
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	590 }
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	591 line := newAGMLine(
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	592 location,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	593 oldCountryCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	594 oldSender,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	595 oldLanguageCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	596 oldDateIssue,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	597 oldReferenceCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	598 oldMeasureDate,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	599 oldValue,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	600 oldDateInfo,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	601 oldSourceOrganization,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	602 )
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	603 line.id = oldID
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	604 gmLines[oldMeasureDate.Unix()] = line
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	605 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	606 if err = gms.Err(); err != nil {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	607 return nil, err
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	608 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	609 return gmLines, nil
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	610 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	611
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	612 func newAGMLine(
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	613 location models.Isrs,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	614 countryCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	615 sender string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	616 languageCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	617 dateIssue time.Time,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	618 referenceCode string,
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	619 measureDate time.Time,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	620 waterLevel float64,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	621 dateInfo time.Time,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	622 sourceOrganization string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	623 ) *agmLine {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	624 return &agmLine{
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	625 Location: location,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	626 CountryCode: countryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	627 Sender: sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	628 LanguageCode: languageCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	629 DateIssue: timetz{dateIssue},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	630 ReferenceCode: referenceCode,
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	631 MeasureDate: timetz{measureDate},
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	632 WaterLevel: waterLevel,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	633 DateInfo: timetz{dateInfo},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	634 SourceOrganization: sourceOrganization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	635 }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	636 }

Mercurial > gemma

annotate pkg/imports/agm.go @ 5670:b75d0b303328